Commodity sales data processing device and storage medium

Abstract

The invention discloses a commodity sales data processing device and a storage medium, which can reduce the burden of shop assistants related to the change operation of commodities. The commodity sales data processing apparatus according to an embodiment includes an acquisition unit that acquires a recognition result of a recognition process for recognizing a commodity based on an image in which the commodity is captured, and a registration unit that executes a registration process for registering the commodity recognized by the recognition process as a purchase object based on the recognition result acquired by the acquisition unit. And a display unit that displays a screen that shows each of the registered commodities, and that accepts a change operation for the registered commodities. A first accepting unit that accepts a change operation on a commodity to be changed displayed on the screen and reflects the content of the change operation on the commodity to be changed displayed on the screen; a second accepting unit that accepts an end operation that instructs the end of the registration process; and a notification unit that performs a process for notifying a shop assistant that the change operation performed before the end operation has been performed if the change operation has been performed.

Description

[0001] This application claims priority to Japanese patent application No. JP2023-112542, filed on Jul. 7, 2023, and cites the contents of the above application, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0002] The embodiments of the present invention relate to a commodity sales data processing device and a storage medium. Background Art

[0003] Currently, there are self-service POS terminals in shopping malls, large retail stores, department stores, etc., where customers can perform product registration and settlement operations for products sold in the store. Among these POS terminals, there are POS terminals that take a picture of the product that the customer wants to buy, recognize the product from the image of the product, and perform product registration based on the recognition result.

[0004] By the way, in the self-service POS terminal, customers can also perform change operations on registered goods. For example, if an erroneous product is recognized from an image and registered, the customer can perform a change operation to delete the product by himself. However, there are also changes that are performed for improper purposes, such as temporarily deleting a registered product and then taking it back as it is. Therefore, when a change operation has been performed, the clerk needs to confirm the change content every time. In addition, depending on the store, there are also cases where customers are prohibited from performing change operations, and the clerk performs the change operation every time.

[0005] However, in the current application method, it is necessary to call the store clerk every time a change operation is performed, which increases the burden on the store clerk, so there is room for improvement. Summary of the invention

[0006] In view of the above problems, the present invention aims to provide a commodity sales data processing device and a storage medium that can reduce the burden on store clerks related to commodity change operations.

[0007] In order to solve the above-mentioned problem, the commodity sales data processing device of the embodiment includes: an acquisition unit for acquiring a recognition result of a recognition process of the commodity based on image recognition of the photographed commodity, a registration unit for performing a registration process of registering the commodity recognized by the recognition process as a purchase object based on the recognition result acquired by the acquisition unit, a display unit for displaying a screen showing each commodity that has been registered, a first receiving unit for accepting a change operation on the registered commodity and reflecting the content of the change operation in the commodity that is the change object displayed on the screen, a second receiving unit for accepting an end operation instructing to end the registration process, and a notification unit for notifying the clerk that the change operation performed before the end operation has been performed when the end operation is performed.

[0008] According to the above-mentioned commodity sales data processing device, since the confirmation work of the store clerk accompanying the change operation of the registered commodity can be performed at one time, the burden on the store clerk can be reduced.

[0009] The above-mentioned commodity sales data processing device further includes: a third receiving unit that receives a confirmation operation performed by a store clerk; and a reflecting unit that reflects the content of the change operation on the registered commodity when the confirmation operation is performed.

[0010] According to the above-mentioned commodity sales data processing device, before the confirmation operation is performed, the customer can be presented with the contents of the change operation while the information of the registered commodity is protected.

[0011] In the above-mentioned commodity sales data processing device, the notification unit notifies the terminal device operated by the store clerk, and the third receiving unit receives the confirmation operation from the terminal device.

[0012] According to the above-mentioned commodity sales data processing device, since the time required for the store clerk who operates the terminal device to move to the commodity sales data processing device can be reduced, the efficiency of the confirmation work can be improved.

[0013] In the above-mentioned commodity sales data processing device, the notification unit notifies the terminal device of information indicating the states of the commodity that is the subject of the change operation before and after the change operation.

[0014] According to the above-mentioned commodity sales data processing device, since it is possible to support the confirmation operation of the status before and after the change in the terminal device, it is possible to realize the efficiency of the confirmation operation.

[0015] The above-mentioned commodity sales data processing device further includes: a settlement unit that performs a settlement process for the registered commodity according to the end operation; and a suppression unit that suppresses the settlement process during a period before the confirmation operation is performed.

[0016] According to the above-mentioned commodity sales data processing device, since it is possible to prevent settlement from being performed in an unconfirmed state without confirmation by the store clerk, it is possible to prevent the occurrence of additional work such as reprocessing of the transaction or additional collection of payment.

[0017] In the above-mentioned commodity sales data processing device, the notification unit notifies the terminal device of image data of a screen in which information corresponding to the content of the change operation generated by the first receiving unit is added to the commodity to be changed.

[0018] According to the above-mentioned commodity sales data processing device, since the store clerk can confirm the state of the screen showing the content of the change operation through the terminal device, the convenience of confirming the change content can be improved.

[0019] In the above-mentioned commodity sales data processing device, the notification unit does not notify the terminal device when the content of change of all commodities that are the target of the change operation is an increase in the number of instructions.

[0020] According to the above-mentioned commodity sales data processing device, when it is assumed that a legitimate change operation has been performed, no notification is made to the store clerk, and thus the workload of the store clerk can be reduced.

[0021] In the above-mentioned commodity sales data processing device, the notification unit does not notify the terminal device when a change operation is performed on a registered commodity whose similarity included in the recognition result acquired by the acquisition unit is smaller than a threshold value.

[0022] According to the above-mentioned commodity sales data processing device, when it is assumed that a legitimate change operation has been performed, no notification is made to the store clerk, and thus the workload of the store clerk can be reduced.

[0023] The above-mentioned commodity sales data processing device further includes: a monitoring camera, wherein the notification unit uses the acceptance of a change operation of a registered commodity as a trigger to notify the terminal device of a monitoring image captured by the monitoring camera.

[0024] According to the above-mentioned commodity sales data processing device, the customer's behavior can be confirmed from the monitoring image by the terminal device that receives the notification information.

[0025] A storage medium according to another aspect of the present invention stores a program that causes a computer of a commodity sales data processing device to function as the following units: an acquisition unit that acquires a recognition result of a recognition process for recognizing a commodity based on an image captured of the commodity; a registration unit that performs a registration process for registering the commodity identified by the recognition process as a purchase object based on the recognition result acquired by the acquisition unit; a display unit that displays a screen showing each of the registered commodities; a first receiving unit that accepts a change operation on a registered commodity and reflects the content of the change operation in the commodity that is the change object shown on the screen; a second receiving unit that accepts an end operation instructing to end the registration process; and a notification unit that, when the end operation is performed, performs a process of notifying a store clerk that the change operation performed before the end operation has been performed.

[0026] According to the above-mentioned storage medium, since the confirmation work of the store clerk accompanying the change operation of the registered commodity can be performed at one time, the burden on the store clerk can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a diagram showing an example of a commodity sales data processing system according to the embodiment.

[0028] Figure 2 This is a diagram for explaining an example of package recognition according to the embodiment.

[0029] Figure 3 This is a diagram showing an example of the hardware configuration of a POS terminal according to the embodiment.

[0030] Figure 4 This is a diagram showing an example of the data structure of the product master file according to the embodiment.

[0031] Figure 5 This is a diagram showing an example of the hardware configuration of the imaging device according to the embodiment.

[0032] Figure 6 This is a diagram showing an example of the hardware configuration of the edge device involved in the embodiment.

[0033] Figure 7 This is a diagram showing an example of the hardware configuration of the store clerk terminal according to the embodiment.

[0034] Figure 8 This is a diagram showing an example of the hardware configuration of the store server according to the embodiment.

[0035] Fig. 9 This is a diagram showing an example of the functional configuration of a POS terminal, an imaging device, and an edge device according to the embodiment.

[0036] Fig.10This is a diagram showing an example of a product selection screen displayed on the POS terminal of the embodiment.

[0037] Fig.11 This is a diagram showing an example of a commodity registration screen displayed on the POS terminal of the embodiment.

[0038] Fig.12 This is a diagram showing an example of a commodity registration screen displayed on the POS terminal of the embodiment.

[0039] Fig.13 This is a sequence diagram showing an example of the flow of processing executed by the commodity sales data processing system of the embodiment.

[0040] Fig.14 This is a flowchart showing an example of processing performed by the POS terminal according to the embodiment.

[0041] Description of Reference Numerals

[0042] 1 system 10 POS terminal 20 camera device 30 edge device 40 clerk terminal

[0043] 50 Store server 101 Recognition result acquisition unit 102 Product registration processing unit

[0044] 103 Change operation receiving unit 104 Notification processing unit 105 Change processing unit

[0045] 106 Payment processing unit 201 Image acquisition unit 202 Image transmission unit

[0046] 203 Code recognition unit 204 Recognition result output unit 301 Image acquisition unit

[0047] 302 Package recognition unit 303 Recognition result output unit 401 Notification receiving unit

[0048] 402 Confirmation Operation Acceptance DETAILED DESCRIPTION

[0049] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the embodiment described below, a commodity sales data processing device in which a customer performs registration and settlement operations for commodities to be purchased is used as an example for description. In addition, the embodiment described below does not limit the present invention.

[0050] Figure 1 FIG. 1 is a diagram showing an example of a commodity sales data processing system according to an embodiment of the present invention. Figure 1In the embodiment, the commodity sales data processing system 1 is set in a store T such as a supermarket, a large retail store, a convenience store, a specialty store, etc. The commodity sales data processing system 1 includes one or more POS (Point of Sales) terminals 10 (commodity sales data processing devices), a camera device 20, an edge device 30, a clerk terminal 40 and a store server 50.

[0051] The POS terminal 10, the edge device 30, the clerk terminal 40, and the store server 50 are connected to a communication line L such as a LAN (Local Area Network) so as to be communicable with each other. In addition, the camera device 20 is connected to each POS terminal 10 via, for example, a USB (Universal Serial Bus) cable U. Alternatively, the camera device 20 may be connected to the communication line L via the POS terminal 10.

[0052] The camera device 20 is a device including a camera element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera device 20 is, for example, disposed on the side facing the customer operating the POS terminal 10, and photographs the goods that the customer wants to purchase. Specifically, in this embodiment, since the customer operating the POS terminal 10 performs a registration process, the customer places the goods to be purchased one by one toward the camera device 20, so that the camera device 20 photographs the goods.

[0053] When the camera 20 acquires an image (hereinafter also referred to as a captured image) by photographing, the camera 20 outputs the acquired image to the edge device 30. Specifically, the camera 20 outputs the image captured at a predetermined frame rate to the edge device 30.

[0054] In addition, the camera device 20 attempts to detect symbols such as a barcode and a two-dimensional code from the image by performing code recognition on the image obtained by shooting. In code recognition, symbols such as a barcode and a two-dimensional code attached to the product are detected from the image, and decoded information obtained by decoding the detected symbols is output to the POS terminal. Because the decoded information contains product designation information that specifies the product with the symbol attached, the photographed product can be specified based on the product designation information. In this embodiment, a product code that can identify each product is used as the product designation information, but as long as it is information that can identify the product, it can be information other than the product code.

[0055] In addition, the camera device 20 can extract an image of a product or an image that may be a product from the captured image and output it to the edge device 30 or use it as a processing object for extracting a symbol. In this case, the image extraction method can use a known technology without particular limitation. For example, the camera device 20 can detect the movement of an object contained in the image by the difference of multiple frames continuous in time, extract the image with the detected movement and use it as a processing object.

[0056] The edge device 30 is an information processing device that performs predetermined processing such as image processing and image recognition. Specifically, the edge device 30 performs package recognition of a commodity from an image input from the imaging device 20 .

[0057] Package recognition is a technique for specifying a product captured by the camera 20 based on the product's appearance features represented by the image. Here, for example, the product's appearance features include the design, pattern, color, texture, shape, etc. of the package.

[0058] Package recognition can use known technologies such as general object recognition, image recognition using models such as deep learning, etc. In package recognition, based on a feature quantity representing the appearance features of a product extracted from an image, one of the products similar to the feature quantity whose similarity (also called likelihood) is above a threshold is designated as the product to be photographed.

[0059] Figure 2 FIG. 2 is a diagram for explaining an example of package recognition. Figure 2 As shown, in package recognition, three types of learned models or dictionary data (hereinafter also collectively referred to as dictionaries) are used, namely, an object detection model 61, a feature extraction model 62, and a comparison dictionary 63, to specify commodities based on images taken by the camera device 20.

[0060] The object detection model 61 is a dictionary for detecting the area of ​​an object reflected in a captured image. When a captured image captured by the imaging device 20 is input, the object detection model 61 detects an area showing a product (hereinafter also referred to as an object image) from the captured image and outputs it.

[0061] The feature extraction model 62 is a dictionary for extracting feature quantities representing the appearance features of the product from the product shown in the area detected by the object detection model 61. When an object image or a camera image including an object image is input, the feature extraction model 62 extracts and outputs feature quantities representing the appearance features of the product shown in the object image.

[0062] The comparison dictionary 63 is a dictionary used to specify a commodity similar to the feature quantity based on the feature quantity of the commodity. For example, the comparison dictionary 63 associates and stores the feature quantity representing the characteristics of the commodity with the commodity code specifying each commodity for a plurality of commodities sold in the store T. In addition, the comparison dictionary 63 calculates the similarity that quantitatively shows the degree to which the plurality of stored commodities are similar to the photographed commodity by comparing the input feature quantity with the stored feature quantity. The higher the similarity, the higher the degree of similarity between the commodity and the photographed commodity (i.e., the accuracy of the photographed commodity). As an example, a commodity with a similarity of 95% or more is very likely to be the same commodity as the photographed commodity.

[0063] The edge device 30 detects the area where the product is reflected from the image captured by the camera 20 using the object detection model 61. Next, the edge device 30 uses the feature extraction model 62 to extract a feature quantity that quantitatively represents the features of the appearance of the product reflected in the area (object image) in the form of a multidimensional vector or the like. Then, the edge device 30 uses the comparison dictionary 63 to compare the extracted feature quantity with the feature quantities of multiple products stored in the dictionary to calculate the similarity, and recognizes the product based on the similarity.

[0064] Specifically, when there is a commodity with a similarity greater than a threshold value (for example, greater than 95%), the edge device 30 designates the commodity with a similarity greater than the threshold value as the commodity photographed by the camera device 20. In addition, when there is no commodity with a similarity greater than the threshold value (for example, when the similarity of any commodity is less than 95%), the edge device 30 does not designate a commodity, but rather multiple commodities with higher similarity as candidate commodities. For example, when the similarity of any commodity is less than the threshold value, the edge device 30 extracts a predetermined number (for example, four) of candidate commodities starting from the commodity with higher similarity. Then, the edge device 30 uses the commodity code of the designated commodity or the candidate commodity as the recognition result of the package recognition, and sends it to the camera device 20 that has sent the camera image. In addition, when the edge device 30 recognizes multiple candidate commodities, it includes the similarity calculated for the candidate commodity in the recognition result and sends it. In addition, when the edge device 30 can uniquely specify the commodity, it also includes the similarity in the recognition result and sends it. Furthermore, when there is a lower limit threshold related to similarity determination and the degree of similarity is lower than the lower limit threshold, the edge device 30 excludes commodities with the degree of similarity from candidate commodities.

[0065] When the camera device 20 receives the recognition result of the package recognition from the edge device 30, it outputs the recognition result to the POS terminal 10. In addition, the camera device 20 outputs the commodity code of the commodity specified by its own code recognition and the commodity code received from the edge device 30, that is, the commodity code specified by the package recognition, to the POS terminal 10 in a recognizable state. For example, the camera device 20 outputs the commodity code of the commodity specified by the code recognition and the commodity code of the commodity specified by the package recognition by combining and recognizable tag values ​​of the code recognition and the package recognition, thereby outputting the commodity code of the commodity specified by the code recognition and the commodity code of the commodity specified by the package recognition.

[0066] In addition, in the present embodiment, the edge device 30 inputs the recognition result of the package recognition to the POS terminal 10 via the camera device 20, but the present invention is not limited thereto and may directly input the recognition result to the POS terminal 10. In this case, the edge device 30 stores, for example, setting information of the IP addresses of the paired POS terminal 10 and the camera device 20, and transmits the recognition result of the commodity or candidate commodity specified by the camera image to the POS terminal 10 corresponding to the camera device 20 as the transmission source of the camera image as the destination.

[0067] The clerk terminal 40 is a terminal device operated by a clerk of the store T. The clerk terminal 40 may be, for example, a portable terminal device or a service terminal disposed in the same area as the POS terminal 10 . The clerk terminal 40 can check the status of the POS terminal 10 .

[0068] The store server 50 is provided in the backstage of the store T, for example. The store server 50 stores a product master file (not shown) associated with the product name, price, product image, etc. of each product sold in the store T in association with the product code of the product. In addition, the store server 50 provides the product master file to each terminal of the POS terminal 10. For example, the store server 50 sends the product master file to each terminal of the POS terminal 10 and stores the product master file locally in the POS terminal 10. In addition, for example, the store server 50 stores the product master file in a manner that can be referenced by each terminal of the POS terminal 10.

[0069] Furthermore, the store server 50 collects the sales data received from the POS terminal 10. The store server 50 manages the sales of the products of the store T based on the collected sales data.

[0070] The POS terminal 10 performs commodity registration processing and settlement processing for commodities sold in the store T by cooperating with other devices. Specifically, the POS terminal is operated by the customer to perform a registration process, which is a process of registering the commodity specified based on the image captured by the camera device 20 when the customer points the commodity at the camera device 20 as a purchase object. In addition, the POS terminal 10 performs a settlement process, which is a process of settling the payment for the currently registered commodity with the amount paid by the customer, in accordance with an operation instructing to end the registration process.

[0071] For example, one or more counters (in the clearing area of ​​store T) are set up. Figure 1 The POS terminal 10 is a commodity sales data processing device operated by the customer himself. When the customer holds the commodity to be purchased at the camera device 20 for registration, the POS terminal 10 performs commodity registration processing of the commodity to be purchased by the customer based on the commodity code of the commodity specified by the above-mentioned code recognition or package recognition. The commodity registration processing refers to the processing of reading the specified commodity code and the commodity information (commodity name, commodity price, etc.) of the commodity corresponding to the commodity code from the commodity master file and storing them in the storage medium.

[0072] In the present embodiment, the storage medium as the storage destination of the commodity registration process includes the POS terminal 10, but the present invention is not limited thereto, and may include, for example, an external device such as the store server 50.

[0073] In addition, the POS terminal 10 performs the settlement processing of the commodities registered through the commodity registration processing according to the operation of instructing to end the commodity registration processing and the operation of instructing to start the settlement processing. The so-called settlement processing refers to the processing of performing the total processing involved in the transaction based on the commodity information stored in the registered commodity storage unit 131 through the commodity registration processing, and specifically refers to the processing of displaying the total amount on the display unit 16 and performing the settlement processing through the medium of cash, credit card, electronic money, etc., the processing of calculating the change based on the prepaid amount and displaying it on the display unit 16 in the case of cash settlement, and the processing of issuing a receipt printed with the commodity information of the commodity that has been settled and the checkout information (total amount, prepaid amount, change amount, points information, etc.) from the printer 18, etc.

[0074] In addition, the POS terminal 10 sends the commodity information and checkout information (also collectively referred to as sales data) of the commodity that has been settled to the store server 50. The timing of sending the sales data from the POS terminal 10 to the store server 50 is when the settlement process is completed in the embodiment, but it is not limited to this. For example, the sales data for one day can be collected and sent when the store T is closed.

[0075] As described above, in the commodity sales data processing system 1 of the present embodiment, the commodity photographed by the camera device 20 is specified by using both code recognition and package recognition. In each of the POS terminals 10, when the photographed commodity can be specified by the first recognition result of the code recognition and package recognition based on the camera image of the camera device 20, the commodity code of the commodity is used for commodity registration processing.

[0076] Next, the configuration of each device included in the commodity sales data processing system 1 will be described.

[0077] Figure 3 FIG. 1 is a diagram showing an example of the hardware configuration of the POS terminal 10. Figure 3 As shown, the POS terminal 10 includes a CPU (Central Processing Unit) 11 as an example of a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory unit 14, and the like.

[0078] The CPU 11 has one or more processing circuits and generally controls the operation of the POS terminal 10. The ROM 12 stores various programs that can be executed by the CPU 11. The RAM 13 is a work area for temporarily storing programs and various data. For example, the RAM 13 stores the recognition results of code recognition or package recognition. In addition, the RAM 13 holds a registered commodity storage unit 131 for storing the commodity code, number of pieces, commodity information, etc. of the commodity that has been registered.

[0079] The memory unit 14 is composed of a non-volatile memory such as a HDD (Hard Disc Drive) or a flash memory that retains the stored information even when the power is turned off. The memory unit 14 stores various programs and setting information that can be executed by the CPU 11. For example, the memory unit 14 stores a control program 141 that can be executed by the CPU 11, a product master file 142, etc. The product master file 142 is associated with the product code of each product sold in the store T to store product information corresponding to the product code, etc.

[0080] Figure 4 142 is a diagram showing an example of the data structure of the product master file 142. Figure 4 As shown, the product master file 142 is associated with the product code of each product, and stores the product information and symbol mark of the product corresponding to the product code.

[0081] Product information includes various information related to the product. For example, product information includes product name, price and product image. Product image is image data showing the appearance of the product. In addition, symbol mark is binary mark information indicating whether a symbol is attached to the product.

[0082] Goods with symbols can be recognized by both code and package identification. On the other hand, goods without symbols are not recognized by code and can only be recognized by package identification. For example, goods without symbols include fresh food and other non-staple foods produced and sold in stores.

[0083] In addition, the data structure of the product master file 142 is not limited to Figure 4 For example, the commodity master file 142 may also be composed of data without symbol marks.

[0084] Back to Figure 3 , the CPU 11, ROM 12, RAM 13 and the memory unit 14 are connected to each other via a bus or the like. Here, the CPU 11, ROM 12 and RAM 13 constitute the control unit 100 of the POS terminal 10. The control unit 100 realizes the functions of the POS terminal 10 by the CPU 11 operating according to the control program 141 stored in the ROM 12 and the memory unit 14 and loaded in the RAM 13.

[0085] Furthermore, the control unit 100 is connected to the operation unit 15 , the display unit 16 , the card reader 17 , the printer 18 , and the like via a bus or the like.

[0086] The operation unit 15 has various operation keys for the operator to perform operation input. The operation unit 15 has a total key as an example of an operation key. The total key is an operation key operated when the product registration process is completed in the POS terminal 10 and the process proceeds to the settlement process. The display unit 16 is, for example, a liquid crystal display, and displays information to the customer who is the operator of the POS terminal 10. Under the control of the control unit 100, the display unit 16 displays, for example, product information (product name, product price, etc.) of the product that has undergone the product registration process and checkout information related to the settlement process. In addition, the operation unit 15 may be a touch panel provided on the display unit 16.

[0087] The card reader 17 is a reading device for reading card information related to settlement from a card, such as a credit card, a debit card, or an electronic money card, when settlement is performed. The printer 18 is a printing device such as a thermal printer. The printer 18 issues a receipt printed with product information, settlement information, and the like related to the product that has been settled, under the control of the control unit 100.

[0088] In addition, the control unit 100 is connected to the communication unit 19 via a bus or the like. The communication unit 19 is a communication interface for connecting to various external devices. The communication unit 19 is connected to the camera 20 via a cable U so as to be communicable. In addition, the communication unit 19 is connected to the edge device 30, the clerk terminal 40, and the store server 50 via a communication line L so as to be communicable.

[0089] Next, the hardware configuration of the imaging device 20 will be described. Figure 5 2 is a diagram showing an example of the hardware configuration of the imaging device 20. Figure 5 As shown, the imaging device 20 includes a CPU 21 as an example of a processor, a ROM 22 , a RAM 23 , a memory unit 24 , and the like.

[0090] The CPU 21 has one or more processing circuits and generally controls the operation of the camera device 20. The ROM 22 stores various programs that can be executed by the CPU 21. The RAM 23 is a work area for temporarily storing programs and various data. For example, the RAM 23 stores images captured by the camera unit 25 described later, intermediate data of code recognition, processing results (recognition results), etc.

[0091] The memory unit 24 is composed of a nonvolatile memory such as a HDD or flash memory that retains stored information even when the power is turned off. The memory unit 24 stores various programs, setting information, etc. that can be executed by the CPU 21. For example, the memory unit 24 stores a control program 241 that can be executed by the CPU 21.

[0092] The CPU 21, ROM 22, RAM 23, and memory unit 24 are connected to each other via a bus or the like. Here, the CPU 21, ROM 22, and RAM 23 constitute a control unit 200 of the imaging device 20. The control unit 200 realizes the functions of the imaging device 20 by the CPU 21 operating in accordance with a control program 241 stored in the ROM 22 and memory unit 24 and loaded in the RAM 23.

[0093] In addition, the control unit 200 is connected to the imaging unit 25 and the communication unit 26 via a bus or the like. The imaging unit 25 is a camera module having an imaging element such as a CCD. The communication unit 26 is a variety of communication interfaces for connecting to external devices. The communication unit 26 is connected to the POS terminal 10 via a cable U so as to be communicable. In addition, the communication unit 26 is connected to the edge device 30 or the like via a communication line L so as to be communicable.

[0094] Next, the hardware configuration of the edge device 30 will be described. Figure 6 FIG. 1 is a diagram showing an example of the hardware configuration of the edge device 30. Figure 6 As shown, the edge device 30 includes a CPU 31 as an example of a processor, a ROM 32 , a RAM 33 , a memory unit 34 , and the like.

[0095] The CPU 31 has one or more processing circuits and generally controls the operation of the edge device 30. The ROM 32 stores various programs that can be executed by the CPU 31. The RAM 33 is a work area for temporarily storing programs and various data. For example, the RAM 33 stores images sent from the camera 20, intermediate data of package recognition, processing results (recognition results), etc.

[0096] The memory unit 34 is composed of a non-volatile memory such as a HDD or a flash memory that retains the stored information even when the power is turned off. The memory unit 34 stores various programs and setting information that can be executed by the CPU 31. For example, the memory unit 34 stores a control program 341 that can be executed by the CPU 31. In addition, in addition to storing the above-mentioned object detection model 61, feature extraction model 62, and comparison dictionary 63, the memory unit 34 also stores threshold setting information 342 that sets a threshold value as an indicator for uniquely specifying a product based on similarity.

[0097] The CPU 31, ROM 32, RAM 33 and memory unit 34 are connected to each other via a bus or the like. Here, the CPU 31, ROM 32 and RAM 33 constitute the control unit 300 of the edge device 30. The control unit 300 realizes the functions of the edge device 30 by the CPU 31 operating in accordance with the control program 341 stored in the ROM 32 and memory unit 34 and loaded in the RAM 33.

[0098] The control unit 300 is connected to the communication unit 35 via a bus or the like. The communication unit 35 is a communication interface for connecting to various external devices. The communication unit 35 is connected to the POS terminal 10, the camera 20, the clerk terminal 40, the store server 50, etc. via a communication line L so as to be communicable.

[0099] Next, the hardware configuration of the store terminal 40 will be described. Figure 7 FIG. 4 is a diagram showing an example of the hardware configuration of the clerk terminal 40. Figure 7 As shown, the clerk terminal 40 includes a CPU 41 as an example of a processor, a ROM 42 , a RAM 43 , a memory unit 44 , and the like.

[0100] The CPU 41 has one or more processing circuits, and generally controls the operation of the clerk terminal 40. The ROM 42 stores various programs executable by the CPU 41. The RAM 43 is a work area for temporarily storing programs and various data.

[0101] The memory unit 44 is composed of a nonvolatile memory such as a HDD or flash memory that retains stored information even when the power is turned off. The memory unit 44 stores various programs and setting information executable by the CPU 41. For example, the memory unit 44 stores a control program 441 executable by the CPU 41.

[0102] The CPU 41, ROM 42, RAM 43, and memory unit 44 are connected to each other via a bus or the like. Here, the CPU 41, ROM 42, and RAM 43 constitute a control unit 400 of the clerk terminal 40. The control unit 400 realizes the functions of the clerk terminal 40 by the CPU 41 operating in accordance with a control program 441 stored in the ROM 42, memory unit 44, and loaded into the RAM 43.

[0103] Furthermore, the control section 400 is connected to the operation section 45 , the display section 46 , and the like via a bus or the like.

[0104] The operation unit 45 has various operation keys for the operator to perform operation input. The display unit 46 is, for example, a liquid crystal display, and displays information to the clerk who is the operator of the clerk terminal 40. Alternatively, the operation unit 45 may be a touch panel provided on the display unit 46.

[0105] The control unit 400 is connected to the communication unit 47 via a bus or the like. The communication unit 47 is a communication interface for connecting to external devices. The communication unit 47 is connected to the POS terminal 10, the camera 20, the edge device 30, and the store server 50 via a communication line L so as to be communicable.

[0106] Next, the hardware configuration of the store server 50 will be described. Figure 8 is a diagram showing an example of the hardware configuration of the store server 50. Figure 8 As shown, the store server 50 includes a CPU 51 as an example of a processor, a ROM 52 , a RAM 53 , a memory unit 54 , and the like.

[0107] The CPU 51 has one or more processing circuits and generally controls the operation of the store server 50. The ROM 52 stores various programs executable by the CPU 51. The RAM 53 is a work area for temporarily storing programs and various data. For example, the RAM 53 temporarily stores the checkout data sent from the store server 50.

[0108] The memory unit 54 is composed of a non-volatile memory such as a HDD or a flash memory that can retain stored information even when the power is turned off. The memory unit 54 stores various programs, setting information, etc. that can be executed by the CPU 51. For example, the memory unit 54 stores a control program 541 that can be executed by the CPU 51. In addition, the memory unit 54 stores a product master file 542, a transaction history DB (database) 543 for storing and managing checkout data, etc. In addition, the data structure of the product master file 542 is the same as that of the product master file 142. As an example, by providing (distributing) the product master file 542 to each terminal of the POS terminal 10, the product master file 542 is stored in each terminal of the POS terminal and used as the product master file 142.

[0109] The CPU 51, ROM 52, RAM 53, and memory unit 54 are connected to each other via a bus or the like. Here, the CPU 51, ROM 52, and RAM 53 constitute a control unit 500 of the store server 50. The control unit 500 realizes the functions of the store server 50 by the CPU 51 operating in accordance with a control program 541 stored in the ROM 52 and memory unit 54 and loaded into the RAM 53.

[0110] The control unit 500 is connected to the communication unit 55 via a bus or the like. The communication unit 55 is a communication interface for connecting to various external devices. The communication unit 55 is connected to the POS terminal 10 or the like via a communication line L so as to be communicable.

[0111] Next, the functional configurations of the POS terminal 10 , the imaging device 20 , the edge device 30 , and the store clerk terminal 40 will be described. Fig. 9 1 is a diagram showing an example of the functional configuration of the POS terminal 10 , the imaging device 20 , the edge device 30 , and the clerk terminal 40 .

[0112] like Fig. 9 As shown, the imaging device 20 includes an image acquisition unit 201, an image transmission unit 202, a code recognition unit 203, and a recognition result output unit 204 as functional components.

[0113] The functional configuration of the camera 20 is implemented by software when the processor of the camera 20 operates according to the control program 241 stored in the ROM 22 and the memory unit 24, but is not limited thereto. For example, part or all of the functional configuration of the camera 20 may be implemented by hardware such as a dedicated circuit.

[0114] The image acquisition unit 201 acquires an image captured by the camera unit 25. The image acquisition unit 201 extracts and acquires an image in which a commodity is shown or an image in which a commodity may be shown from the image captured by the camera unit 25 by using a known technique such as motion detection.

[0115] The image sending unit 202 sends the image acquired by the image acquisition unit 201 to the edge device 30. The code recognition unit 203 performs code recognition on the image acquired by the image acquisition unit 201 to try to detect a symbol such as a barcode from the image. When the code recognition unit 203 detects a symbol from the image, it decodes the symbol and obtains decoding information including the product code as a recognition result of the code recognition.

[0116] The recognition result output unit 204 outputs the recognition result obtained by the code recognition of the code recognition unit 203 to the POS terminal 10. In addition, the recognition result output unit 204 outputs the recognition result of the package recognition received from the edge device 30 to the POS terminal 10. In addition, the recognition result output unit 204 can output the recognition results of the code recognition and the package recognition in a recognizable state by combining and outputting the recognition results with a tag value indicating the recognition method.

[0117] like Fig. 9 As shown, the edge device 30 includes an image acquisition unit 301, a package recognition unit 302, and a recognition result output unit 303 as functional components.

[0118] The functional configuration of the edge device 30 is implemented by software when the processor of the edge device 30 operates according to the control program 341 stored in the ROM 32 and the memory unit 34, but is not limited thereto. For example, part or all of the functional configuration of the edge device 30 may be implemented by hardware such as a dedicated circuit.

[0119] The image acquisition unit 301 acquires images sent from each camera of the camera device 20 via the communication unit 35. The package recognition unit 302 performs the above-mentioned package recognition on the image acquired by the image acquisition unit 301, thereby attempting to detect an object image and extract a feature value from the image. When the package recognition unit 302 confirms that there is a commodity with a similarity above a threshold value in the comparison dictionary 63 based on the feature value extracted from the image, the package recognition unit 302 outputs the commodity code of the commodity as the recognition result. In addition, when there is no commodity with a similarity above the threshold value, the package recognition unit 302 extracts a specified number of commodities with high similarity from the comparison dictionary 63, and outputs information associated with the similarity of the commodity in the commodity code of each extracted commodity as the recognition result.

[0120] In addition, when there are a plurality of commodities having a similarity greater than or equal to a threshold value, the package recognition unit 302 outputs information in which the commodity codes of these commodities and the similarities of the commodities are associated as the recognition result.

[0121] The recognition result output unit 303 outputs the recognition result after performing package recognition on the image to an external device related to the transmission source of the image. Specifically, the recognition result output unit 303 transmits the recognition result after performing package recognition on the image to the camera device 20 of the transmission source of the image via the communication unit 35. In addition, the recognition result output unit 303 can send the recognition result to the POS terminal 10 corresponding to the camera device 20 of the transmission source as the destination.

[0122] In addition, if Fig. 9 As shown, the POS terminal 10 includes a recognition result acquisition unit 101, a commodity registration processing unit 102, a change operation acceptance unit 103, a notification processing unit 104, a change processing unit 105, and a settlement processing unit 106 as functional components.

[0123] The functional configuration of the POS terminal 10 is implemented by software when the processor of the POS terminal 10 operates according to the control program 141 stored in the ROM 12 and the memory unit 14, but is not limited thereto. For example, part or all of the functional configuration of the POS terminal 10 may be implemented by hardware such as a dedicated circuit.

[0124] The recognition result acquisition unit 101 is an example of an acquisition unit. The recognition result acquisition unit 101 acquires the recognition results of code recognition and package recognition related to the own POS terminal 10, which are performed by the camera device 20 and the edge device 30. Specifically, the recognition result acquisition unit 101 acquires the recognition results of code recognition and package recognition sent from the camera device 20 via the communication unit 19. In addition, in the case of a configuration in which the recognition result of package recognition is directly sent from the edge device 30 (recognition result output unit 303) to the POS terminal 10 (see the dotted line in the figure), the recognition result acquisition unit 101 can acquire the recognition result of package recognition from the edge device 30.

[0125] The commodity registration processing unit 102 is an example of a registration unit and a display unit. The commodity registration processing unit 102 performs commodity registration processing of the commodity to be purchased by the customer based on the recognition result obtained by the recognition result obtaining unit 101. In other words, the commodity registration processing unit 102 performs commodity registration processing based on the recognition result of the code recognition of the image of the photographed commodity and the recognition result of the package recognition.

[0126] Specifically, the commodity registration processing unit 102 performs commodity registration processing for a commodity when the commodity is uniquely specified by the identification result of the code identification or the package identification, that is, when the commodity code of a commodity is included in the identification result. In addition, the commodity registration processing unit 102 performs commodity registration processing by using the commodity code of the identification result obtained first in the code identification and the package identification, thereby realizing high-speed processing.

[0127] In addition, when the recognition result of package recognition including multiple commodity codes and their similarities is obtained without obtaining the recognition result of code recognition, that is, when the recognition result including multiple candidate commodities is obtained, the commodity registration processing unit 102 performs commodity registration processing based on the recognition result. Specifically, when multiple candidate commodities are recognized, the commodity registration processing unit 102 causes the display unit 16 to display a screen (hereinafter also referred to as a commodity selection screen) for selecting these candidate commodities.

[0128] Fig.10 FIG. 1 is a diagram showing an example of a commodity selection screen A displayed on the POS terminal 10. Fig.10 As shown, the commodity selection screen A has a display area Aa for displaying commodities corresponding to each of the commodity codes included in the recognition result as candidate commodities.

[0129] In the display area Aa, the product name, price, image, etc. of the candidate product are displayed based on the product information of the candidate product. Fig.10 In FIG. 4 , four candidate products are shown, and a display area Aa is set for each candidate product. In addition, on the product selection screen A, there is no particular restriction on the configuration position of the candidate products, but it is preferred to configure them according to the similarity. For example, it can be configured to configure them in the order of upper left → upper right → lower left → lower right starting from the candidate products with the highest similarity.

[0130] In addition, the display area Aa also functions as a control for selecting a product to be registered. When the customer selects any display area Aa, the product registration processing unit 102 performs product registration processing based on the product code corresponding to the candidate product of the selected display area Aa. Then, the product registration processing unit 102 removes the product selection screen A.

[0131] Furthermore, the commodity selection screen A has a return button Ab. When the return button Ab is operated, the commodity registration processing unit 102 clears the commodity selection screen A and waits until a new recognition result is obtained.

[0132] When executing the product registration process, the product registration processing unit 102 causes the display unit 16 to display a screen (also referred to as a product registration screen) showing product information such as product names and prices of registered products in the order of registered products.

[0133] Fig.11 FIG. 1 is a diagram showing an example of a commodity registration screen displayed on the POS terminal 10. Fig.11 As shown, the commodity registration screen B has a commodity display area Ba and a total display area Bb.

[0134] In the commodity display area Ba, information such as the commodity name, quantity, and price (unit price) of each commodity registered through the commodity registration process (hereinafter also referred to as registered commodity) is displayed in association with the ID indicating the registration order. In addition, the total number of registered commodities and the total purchase amount are displayed in the total display area Bb. Specifically, the commodity registration processing unit 102 calculates the total number of commodities and the total amount based on the price and number of registered commodities, and displays them in the total display area Bb.

[0135] Furthermore, a change operation of a registered commodity can be accepted in the commodity registration screen B. Specifically, the change operation accepting unit 103 accepts a change operation of a registered commodity via the operating unit 15. Here, the change operation of a registered commodity refers to an operation of deleting a registered commodity, changing (increasing or decreasing) the number of registered items, and the like.

[0136] The change operation accepting unit 103 is an example of a first accepting unit. When accepting a change operation, the change operation accepting unit 103 reflects the content of the change operation on the commodity registration screen B. In addition, the change operation accepting unit 103 generates change content information that associates the ID of the registered commodity that is the object of the change operation with the content of the change operation, and stores it in the RAM 13 or the like.

[0137] Here, the change operation accepting unit 103 changes the display of the commodity registration screen B according to the content of the change operation, but does not change the actual registered commodity storage unit 131 that has been registered through the commodity registration process. Fig.11 If you instruct to delete the registered product with ID "1" shown in the figure, Fig.12 As shown in FIG. 1 , the change operation receiving unit 103 adds a strikethrough to the registered commodity with ID “1” and displays it. Fig.11 If the number of registered products with ID "4" is reduced by 1, Fig.12 As shown, the number of items with ID "4" is displayed minus 1. In addition, when an instruction is given to increase or decrease the number of items, it is preferable to highlight the changed number so that the item can be recognized as being the subject of the change operation.

[0138] Then, the change operation accepting unit 103 updates and displays the total number of items and the total amount in the total display area Bb according to the content of the change operation. This allows the operator of the POS terminal 10 to intuitively confirm the content of the change operation that he / she has performed.

[0139] Furthermore, a checkout button Bc for instructing to end the product registration process (or start the payment process) is provided on the product registration screen B. The checkout button Bc is an example of a second receiving unit.

[0140] Incidentally, in a self-service commodity sales data processing device where customers perform commodity registration operations themselves, since the customers have the initiative in the operations, changes may be made for not only legitimate reasons but also illegitimate reasons.

[0141] For example, in package recognition, products are recognized by an accuracy value called similarity, so there is a possibility that products are registered incorrectly. In this case, in order to register the correct product, a legitimate change operation to delete the incorrectly registered product is sometimes necessary. On the other hand, there is a change operation based on an improper reason, such as temporarily deleting a registered product for the purpose of stealing, etc., and taking back the deleted product as it is.

[0142] For the above reasons, when using a self-service commodity sales data processing device to change the registered commodity, it is preferred that the clerk confirms the change. However, if the clerk is called every time the change operation is performed, it will bring a large burden to the clerk, and there is room for improvement.

[0143] Therefore, in the POS terminal 10 of the present embodiment, the store clerk's confirmation is performed once when performing a change operation of a registered commodity, thereby reducing the burden on the store. Specifically, when the end of the commodity registration process or the start of the settlement process is instructed by operating the checkout button Bc, etc., the notification processing unit 104 determines whether a change operation of the registered commodity has been performed before the operation is performed based on the presence or absence of change content information.

[0144] The notification processing unit 104 is an example of a notification unit. When there is change content information, the notification processing unit 104 determines that a change operation of the registered commodity has been performed, and sends information notifying that the change operation has been performed (hereinafter also referred to as notification information) to the clerk terminal. For example, the notification information includes terminal information such as a terminal identifier that can specify the POS terminal 10, and also includes change content information.

[0145] Furthermore, the notification information may include information about the commodity registered in the registered commodity storage unit 131. For example, the notification information may include information about the registered commodity corresponding to the ID included in the change content information among the registered commodities registered in the registered commodity storage unit 131, that is, information about the registered commodity before the change operation. Accordingly, since the information and status of the registered commodity before and after the change operation can be confirmed by the clerk terminal 40 receiving the notification information, the convenience of confirming the change content can be improved.

[0146] In addition, the notification information may include image data of the product registration screen B displayed on the display unit 16 of the POS terminal 10. That is, it may include image data of a screen in which information corresponding to the content of the change operation generated by the first receiving unit (deletion line, highlighting of the number of pieces) is added to the product to be changed. In this case, the image data may be a snapshot of the product registration screen B or image data of a real-time product registration screen B displayed on the display unit 16 that is streamed. Accordingly, since the state of the product registration screen B showing the content of the change operation can be confirmed by the clerk terminal 40 that receives the notification information, the convenience of confirming the content of the change can be improved.

[0147] The change processing unit 105 is an example of a third receiving unit, a reflecting unit, and a suppressing unit. The change processing unit 105 performs processing to reflect the content of the registration process changed by the customer in the registered commodity storage unit 131 based on the change content information. Specifically, when the commodity registration process is terminated or the settlement process is started by an operation instruction such as the checkout button Bc in a state where the change content information exists, the change processing unit 105 suppresses the execution of the settlement process to keep the registered commodity storage unit 131 in a state where the operation is possible.

[0148] Furthermore, the change processing unit 105 performs processing to reflect the change contents of the registered commodity stored in the change content information in the registered commodity storage unit 131 according to the confirmation operation of the store clerk. Here, the confirmation operation performed by the store clerk can be performed via the operation unit 15 of the POS terminal 10 or can be performed remotely from the store clerk terminal 40. In the former case, after the change processing unit 105 reads the store clerk ID of the store clerk via, for example, the card reader 17 or the camera device 20, the change contents are reflected in the registered commodity storage unit 131 on the condition that an operation indicating permission of the change contents has been performed. In addition, in the latter case, the change processing unit 105 reflects the change contents in the registered commodity storage unit 131 on the condition that, for example, information indicating permission of the change contents has been received from the store clerk terminal 40.

[0149] In addition, the clerk can confirm the change content for each registered commodity that has been changed, or can confirm the change content for all registered commodities that have been changed at once. In addition, the change processing unit 105 can cancel the change content and make further changes according to the clerk's operation.

[0150] When all the changes are reflected in the registered commodity storage unit 131, the change processing unit 105 allows the transition to the settlement process, thereby notifying the settlement processing unit 106 to start the settlement process. In addition, in a state where there is no change content information, when the end of the commodity registration process or the start of the settlement process is instructed, the settlement process is immediately started by the settlement processing unit 106.

[0151] The settlement processing unit 106 is an example of a settlement unit. The settlement processing unit 106 performs settlement processing on the commodities registered by the commodity registration processing. Specifically, the settlement processing unit 106 calculates the total amount of the registered commodities based on the information registered in the registered commodity storage unit 131, and performs settlement processing to settle the total amount with the payment amount paid by the customer.

[0152] The settlement process may be performed by the POS terminal 10 alone or in cooperation with other devices (eg, a checkout device, a settlement server, etc.).

[0153] For example, when the settlement processing unit 106 receives an operation indicating the start of settlement via the operation unit 15, the display unit 16 displays a screen that allows the selection of settlement methods such as cash settlement, electronic money settlement, and credit card settlement. When cash settlement is selected, the settlement processing unit 106 performs a settlement process for settling the total amount of registered goods with the payment amount paid in via a money payment device (not shown). In addition, when electronic money settlement or credit card settlement is selected, the settlement processing unit 106 receives input of settlement information for settlement with the selected settlement method, and cooperates with an external settlement server to perform a settlement process for settling the total amount of sales and registered goods based on the input settlement information. The relevant settlement information, for example, can be either a credit card number or a QR code for code settlement. In addition, in the case of a QR code for code settlement, the input can be received via the camera device 20.

[0154] In addition, if Fig. 9 As shown, the clerk terminal 40 includes a notification receiving unit 401 and a confirmation operation receiving unit 402 as functional components.

[0155] The functional configuration of the clerk terminal 40 is a software configuration implemented by the processor of the clerk terminal 40 operating according to the control program 441 stored in the ROM 42 and the memory unit 44, but is not limited thereto. For example, part or all of the functional configuration included in the clerk terminal 40 may be implemented by hardware such as a dedicated circuit.

[0156] The notification receiving unit 401 performs notification based on the notification information sent from the POS terminal 10. Specifically, when the notification receiving unit 401 receives the notification information from the POS terminal 10, it causes the display unit 46 to display the terminal information and the content of the change operation included in the notification information, thereby notifying the store clerk that the change operation of the registered commodity has been performed through the POS terminal 10. In addition, when the notification information includes the status of the registered commodity before and after the change operation, the notification receiving unit 401 can display the status of the registered commodity before and after the change operation for comparison. In addition, the method of notification is not limited to display, and notification can be performed using sound.

[0157] The confirmation operation accepting unit 402 accepts an operation of allowing a change operation or the like by the POS terminal 10 via the operation unit 45. For example, when the confirmation operation accepting unit 402 accepts an operation of allowing a change operation of the terminal information, it transmits instruction information indicating the content of the allowed change to the POS terminal 10 corresponding to the terminal information.

[0158] In the case of an application in which the change operation is confirmed by the POS terminal 10 , the confirmation operation receiving unit 402 may be removed from the clerk terminal 40 .

[0159] Next, the flow of processing executed by the commodity sales data processing system 1 will be described. Fig.13 This is a sequence diagram showing an example of the flow of processing executed by the commodity sales data processing system 1.

[0160] First, in the POS terminal 10, the customer performs a registration operation of placing the product to be purchased at the camera device 20. The image acquisition unit 201 of the camera device 20 acquires the image captured by the camera unit 25 (step S1). Next, the image transmission unit 202 of the camera device 20 transmits the captured image to the edge device 30 (step S2). In addition, the code recognition unit 203 of the camera device 20 performs code recognition on the captured image (step S3).

[0161] When an image is sent from the camera device 20, the package recognition unit 302 of the edge device 30 performs package recognition on the image (step S4). Here, when the product is uniquely identified by package recognition, or multiple candidate products are identified, the recognition result output unit 303 of the edge device 30 outputs the recognition result including the product code of the identified product to the camera device 20 (step S5).

[0162] When the recognition result output unit 204 of the camera device 20 receives the recognition result from the edge device 30, it outputs the recognition result to the POS terminal 10 (step S6). In addition, when the code recognition unit 203 uniquely identifies the commodity by code recognition, it outputs the recognition result including the commodity code of the identified commodity to the POS terminal 10 (step S6).

[0163] In addition, when photographing the product through the customer's registration operation, the product enters the imaging range of the imaging device 20 in order from the edge of the product. Therefore, it is highly likely that the recognition result of the package recognition is output first and the recognition result of the code recognition is output later to the POS terminal 10. However, the order of outputting the recognition results of the code recognition and the package recognition is not limited to this.

[0164] When the product registration processing unit 102 of the POS terminal 10 receives the recognition result from the camera 20, it performs product registration processing based on the recognition result (step S7). In addition, each time a customer performs a product registration operation, the POS terminal 10, the camera 20 and the edge device 30 perform the above-mentioned steps S1 to S7.

[0165] Furthermore, when the change operation accepting unit 103 of the POS terminal 10 accepts the change operation for the registered commodity (step S8), it generates change content information and updates the display of the commodity registration screen B according to the content of the change operation (step S9).

[0166] Next, when the POS terminal 10 accepts the end operation of commodity registration (step S10), the notification processing unit 104 transmits notification information to the clerk terminal 40 because the change operation of the registered commodity was performed in step S8 (step S11).

[0167] On the other hand, in the clerk terminal 40, when receiving the notification information from the POS terminal 10, the notification receiving unit 401 notifies the clerk that the change operation of the registered commodity has been performed through the POS terminal 10 based on the notification information (step S12). Here, for example, in the clerk terminal 40, when a confirmation operation indicating that the change content is allowed is performed, the confirmation operation receiving unit 402 receives the confirmation operation (step S13) and sends the indication information indicating that the change content is allowed to the POS terminal 10 (step S14).

[0168] When receiving the instruction information from the clerk terminal 40, the change processing unit 105 of the POS terminal 10 determines that the change content has been confirmed, and reflects the change content of the registered commodity stored in the change content information in the registered commodity storage unit 131 (step S15).

[0169] Next, the settlement processing unit 106 of the POS terminal 10 calculates the total amount of the registered commodities registered in the registered commodity storage unit 131 , and executes settlement processing based on the total amount (step S16 ).

[0170] Next, refer to Fig.14 , the processing performed by the POS terminal 10 is described. Here, Fig.14 is a flowchart showing an example of processing performed by the POS terminal 10. Fig.13 The processing corresponds to the processing in steps S7 to S15. In addition, it is assumed that the commodity registration screen B is displayed on the display unit 16 as a premise of this processing.

[0171] First, the recognition result acquisition unit 101 determines whether the recognition result has been acquired from the camera 20 (step S21). Here, if the recognition result has not been acquired (step S21; No), the process transitions to step S26. In addition, when the recognition result acquisition unit 101 acquires the recognition result from the camera 20 (step S21; Yes), the process transitions to step S22.

[0172] The commodity registration processing unit 102 determines whether the commodity can be uniquely specified based on the recognition result obtained in step S21 (step S22). Specifically, the commodity registration processing unit 102 determines whether one commodity code recognized by code recognition or package recognition is included in the recognition result.

[0173] Here, when it is determined that the commodity can be uniquely specified (step S22 ; Yes), the commodity registration processing unit 102 executes commodity registration processing based on the commodity code included in the recognition result (step S25 ), and the process proceeds to step S26 .

[0174] When executing the commodity registration process, the commodity registration processing unit 102 displays commodity information such as commodity names and prices of the registered commodities on the commodity registration screen B in the order of the registered commodities.

[0175] On the other hand, when it is determined that the commodity cannot be uniquely specified (step S22; No), that is, when the recognition result includes commodity codes of a plurality of candidate commodities, the commodity registration processing unit 102 causes the display unit 16 to display a commodity selection screen A on which one commodity can be selected from each candidate commodity (step S23). Next, the commodity registration processing unit 102 waits until a commodity is selected from the commodity selection screen A (step S24; No).

[0176] In step S24, when a product is selected from the product selection screen A (step S24; Yes), the product registration processing unit 102 performs product registration processing based on the product code of the selected product (step S25), and transitions to step S26. In addition, the product registration processing unit 102 removes the product selection screen A and displays the product registration screen B in accordance with the selection of the product in step S15.

[0177] Next, the change operation accepting unit 103 determines whether a change operation for any registered commodity has been accepted (step S26 ). If a change operation has not been accepted (step S26 ; No), the process proceeds to step S29 .

[0178] When the change operation accepting unit 103 determines that the change operation has been accepted (step S26; Yes), it updates the product registration screen B based on the accepted change content (step S27). Next, the change operation accepting unit 103 stores change content information indicating the change content (step S28), and the process transitions to step S29.

[0179] Next, the control unit 100 of the POS terminal 10 determines whether an operation to end the commodity registration process (or an operation to start the settlement process) has been received (step S29). If there is no instruction to end the commodity registration process (step S29; No), the process returns to step S21 and prepares to obtain a new recognition result.

[0180] Furthermore, when receiving an operation indicating the end of the product registration process (step S29; Yes), the notification processing unit 104 (and the change processing unit 105) determines whether a change operation of the registered product has been performed before the end instruction (step S30). Here, when no change operation has been performed (step S30; No), the process immediately transitions to Fig.13 Step S16 in .

[0181] Furthermore, when it is determined in step S30 that a change operation has been performed (step S30; Yes), the notification processing unit 104 sends a notification message to the clerk POS terminal 40 notifying that a change operation of the registered commodity has been performed through the POS terminal 10 (step S31). Furthermore, the change processing unit 105 waits until the clerk performs a confirmation operation (step S32; No).

[0182] Furthermore, while waiting for the confirmation operation by the store clerk, the change processing unit 105 may cause the display unit 16 to display a message to guide the store clerk to confirm or a message to guide the clerk to wait for settlement before completing confirmation of the change details.

[0183] When the change processing unit 105 receives the confirmation operation performed by the clerk via the direct operation of the operation unit 15 or the remote operation of the clerk terminal 40 (step S32; yes), it reflects the change content recorded in the change content information to the registered product storage unit 131 (step S33), and then the processing transitions to Fig.13 Step S16.

[0184] As described above, the POS terminal 10 obtains the recognition results of code recognition and package recognition based on the image of the product, and performs the product registration process of the product recognized by the recognition process. In addition, the POS terminal 10 displays the product registration screen for each product to be registered, and when accepting the change operation of the registered product, the content of the change operation is reflected in the product of the change object represented by the product registration screen. Then, when the POS terminal 10 receives the operation of ending the product registration process or starting the settlement process, it sends a notification message containing the content of the change operation performed before the operation to the clerk terminal 40. Accordingly, in the POS terminal 10, since the clerk can perform the confirmation operation at one time along with the change operation of the registered product, the burden on the clerk can be reduced.

[0185] In addition, the POS terminal 10 reflects only the content of the change operation on the commodity registration screen before the clerk performs the confirmation operation, and reflects the content of the change operation in the registered commodity storage unit 131 on the condition that the confirmation operation has been performed. Accordingly, the POS terminal 10 can prompt the customer with the content of the change operation while protecting the registered commodity storage unit 131. In addition, in the POS terminal 10, the recognition of the registered commodity that is the object of the change operation can be improved when the clerk confirms the commodity registration screen, so that the efficiency of the confirmation operation can be achieved.

[0186] The POS terminal 10 receives the confirmation operation from the clerk terminal 40 and reflects the contents of the change operation in the registered commodity storage unit 131. Thus, the POS terminal 10 can reduce the time required for the clerk operating the clerk terminal 40 to move to the POS terminal 10, thereby improving the efficiency of the confirmation operation.

[0187] Furthermore, even if the POS terminal 10 receives an operation to terminate the commodity registration process or to start the settlement process when the change operation has been performed, the settlement process is suppressed until the clerk performs the confirmation operation. Thus, the POS terminal 10 can prevent the settlement from being performed in an unconfirmed state without the clerk's confirmation, thereby preventing the occurrence of additional operations such as reprocessing of the transaction and additional collection of payment. Therefore, the POS terminal 10 can realize smooth transaction.

[0188] Furthermore, the POS terminal 10 notifies the clerk terminal 40 of information indicating the status of the merchandise before and after the change as notification information. Thus, the POS terminal 10 can support the clerk terminal 40 in confirming the status before and after the change, thereby making the confirmation operation more efficient.

[0189] In addition, the above-mentioned embodiment can also be appropriately modified and implemented by changing a part of the composition or function of each device. Therefore, the following describes several modified examples involved in the above-mentioned embodiment as other embodiments. In addition, the following mainly describes the points different from the above-mentioned embodiment, and the detailed description of the points common to the described content is omitted. In addition, the modified examples described below can be implemented individually or in appropriate combination.

[0190] (Variation 1)

[0191] In the above embodiment, the form of accepting the change operation of the registered commodity is described, but it is not limited to the commodity registered by any method of code recognition and package recognition (hereinafter also referred to as commodity designation method). However, it is not limited to this form, and the content of the processing related to the change operation can be changed according to the commodity designation method.

[0192] Specifically, when executing the commodity registration processing, the commodity registration processing unit 102 of this modification example combines and registers the commodity designation method used to designate the commodity with the commodity information, etc. Then, the change operation receiving unit 103, the notification processing unit 104, and the change processing unit 105 execute processing according to the commodity designation method of the registered commodity that is the object of the change operation.

[0193] For example, the change operation accepting unit 103 may be configured to allow the change operation to be performed on the registered commodity registered by using the package identification, but not to allow the change operation to be performed on the registered commodity registered by using the code identification. In addition, for example, the notification processing unit 104 may also include the commodity designation method of the commodity as the object of the change operation in the notification information and send it. In addition, for example, the change processing unit 105 may make the confirmation operation methods for the code identification and the package identification different. As an example, when the registered commodity registered by the package identification is included in the object of the change operation, confirmation by remote operation may not be allowed, and confirmation operation may be required to be performed through the POS terminal 10 of the user.

[0194] As described above, in the POS terminal 10 according to the present modification, the content of the change operation process can be changed according to the commodity specifying method, and thus various application forms can be supported.

[0195] (Variation 2)

[0196] In the above embodiment, when the registered commodity is changed, the confirmation operation of the clerk is required. However, the present invention is not limited to this, and the confirmation operation of the clerk is not required when the registered commodity to be changed meets the prescribed conditions.

[0197] For example, the similarity calculated when the package is identified may be used as an indicator to switch whether the confirmation operation of the clerk is required. In this case, when the commodity registration processing unit 102 performs the commodity registration processing, it combines the similarity calculated when the package is identified for the commodity to be registered with the commodity information and the like and registers them. In addition, when the commodity is specified by code identification, for example, the similarity may be left blank or may be 100%.

[0198] Incidentally, the products registered due to misidentification tend to have a low similarity. Therefore, a threshold value may be set for the similarity at which misidentification may occur, and the configuration may be used to switch whether the confirmation operation of the clerk is required based on the threshold value. For example, when the value is less than the threshold value, it is determined that a legitimate change operation (deletion operation) for registering the correct product has been performed, and the confirmation operation of the clerk is not required. On the other hand, when the value is above the threshold value, regardless of whether the correct product has been registered, since the change operation of the product has been performed, it is determined that an improper change operation may have been performed and the confirmation operation of the clerk is required.

[0199] In this case, when the similarity of all the registered products that are the object of the change operation is less than the threshold value, the notification processing unit 104 does not notify the clerk terminal 40. In addition, when the similarity of some of the registered products that are the object of the change operation is greater than the threshold value, the notification processing unit 104 performs notification to the clerk terminal 40 in the same manner as in the above-mentioned embodiment. In addition, the notification processing unit 104 is configured to exclude information related to the registered products whose similarity is less than the threshold value from the notification information.

[0200] Furthermore, when the similarity of all the registered commodities to be changed is less than the threshold value, the change processing unit 105 does not require the confirmation operation of the store clerk, and reflects the change content stored in the change content information in the registered commodity storage unit 131. Furthermore, when the similarity of a part (or all) of the registered commodities to be changed is greater than the threshold value, the change processing unit 105 needs to hold the reflection to the registered commodity storage unit 131 before performing the confirmation operation.

[0201] As described above, the POS terminal 10 according to the present modification example can also change the necessity and method of confirmation by the store clerk according to the registered commodity to be changed, and thus can cope with various application forms.

[0202] (Variant 3)

[0203] In the above embodiment, the confirmation operation by the store clerk is required regardless of the content of the change operation. However, the present invention is not limited to this, and the confirmation operation by the store clerk may not be required when the content of the change operation meets the prescribed conditions.

[0204] Specifically, when the content of the change operation is to instruct to increase the number of items, it is assumed that it is a legitimate change operation to increase the number of items of the previously registered products in order to omit repeated registration operations for multiple products of the same category. Therefore, when the content of the change operation is to instruct to increase the number of items, the confirmation operation of the clerk is not required.

[0205] In this case, when the content of change of all the registered commodities that are the object of the change operation is to increase the number of pieces, the notification processing unit 104 does not notify the clerk terminal 40. In addition, for some (or all) of the registered commodities that are the object of the change operation, when the content of change is to delete or reduce the number of pieces, the notification processing unit 104 performs notification to the clerk terminal 40 in the same manner as in the above-mentioned embodiment. In addition, the notification processing unit 104 may be configured to exclude information related to the registered commodities whose content of change is to increase the number of pieces from the notification information.

[0206] Furthermore, when the content of the change for all the registered commodities that are the object of the change operation is to increase the number of items, the change processing unit 105 does not require the confirmation operation of the store clerk, and reflects the content of the change stored in the change content information in the registered commodity storage unit 131. Furthermore, when the content of the change for a part (or all) of the registered commodities that are the object of the change operation is to delete or reduce the number of items, the change processing unit 105 requires the confirmation operation of the store clerk as in the above-mentioned embodiment, and withholds the reflection to the registered commodity storage unit 131 before the confirmation operation is performed.

[0207] As described above, in the POS terminal 10 according to this modification, since the necessity and confirmation method of the clerk can be changed according to the content of the change operation, various application forms can be supported.

[0208] (Variant 4)

[0209] In the above embodiment, when a change operation is performed on a registered commodity, a notification is sent to the clerk terminal 40 at the timing of instructing to end the commodity registration process (or instructing to start the settlement process). However, the timing of sending notification to the clerk terminal 40 is not limited to this, and notification may be sent at the timing of the change operation.

[0210] In this case, the notification processing unit 104 includes change content information indicating the change content that has been performed in the most recent change operation in the notification information, and transmits it to the clerk terminal 40. In addition, the change processing unit 105 reflects the change content of the change operation in the registered commodity storage unit 131 according to the confirmation operation of the clerk. In this case, the change processing unit 105 can perform the registration and change operation of the new commodity even if the confirmation operation of the clerk is not performed. However, it is preferable to suppress the settlement process so that it is not performed when the confirmation operation of the clerk is not performed.

[0211] The notification processing unit 104 may notify the clerk terminal 40 each time a change operation is performed, or may control whether to notify the clerk terminal 40 based on the content of the change operation, the similarity of the registered product that is the target of the change operation, and the like.

[0212] For example, the notification processing unit 104 may be configured not to notify the terminal 40 when the content of the change operation is to instruct to increase the number of items. In addition, the notification processing unit 104 may be configured not to notify the clerk terminal 40 when a change operation is performed on a registered commodity whose similarity is less than a threshold value. In other words, when a change operation is performed on a registered commodity whose similarity is less than a threshold value (for example, less than 95%) included in the recognition result obtained by the registered processing acquisition unit, the clerk terminal 40 is not notified. In this way, when there is a high possibility that a legitimate change operation is performed, by excluding it from the objects of notification, the chances of the clerk performing the confirmation operation can be reduced, and the burden on the clerk can be reduced.

[0213] Furthermore, the notification processing unit 104 may use a notification at the time of instructing the end of the product registration process (or instructing the start of the settlement process) and a notification at the time of performing a change operation. For example, when the content of the change operation is to delete a registered product and the similarity of the registered product is above a threshold, in the case where there is a possibility of an improper change operation, the notification processing unit 104 performs a notification at the time of performing the change operation. On the other hand, when performing a change operation under other conditions, the notification is performed at the time of instructing the end of the product registration process (or instructing the start of the settlement process).

[0214] Accordingly, the POS terminal 10 according to the present modification example can notify the clerk when an improper change operation is performed, so that the change content can be quickly confirmed. In addition, the POS terminal 10 notifies the clerk when a legitimate change operation such as a change operation may be performed on a registered commodity with a similarity less than a threshold value when the commodity registration process is terminated, so that the confirmation operation can be completed at one time. Therefore, in the POS terminal 10, the chances of the clerk performing the confirmation operation can be reduced, and the burden on the clerk can be reduced.

[0215] (Change Example 5)

[0216] In the above embodiment, the state of the registered commodity before and after the change operation is included in the notification information and sent. In this variation, the actual state of the registered commodity as the object of the change operation is included in the notification information and sent in order to achieve further efficiency of the confirmation operation.

[0217] In this case, the POS terminal 10 further includes an imaging device (hereinafter also referred to as a surveillance camera) such as a camera that can capture images of commodities brought by the customer to the POS terminal 10. The imaging device 20 can be used to implement a surveillance camera.

[0218] In addition, the change operation accepting unit 103 of this variant uses the acceptance of the change operation of the registered product as a trigger to obtain the surveillance image (still image or dynamic image) taken by the surveillance camera at a specified time, and stores it in association with the change content information of the registered product. In addition, when sending the notification information to the clerk terminal 40, the notification processing unit 104 of this variant includes the surveillance image associated with the change content information in the notification information and sends it. That is, the notification unit uses the acceptance of the change operation of the registered product as a trigger to notify the clerk terminal 40 of the surveillance image of the specified time obtained by the surveillance camera.

[0219] According to this, the clerk terminal 40 receiving the notification information can confirm the content of the change operation and the status of the commodity shown on the monitoring screen. Specifically, the clerk at the clerk terminal 40 can confirm the actual status of the commodity or the customer's action when the change operation is performed from the monitoring image. Therefore, the POS terminal 10 involved in this embodiment can provide the clerk with judgment materials on whether it is an improper operation, so it can support the confirmation operation when the change operation is performed.

[0220] (Variant 6)

[0221] In the above embodiment, when the change operation of the registered commodity is performed, the change operation is notified to the clerk by sending notification information to the clerk terminal 40. However, the notification method is not limited to this, and the clerk may be notified by other methods.

[0222] For example, if the POS terminal 10 is equipped with a notification device such as a warning light, the notification processing unit 104 may be in a form of controlling the notification device to notify that the change operation has been performed. In this case, the clerk will recognize that the change operation has been performed by checking the status of the notification device such as the warning light, and go to the POS terminal 10 where the change operation has been performed.

[0223] According to this, even if the POS terminal 10 has a system configuration that does not use the clerk terminal 40 , it is possible to achieve the same effects as those of the above-described embodiment.

[0224] (Variant Example 7)

[0225] In the above embodiment, the camera device 20 performs code recognition, but the present invention is not limited thereto. For example, the code recognition may be performed by the POS terminal 10 or the edge device 30 .

[0226] For example, when code recognition is performed by the POS terminal 10, the image transmission unit 202 of the camera 20 transmits (outputs) the captured image to the POS terminal 10. In addition, the POS terminal 10 includes a functional configuration of a code recognition unit 203, and the code recognition unit 203 performs code recognition on the image transmitted from the camera 20.

[0227] Furthermore, for example, in the case where code recognition is performed by the edge device 30, the edge device 30 includes a functional configuration of a code recognition unit 203, by which code recognition is performed on an image sent from the camera 20. Also, in this case, the recognition result output unit 303 of the edge device 30 outputs recognition results of code recognition and package recognition in order, starting from the recognition result recognized first.

[0228] Furthermore, in the above-mentioned embodiment, the edge device 30 performs package recognition, but the present invention is not limited thereto. For example, the package recognition may be performed by the POS terminal 10 or the camera device 20 .

[0229] For example, when the POS terminal 10 performs package recognition, the image transmission unit 202 of the camera device 20 transmits (outputs) the captured image to the POS terminal 10. In addition, the POS terminal 10 includes a functional structure of a package recognition unit 302, and the package recognition unit 302 performs package recognition on the image sent from the camera device 20.

[0230] In addition, for example, when package recognition is performed by the camera device 20, the camera device 20 includes a functional structure of a package recognition unit 302, and package recognition is performed on the image obtained by the shooting by the package recognition unit 302. In addition, in this case, the recognition result output unit 204 of the camera device 20 outputs the recognition results of the code recognition and the package recognition in order from the first recognition result.

[0231] In the above embodiment, the edge device 30 is described as an independent device, but the invention is not limited thereto. For example, the edge device 30 may be implemented by a cloud (cloud computing) composed of a plurality of information processing devices. In addition, the store server 50 may have the function of the edge device 30 .

[0232] (Variant 8)

[0233] In the above embodiment, the POS terminal 10 is described as a device operated by the customer himself. However, the POS terminal 10 is not limited to this, and may be a device operated by a clerk of the store T. In this case, the camera 20 captures the product based on the registration operation performed by the clerk.

[0234] (Variant 9)

[0235] In the above embodiment, the POS terminal 10 is described as having a configuration of one camera device 20 (or camera unit 25). However, the present invention is not limited thereto, and may also include a configuration of multiple camera devices 20 (or camera units 25). In this case, code recognition may be performed by each camera device of the camera device 20, or by one camera device 20. In addition, the multiple camera devices 20 (or camera units 25) are preferably configured to photograph the product from different photographing positions. In this way, the product can be photographed from multiple angles, thereby improving the recognition speed and recognition rate.

[0236] In addition, the program executed by each device of the above-mentioned embodiment (and modified example) is provided in a state installed in a ROM, a storage unit, etc. The program executed by each device of the above-mentioned embodiment can be configured to be stored in a computer-readable storage medium such as a CD-ROM, a floppy disk (FD), a CD-R, or a DVD (Digital Versatile Disk) in an installable or executable form and provided.

[0237] Furthermore, the programs executed by the devices of the above-mentioned embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. In addition, the programs executed by the devices of the above-mentioned embodiments may be provided or distributed via a network such as the Internet.

[0238] Although the embodiments of the present invention have been described above, the embodiments are presented only as examples and are not intended to limit the scope of the invention. These novel embodiments and their variations can be implemented in various other forms, and can be omitted, replaced, changed and combined in various ways without departing from the scope of the invention. These embodiments and their variations are included in the scope and spirit of the invention, and are included in the invention described in the claims and their equivalents.

Claims

1. A commodity sales data processing device, characterized in that: include: An acquisition unit that acquires a recognition result of a recognition process for recognizing the commodity based on an image in which the commodity is photographed; a registration unit that performs a registration process of registering the commodity identified by the identification process as a purchase object based on the identification result obtained by the acquisition unit; A display unit that displays a screen showing each of the registered commodities; a first receiving unit that receives a change operation on a registered commodity and adds information corresponding to the content of the change operation to the commodity to be changed shown on the screen; a second receiving unit for receiving an end operation instructing to end the registration process; as well as The notification unit performs processing for notifying the store clerk that the change operation performed before the end operation has been performed when the end operation is performed.

2. The commodity sales data processing device according to claim 1, wherein: Also includes: The third receiving section receives the confirmation operation performed by the store clerk; as well as The reflecting unit reflects the content of the change operation on the registered commodity when the confirmation operation is performed.

3. The commodity sales data processing device according to claim 2, wherein: The notification unit notifies the terminal device operated by the store clerk. The third receiving unit receives the confirmation operation from the terminal device.

4. The commodity sales data processing device according to claim 3, wherein: The notification unit notifies the terminal device of information indicating the states of the product that is the subject of the change operation before and after the change operation.

5. The commodity sales data processing device according to claim 2, wherein: Also includes: a settlement unit, performing settlement processing of the registered goods according to the end operation; as well as The suppression unit suppresses the settlement process until the confirmation operation is performed.

6. The commodity sales data processing device according to claim 3, wherein: The notification unit notifies the terminal device of image data of a screen in which information corresponding to the content of the change operation generated by the first receiving unit is added to the commodity to be changed.

7. The commodity sales data processing device according to claim 3, wherein: The notification unit does not notify the terminal device when the content of change of all the products that are the target of the change operation is an increase in the number of instructions.

8. The commodity sales data processing device according to claim 3, wherein: The notification unit does not notify the terminal device when a change operation is performed on a registered commodity whose similarity included in the recognition result acquired by the acquisition unit is smaller than a threshold value.

9. The commodity sales data processing device according to claim 3, wherein: Also includes: Surveillance cameras, including The notification unit notifies the terminal device of the surveillance image captured by the surveillance camera using the receipt of a change operation of a registered commodity as a trigger.

10. A storage medium storing a program for causing a computer of a commodity sales data processing device to function as the following units: An acquisition unit that acquires a recognition result of a recognition process for recognizing the commodity based on an image in which the commodity is photographed; a registration unit that performs a registration process of registering the commodity identified by the identification process as a purchase object based on the identification result obtained by the acquisition unit; A display unit that displays a screen showing each of the registered commodities; a first receiving unit that receives a change operation on a registered commodity and reflects the content of the change operation on the commodity to be changed shown on the screen; a second receiving unit for receiving an end operation instructing to end the registration process; as well as The notification unit, when the end operation is performed, performs processing for notifying the store clerk that the change operation performed before the end operation has been performed.

Images