Commodity sales data processing device and program

Abstract

To provide a commodity sales data processing device and a program that can reduce a burden on a shop clerk related to an operation of changing a commodity.SOLUTION: A commodity sales data processing device of an embodiment includes: acquisition means for acquiring a recognition result of recognition processing of recognizing a commodity based on an image captured by imaging the commodity; registration means for, based on the recognition result acquired by the acquisition means, executing registration processing of registering commodities recognized by the recognition processing as a purchase target; display means for displaying a screen that represents each of the registered commodities; first accepting means for accepting a changing operation for a registered commodity and reflecting a content of the changing operation on a commodity targeted for changing, displayed on the screen; second accepting means for accepting a termination operation of instructing termination of the registration processing; and notification means for, when the termination operation is performed, notifying a shop clerk of the fact that the changing operation performed before the termination operation has been performed.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product sales data processing device and a program. [Background technology]

[0002] Conventionally, there exist self-service POS terminals in stores such as shopping centers, mass retailers, and department stores, where customers themselves perform product registration processing operations and payment processing operations for products sold in the store. Some of these POS terminals capture images of products purchased by customers, recognize the products from the captured images, and perform product registration processing based on the recognition results.

[0003] Incidentally, in the above-mentioned self-service POS terminal, customers can also change registered products. For example, if an incorrect product has been recognized from an image and registered, the customer can delete the product themselves. However, since there are cases where changes are made for improper purposes, such as deleting a registered product and then taking the product home as is, a store clerk must check the changes each time a change is made. Also, some stores prohibit customers from making changes and have store clerks make the changes each time.

[0004] However, in the conventional operating method, it is necessary to call a store clerk every time a change operation is performed, which places a heavy burden on the store clerk and leaves room for improvement. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a product sales data processing device and a program that can reduce the burden on a store clerk involved in changing products. [Means for solving the problem]

[0006] The product sales data processing device of the embodiment comprises an acquisition means for acquiring a recognition result of a recognition process that recognizes a product based on an image of the product, a registration means for executing a registration process that registers the product recognized in the recognition process as a purchase target based on the recognition result acquired by the acquisition means, a display means for displaying a screen representing each of the registered products, a first acceptance means for accepting a change operation for the registered products and reflecting the content of the change operation in the product to be changed displayed on the screen, a second acceptance means for accepting an end operation that instructs the end of the registration process, and a notification means for performing a process to notify a store clerk that the change operation performed up to the end operation has been performed when the end operation is performed. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a product sales data processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of package recognition according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the POS terminal according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data configuration of the product master according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a hardware configuration of the imaging device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of the edge device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of the store clerk terminal according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a hardware configuration of a store server according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of functional configurations of the POS terminal, the imaging device, and the edge device according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a product selection screen displayed on the POS terminal of the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a product registration screen displayed on the POS terminal of the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a product registration screen displayed on the POS terminal of the embodiment. [Figure 13] FIG. 13 is a sequence diagram illustrating an example of a flow of processing executed in the product sales data processing system according to the embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an example of a process performed by the POS terminal of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, a product sales data processing device in which a customer himself performs the registration operation and the payment operation of a product to be purchased will be described as an example. Note that the present invention is not limited to the embodiment described below.

[0009] Fig. 1 is a diagram showing an example of a product sales data processing system according to an embodiment. In Fig. 1, the product sales data processing system 1 is installed in a store T such as a supermarket, a mass retailer, a convenience store, or a specialty store. The product sales data processing system 1 includes one or more POS (Point of Sales) terminals 10 (product sales data processing devices), an imaging device 20, an edge device 30, a store clerk terminal 40, and a store server 50.

[0010] The POS terminal 10, the edge device 30, the clerk terminal 40, and the store server 50 are connected to a communication line L such as a LAN (Local Area Network) that allows mutual notification. The imaging device 20 is connected to each POS terminal 10 by, for example, a USB (Universal Serial Bus) cable U. The imaging device 20 is also connected to the communication line L. The imaging device 20 may be configured to be connected to the communication line L via the POS terminal 10.

[0011] The imaging device 20 is a device equipped with an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imaging device 20 is provided, for example, facing the customer who operates the POS terminal 10, and captures images of products purchased by the customer. Specifically, in this embodiment, in order to perform registration processing, the customer who operates the POS terminal 10 holds each of the products he or she wishes to purchase over the imaging device 20, causing the imaging device 20 to capture an image of the product.

[0012] When the imaging device 20 captures an image (hereinafter also referred to as a captured image), the imaging device 20 outputs the captured image to the edge device 30. Specifically, the imaging device 20 outputs an image captured at a predetermined frame rate to the edge device 30.

[0013] The imaging device 20 also performs code recognition on the image obtained by imaging, attempting to detect symbols such as barcodes and two-dimensional codes from the image. In code recognition, symbols such as barcodes and two-dimensional codes attached to products are detected from the image, and the detected symbols are decoded to obtain decoded information, which is output to the POS terminal 10. The decoded information includes product identification information that identifies the product to which the symbol is attached, so that the imaged product can be identified based on the product identification information. In this embodiment, a product code capable of identifying each product is used as the product identification information, but information other than the product code may be used as long as it is capable of identifying the product.

[0014] The imaging device 20 may extract images that include or may include a commodity from the images obtained by imaging, and output the images to the edge device 30 or use known techniques for symbol extraction. In this case, the image extraction method is not particularly limited, and known techniques can be used. For example, the imaging device 20 may detect the movement of an object included in an image from the difference between multiple temporally consecutive frames, and extract an image in which the movement is detected, and use the extracted image as a processing target.

[0015] The edge device 30 is an information processing device that performs predetermined processing such as image processing, image recognition, etc. Specifically, the edge device 30 performs package recognition to recognize a product from an image input from the imaging device 20.

[0016] Package recognition is a technology for identifying a product imaged by the imaging device 20 based on the external characteristics of the product depicted in the image. Here, the external characteristics of the product include, for example, the design, pattern, color, texture, shape, etc. of the package.

[0017] For package recognition, known technologies such as general object recognition and image recognition using a model such as deep learning can be used. In package recognition, based on a feature amount indicating the external characteristics of a product extracted from an image, one product whose similarity (also called likelihood) is equal to or exceeds a threshold is identified as the product that is the subject of the image, among products similar to the feature amount.

[0018] Fig. 2 is a diagram for explaining an example of package recognition. As shown in Fig. 2, in package recognition, a product is identified from an image captured by the imaging device 20 using three types of trained models or dictionary data (hereinafter collectively referred to as dictionaries), namely, an object detection model 61, a feature extraction model 62, and a matching dictionary 63.

[0019] The object detection model 61 is a dictionary for detecting an area of ​​an object appearing in a captured image. When a captured image captured by the imaging device 20 is input, the object detection model 61 detects an area showing a commodity (hereinafter also referred to as an object image) from the captured image and outputs the area.

[0020] The feature extraction model 62 is a dictionary for extracting feature amounts indicating the external characteristics of a product from the product depicted in the area detected by the object detection model 61. When an object image or a captured image including an object image is input, the feature extraction model 62 extracts and outputs feature amounts indicating the external characteristics of the product depicted in the object image.

[0021] The matching dictionary 63 is a dictionary for identifying products similar to the features of a product based on the features of the product. For example, the matching dictionary 63 stores, for multiple products sold at the store T, the features indicating the characteristics of the product in association with the product code that identifies each product. The matching dictionary 63 also compares the input features with the stored features to calculate a similarity that quantitatively indicates how similar the many stored products are to the imaged product. The higher the similarity of a product, the higher the degree of similarity of the product to the imaged product (i.e., the probability that it is the imaged product). As an example, a product with a similarity of 95% or more is extremely likely to be the same product as the imaged product.

[0022] The edge device 30 uses an object detection model 61 to detect an area in which a product is captured from an image captured by the imaging device 20. Next, the edge device 30 uses a feature extraction model 62 to extract features that quantitatively represent the external features of the product captured in that area (object image) using a multidimensional vector or the like. Then, the edge device 30 uses a comparison dictionary 63 to compare the extracted features with features of many products stored in the dictionary to calculate a similarity, and recognizes the product according to the similarity.

[0023] Specifically, when there is one product whose similarity is equal to or greater than a threshold (e.g., 95% or greater), the edge device 30 specifies the product whose similarity is equal to or greater than the threshold as a product captured by the imaging device 20. When there is no product whose similarity is equal to or greater than the threshold (e.g., when the similarity of all products is less than 95%), the edge device 30 does not specify one product, but specifies multiple products whose similarity is high as candidate products. For example, when the similarity of all products is less than the threshold, the edge device 30 extracts a predetermined number (e.g., four) of candidate products from those whose similarity is high. Then, the edge device 30 transmits the product code of the specified product or the candidate product as the recognition result of the package recognition to the imaging device 20 that transmitted the captured image. When the edge device 30 recognizes multiple candidate products, it transmits the similarity calculated for the candidate product together with the recognition result. When the edge device 30 can uniquely identify a product, it may transmit the similarity together with the recognition result. In addition, the edge device 30 has a lower threshold for similarity determination, and when the similarity falls below the lower threshold, the edge device 30 excludes the product having the similarity from the candidate products.

[0024] When the imaging device 20 receives the recognition result of the package recognition from the edge device 30, it outputs the recognition result to the POS terminal 10. The imaging device 20 outputs the product code of the product identified by its own code recognition and the product code received from the edge device 30, i.e., the product code identified by the package recognition, to the POS terminal 10 in a distinguishable state. For example, the imaging device 20 outputs a flag value that can distinguish between the code recognition and the package recognition together, thereby outputting the product code of the product identified by the code recognition and the product code of the product identified by the package recognition in a distinguishable state.

[0025] In this embodiment, the edge device 30 inputs the package recognition result to the POS terminal 10 via the imaging device 20, but the present invention is not limited to this, and the recognition result may be input directly to the POS terminal 10. In this case, the edge device 30 stores, for example, setting information that associates the IP addresses of the paired POS terminal 10 and imaging device 20, and transmits the recognition result of the product or candidate product identified from the captured image to the POS terminal 10 corresponding to the imaging device 20 that transmitted the captured image.

[0026] The clerk terminal 40 is a terminal device operated by a clerk of the store T. The clerk terminal 40 may be, for example, a portable terminal device, or an attendant terminal or the like that is placed in the same area as the POS terminal 10. The clerk terminal 40 is capable of checking the status of the POS terminal 10.

[0027] The store server 50 is installed, for example, in the back yard of the store T. The store server 50 stores a product master (not shown) that associates the product name, price, product image, etc. of each product sold at the store T with the product code of the product. The store server 50 also provides the product master to each of the POS terminals 10. For example, the store server 50 transmits the product master to each of the POS terminals 10, and stores the product master locally in the POS terminals 10. For example, the store server 50 also stores the product master so that each of the POS terminals 10 can refer to it.

[0028] In addition, the store server 50 collects the sales data received from the POS terminal 10. The store server 50 manages the sales of products in the store T based on the collected sales data.

[0029] The POS terminal 10 cooperates with other devices to perform product registration processing and payment processing for products sold in the store T. Specifically, the POS terminal 10 is operated by a customer and executes a registration processing to register, as a purchase target, a product identified based on an image captured by the imaging device 20 when the customer holds the product over the imaging device 20. In addition, in response to an operation to end the registration processing, the POS terminal 10 executes a payment processing to settle the price of the products registered up to that point with the amount paid by the customer.

[0030] One or more POS terminals 10 (three in FIG. 1) are installed in the checkout area of ​​store T, for example. The POS terminal 10 is a product sales data processing device that is operated by the customer himself. When a customer performs a registration operation by holding the product to be purchased over the imaging device 20, the POS terminal 10 executes a product registration process for the product purchased by the customer, based on the product code of the product identified by the above-mentioned code recognition or package recognition. The product registration process refers to a process of reading the identified product code and product information (product name, product price, etc.) of the product corresponding to the product code from the product master and storing them in a storage medium.

[0031] In this embodiment, the storage medium in which the product registration process is stored is provided in the POS terminal 10, but the present invention is not limited to this and may be provided in an external device such as the store server 50, for example.

[0032] In addition, the POS terminal 10 executes a payment process for the products registered in the product registration process in response to an operation to instruct the end of the product registration process or an operation to instruct the start of a payment process. The payment process refers to a closing process for the transaction based on the product information stored in the registered product storage unit 131 by the product registration process, specifically, displaying the total amount on the display unit 16, payment processing by media such as cash, credit card, electronic money, etc., a process of calculating the change based on the deposit in the case of cash payment and displaying it on the display unit 16, and a process of issuing a receipt on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the products for which payment has been processed are printed from the printer 18 described later.

[0033] The POS terminal 10 also transmits product information and transaction information (collectively referred to as sales data) of the products for which payment has been processed to the store server 50. In this embodiment, the timing of transmission of sales data from the POS terminal 10 to the store server 50 is when the payment process is completed, but this is not limited thereto, and for example, the sales data for one day may be transmitted collectively when the store T closes.

[0034] As described above, in the product sales data processing system 1 of this embodiment, the product imaged by the imaging device 20 is identified by using both code recognition and package recognition. When each of the POS terminals 10 can identify the imaged product by the recognition result acquired first among the recognition results of the code recognition and package recognition performed based on the image captured by the imaging device 20, the product registration process is performed using the product code of the product.

[0035] Next, the configuration of each device included in the product sales data processing system 1 will be described.

[0036] Fig. 3 is a diagram showing an example of a hardware configuration of the POS terminal 10. As shown in Fig. 3, the POS terminal 10 includes a CPU (Central Processing Unit) 11, which is an example of a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory unit 14, and the like.

[0037] The CPU 11 has one or more processing circuits and controls the overall operation of the POS terminal 10. The ROM 12 stores various programs that the CPU 11 can execute. The RAM 13 is a work area for temporarily storing programs and various data. For example, the RAM 13 stores the recognition results of code recognition or package recognition. The RAM 13 also includes a registered product storage unit 131 for storing product codes, points, product information, etc. of products that have been registered.

[0038] The memory unit 14 is composed of a non-volatile memory such as a hard disk drive (HDD) or a flash memory that retains stored information even when the power is turned off. The memory unit 14 stores various programs that the CPU 11 can execute, setting information, and the like. For example, the memory unit 14 stores a control program 141 that the CPU 11 can execute, a product master 142, and the like. The product master 142 stores product information, etc., of a product corresponding to a product code in association with the product code of each product sold at the store T.

[0039] Fig. 4 is a diagram showing an example of the data configuration of the product master 142. As shown in Fig. 4, the product master 142 stores product information and a symbol flag of a product corresponding to the product code in association with the product code of each product.

[0040] The product information includes various information related to the product. For example, the product information includes the product name, price, and product image. The product image is image data representing the appearance of the product. The symbol flag is binary flag information indicating whether or not a symbol is attached to the product.

[0041] Products with symbols are subject to code recognition and may also be subject to package recognition. On the other hand, products without symbols are not subject to code recognition and can only be recognized by package recognition. Examples of products without symbols include fresh foods and prepared foods produced and sold in stores.

[0042] It should be noted that the data configuration of the product master 142 is not limited to the example in Fig. 4. For example, the product master 142 may have a data configuration in which the symbol flag is removed.

[0043] 3, the CPU 11, ROM 12, RAM 13, and memory unit 14 are connected to one another via a bus or the like. Here, the CPU 11, ROM 12, and RAM 13 constitute a control unit 100 of the POS terminal 10. The control unit 100 realizes the functions of the POS terminal 10 by the CPU 11 operating in accordance with a control program 141 stored in the ROM 12 and memory unit 14 and expanded in the RAM 13.

[0044] In addition, the control unit 100 is connected to an operation unit 15, a display unit 16, a card reader 17, a printer 18, etc. via a bus or the like.

[0045] The operation unit 15 has various operation keys that allow the operator to input operations. The operation unit 15 has a closing key as an example of an operation key. The closing key is an operation key that is operated when terminating the product registration process in the POS terminal 10 and proceeding to the payment process. The display unit 16 is, for example, a liquid crystal display, and displays information to the customer who is the operator of the POS terminal 10. Under the control of the control unit 100, the display unit 16 displays, for example, product information (product name, product price, etc.) of the product that has been registered, and accounting information related to the payment process. The operation unit 15 may be a touch panel provided on the display unit 16.

[0046] The card reader 17 is a reading device for reading card information related to a payment from a card, for example, a credit card, a debit card, an electronic money card, etc. The printer 18 is a printing device, for example, a thermal printer, etc. Under the control of the control unit 100, the printer 18 issues a receipt on which product information related to the product for which payment has been processed, payment information, etc. are printed.

[0047] The control unit 100 is also connected to a communication unit 19 via a bus or the like. The communication unit 19 is a variety of communication interfaces for connecting to external devices. The communication unit 19 is communicatively connected to the imaging device 20 via a cable U. The communication unit 19 is also communicatively connected to the edge device 30, the clerk terminal 40, and the store server 50 via a communication line L.

[0048] Next, a description will be given of a hardware configuration of the imaging device 20. Fig. 5 is a diagram showing an example of a hardware configuration of the imaging device 20. As shown in Fig. 5, the imaging device 20 includes a CPU 21, which is an example of a processor, a ROM 22, a RAM 23, a memory unit 24, and the like.

[0049] The CPU 21 has one or more processing circuits and comprehensively controls the operation of the imaging device 20. The ROM 22 stores various programs that the CPU 21 can execute. The RAM 23 is a work area for temporarily storing programs and various data. For example, the RAM 23 stores images captured by the imaging unit 25 (described later), intermediate data of code recognition, processing results (recognition results), etc.

[0050] The memory unit 24 is composed of a non-volatile memory such as a HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 24 stores various programs and setting information that can be executed by the CPU 21. For example, the memory unit 24 stores a control program 241 that can be executed by the CPU 21.

[0051] The CPU 21, the ROM 22, the RAM 23, and the memory unit 24 are connected to one another via a bus or the like. Here, the CPU 21, the ROM 22, and the RAM 23 constitute a control unit 200 of the imaging device 20. The control unit 200 realizes the functions of the imaging device 20 by the CPU 21 operating in accordance with a control program 241 stored in the ROM 22 or the memory unit 24 and expanded in the RAM 23.

[0052] The control unit 200 is also connected to an imaging unit 25 and a communication unit 26 via a bus or the like. The imaging unit 25 is a camera module equipped with an imaging element such as a CCD. The communication unit 26 is various communication interfaces for connecting to external devices. The communication unit 26 is communicatively connected to the POS terminal 10 via a cable U. The communication unit 26 is also communicatively connected to an edge device 30 or the like via a communication line L.

[0053] Next, a description will be given of a hardware configuration of the edge device 30. Fig. 6 is a diagram illustrating an example of a hardware configuration of the edge device 30. As illustrated in Fig. 6, the edge device 30 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like.

[0054] The CPU 31 has one or more processing circuits and comprehensively controls the operation of the edge device 30. The ROM 32 stores various programs that the CPU 31 can execute. The RAM 33 is a work area for temporarily storing programs and various data. For example, the RAM 33 stores images transmitted from the imaging device 20, intermediate data of package recognition, processing results (recognition results), etc.

[0055] The memory unit 34 is composed of a non-volatile memory such as an HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 34 stores various programs and setting information that can be executed by the CPU 31. For example, the memory unit 34 stores a control program 341 that can be executed by the CPU 31. The memory unit 34 also stores the above-mentioned object detection model 61, feature extraction model 62, matching dictionary 63, and the like, as well as threshold setting information 342 that sets a threshold that serves as an index for uniquely identifying a product based on the similarity.

[0056] The CPU 31, the ROM 32, the RAM 33, and the memory unit 34 are connected to one another via a bus or the like. Here, the CPU 31, the ROM 32, and the RAM 33 constitute a control unit 300 of the edge device 30. The control unit 300 realizes the functions of the edge device 30 by the CPU 31 operating in accordance with a control program 341 stored in the ROM 32 and the memory unit 34 and loaded in the RAM 33.

[0057] The control unit 300 is also connected to a communication unit 35 via a bus or the like. The communication unit 35 is a variety of communication interfaces for connecting to external devices. The communication unit 35 is connected to the POS terminal 10, the imaging device 20, the clerk terminal 40, the store server 50, and the like via a communication line L so as to be able to communicate with them.

[0058] Next, a description will be given of the hardware configuration of the clerk terminal 40. Fig. 7 is a diagram showing an example of the hardware configuration of the clerk terminal 40. As shown in Fig. 7, the clerk terminal 40 includes a CPU 41, which is an example of a processor, a ROM 42, a RAM 43, a memory unit 44, and the like.

[0059] The CPU 41 has one or more processing circuits and comprehensively controls the operation of the store clerk terminal 40. The ROM 42 stores various programs that can be executed by the CPU 41. The RAM 43 is a work area for temporarily storing programs and various data.

[0060] The memory unit 44 is composed of a non-volatile memory such as a HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 44 stores various programs and setting information that can be executed by the CPU 41. For example, the memory unit 44 stores a control program 441 that can be executed by the CPU 41.

[0061] The CPU 41, ROM 42, RAM 43, and memory unit 44 are connected to one another via a bus or the like. Here, the CPU 41, ROM 42, and RAM 43 constitute a control unit 400 of the store clerk terminal 40. The control unit 400 realizes the functions of the store clerk terminal 40 by the CPU 41 operating in accordance with a control program 441 stored in the ROM 42 and memory unit 44 and expanded in the RAM 43.

[0062] In addition, the control unit 400 is connected to an operation unit 45, a display unit 46, and the like via a bus or the like.

[0063] The operation unit 45 has various operation keys for an operator to perform operation input. The display unit 46 is, for example, a liquid crystal display, and displays information to a store clerk who is the operator of the store clerk terminal 40. The operation unit 45 may be a touch panel provided on the display unit 46.

[0064] The control unit 400 is connected to a communication unit 47 via a bus or the like. The communication unit 47 is a variety of communication interfaces for connecting to external devices. The communication unit 47 is connected to the POS terminal 10, the imaging device 20, the edge device 30, and the store server 50 via a communication line L so as to be able to communicate with them.

[0065] Next, a description will be given of the hardware configuration of the store server 50. Fig. 8 is a diagram showing an example of the hardware configuration of the store server 50. As shown in Fig. 8, the store server 50 includes a CPU 51, which is an example of a processor, a ROM 52, a RAM 53, a memory unit 54, and the like.

[0066] The CPU 51 has one or more processing circuits and generally controls the operation of the store server 50. The ROM 52 stores various programs that the CPU 51 can execute. The RAM 53 is a work area for temporarily storing programs and various data. For example, the RAM 53 temporarily stores accounting data transmitted from the store server 50.

[0067] The memory unit 54 is composed of a non-volatile memory such as a HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 54 stores various programs that the CPU 51 can execute, setting information, and the like. For example, the memory unit 54 stores a control program 541 that the CPU 51 can execute. The memory unit 54 also stores a product master 542, a transaction history DB (database) 543 for storing and managing accounting data, and the like. The data structure of the product master 542 is the same as that of the product master 142. As an example, the product master 542 is provided (distributed) to each of the POS terminals 10, and the product master 542 is stored and used as the product master 142 in each of the POS terminals 10.

[0068] The CPU 51, ROM 52, RAM 53, and memory unit 54 are connected to one another via a bus or the like. Here, the CPU 51, ROM 52, and RAM 53 constitute a control unit 500 of the store server 50. The control unit 500 realizes the functions of the store server 50 by the CPU 51 operating in accordance with a control program 541 stored in the ROM 52 and memory unit 54 and loaded into the RAM 53.

[0069] The control unit 500 is connected to a communication unit 55 via a bus or the like. The communication unit 55 is a variety of communication interfaces for connecting to external devices. The communication unit 55 is connected to the POS terminal 10 or the like via a communication line L so as to be able to communicate with the POS terminal 10 or the like.

[0070] Next, a description will be given of the functional configurations of the above-mentioned POS terminal 10, imaging device 20, edge device 30, and clerk terminal 40. Fig. 9 is a diagram showing an example of the functional configurations of the POS terminal 10, imaging device 20, edge device 30, and clerk terminal 40.

[0071] As shown in FIG. 9, the imaging device 20 includes an image acquisition unit 201, an image transmission unit 202, a code recognition unit 203, and a recognition result output unit 204 as functional components.

[0072] The functional configuration of the imaging device 20 is a software configuration realized by the processor of the imaging device 20 operating in accordance with the control program 241 stored in the ROM 22 or the memory unit 24, but is not limited to this. For example, a part or all of the functional configuration of the imaging device 20 may be realized by hardware such as a dedicated circuit.

[0073] The image acquisition unit 201 acquires images captured by the imaging unit 25. The image acquisition unit 201 may extract and acquire images in which a commodity is shown or images in which a commodity may be shown, from images captured by the imaging unit 25, using a known technique such as motion detection.

[0074] The image transmission unit 202 transmits the image acquired by the image acquisition unit 201 to the edge device 30. The code recognition unit 203 attempts to detect a symbol such as a barcode from the image acquired by the image acquisition unit 201 by executing code recognition on the image acquired by the image acquisition unit 201. When the code recognition unit 203 detects a symbol from the image, it decodes the symbol and acquires decoded information including a product code as the recognition result of the code recognition.

[0075] The recognition result output unit 204 outputs the recognition result obtained by the code recognition by the code recognition unit 203 to the POS terminal 10. In addition, the recognition result output unit 204 outputs the recognition result of the package recognition received from the edge device 30 to the POS terminal 10. Note that the recognition result output unit 204 may output the recognition results of the code recognition and the package recognition together with a flag value indicating the recognition method, so as to be identifiable.

[0076] As shown in FIG. 9, the edge device 30 includes an image acquisition unit 301, a package recognition unit 302, and a recognition result output unit 303 as functional components.

[0077] The functional configuration of the edge device 30 is a software configuration realized by the processor of the edge device 30 operating in accordance with the control program 341 stored in the ROM 32 or the memory unit 34, but is not limited to this. For example, some or all of the functional configuration of the edge device 30 may be realized by hardware such as a dedicated circuit.

[0078] The image acquisition unit 301 acquires images transmitted from each of the imaging devices 20 via the communication unit 35. The package recognition unit 302 performs the above-mentioned package recognition on the image acquired by the image acquisition unit 301, thereby attempting to detect an object image from the image and extract features. When the package recognition unit 302 confirms that there is one product in the matching dictionary 63 whose similarity is equal to or greater than the threshold based on the features extracted from the image, it outputs the product code of the product as a recognition result. Furthermore, when there is no product whose similarity is equal to or greater than the threshold, the package recognition unit 302 extracts a predetermined number of products with high similarity from the matching dictionary 63, and outputs information associating the similarity of the product with the product code of each extracted product as a recognition result.

[0079] If there are a plurality of products whose similarity is equal to or greater than the threshold, the package recognition unit 302 outputs information associating the product codes of those products with the similarity of the products as the recognition result.

[0080] The recognition result output unit 303 outputs the recognition result of the package recognition executed on the image to an external device related to the transmission source of the image. Specifically, the recognition result output unit 303 transmits the recognition result of the package recognition executed on the image to the imaging device 20 that transmitted the image via the communication unit 35. Note that the recognition result output unit 303 may transmit the recognition result to the POS terminal 10 corresponding to the imaging device 20 that transmitted the image.

[0081] As shown in FIG. 9, the POS terminal 10 has a recognition result acquisition unit 101, a product registration processing unit 102, a change operation receiving unit 103, a notification processing unit 104, a change processing unit 105, and a payment processing unit 106 as functional components.

[0082] The functional configuration of the POS terminal 10 is a software configuration realized by the processor of the POS terminal 10 operating in accordance with the control program 141 stored in the ROM 12 or the memory unit 14, but is not limited to this. For example, a part or all of the functional configuration of the POS terminal 10 may be realized by hardware such as a dedicated circuit.

[0083] The recognition result acquisition unit 101 is an example of an acquisition means. The recognition result acquisition unit 101 acquires the recognition results of the code recognition and package recognition for the POS terminal 10 itself, which are performed by the imaging device 20 and the edge device 30. Specifically, the recognition result acquisition unit 101 acquires the recognition results of the code recognition and package recognition transmitted from the imaging device 20 via the communication unit 19. Note that in the case of a configuration in which the recognition result of the package recognition is directly transmitted from the edge device 30 (recognition result output unit 303) to the POS terminal 10 (see the dashed line in the figure), the recognition result acquisition unit 101 may acquire the recognition result of the package recognition from the edge device 30.

[0084] The product registration processing unit 102 is an example of a registration means and a display means. The product registration processing unit 102 executes a product registration process for a product purchased by a customer, based on the recognition result acquired by the recognition result acquisition unit 101. In other words, the product registration processing unit 102 executes a product registration process based on the recognition result of the code recognition of the image of the product and the recognition result of the package recognition.

[0085] Specifically, when a product is uniquely identified by the recognition result of code recognition or package recognition, that is, when the recognition result includes the product code of a product, the product registration processing unit 102 executes product registration processing for the product. Note that the product registration processing unit 102 speeds up the processing by executing the product registration processing using the product code of the recognition result obtained first from the code recognition or package recognition.

[0086] Furthermore, when a package recognition result including multiple product codes and their similarities is obtained without obtaining a code recognition result, that is, when a recognition result including multiple candidate products is obtained, the product registration processing unit 102 performs product registration processing based on the recognition result. Specifically, when multiple candidate products are recognized, the product registration processing unit 102 causes the display unit 16 to display a screen on which the candidate products can be selected (hereinafter, also referred to as a product selection screen).

[0087] Fig. 10 is a diagram showing an example of a product selection screen A displayed on the POS terminal 10. As shown in Fig. 10, the product selection screen A has a display area Aa for displaying products corresponding to each of the product codes included in the recognition result as candidate products.

[0088] In the display area Aa, the product name, price, image, etc. of the candidate product are displayed based on the product information of the candidate product. In FIG. 10, four candidate products are shown, and a display area Aa is provided for each candidate product. Note that, although the arrangement position of the candidate products on the product selection screen A is not particularly important, it is preferable to arrange them according to the degree of similarity. For example, the products may be arranged in the order of the highest degree of similarity, from top left → top right → bottom left → bottom right.

[0089] The display area Aa also functions as an operator for selecting a product to be registered. When a customer selects one of the display areas Aa, the product registration processing unit 102 executes a product registration process based on the product code corresponding to the candidate product in the selected display area Aa. Then, the product registration processing unit 102 erases the product selection screen A.

[0090] Furthermore, the product selection screen A has a back button Ab. When the product registration processing unit 102 accepts the operation of the back button Ab, it erases the product selection screen A and waits until a new recognition result is acquired.

[0091] In addition, when the product registration processing unit 102 executes the product registration processing, it causes the display unit 16 to display a screen (also called a product registration screen) that shows product information such as the product names and prices of the registered products in the order in which the products are registered.

[0092] Fig. 11 is a diagram showing an example of a product registration screen displayed on the POS terminal 10. As shown in Fig. 11, the product registration screen B has a product display area Ba and a total display area Bb.

[0093] The product display area Ba displays information such as the product name, points, and price (unit price) of each product (hereinafter also referred to as registered product) registered in the product registration process in association with an ID indicating the registration order. The total display area Bb displays the points and total purchase amount of the registered products. Specifically, the product registration processing unit 102 calculates the total points and total amount based on the price and points of the registered products, and displays them in the total display area Bb.

[0094] In addition, the product registration screen B is capable of accepting changes to registered products. Specifically, the change operation accepting unit 103 accepts changes to registered products via the operation unit 15. Here, the change operation to registered products means an operation to delete a registered product, an operation to change (increase or decrease) the registered points, etc.

[0095] The change operation receiving unit 103 is an example of a first receiving means. When the change operation receiving unit 103 receives a change operation, the change operation is reflected in the product registration screen B. In addition, the change operation receiving unit 103 generates change content information that associates the ID of the registered product that is the target of the change operation with the content of the change operation, and stores the change content information in the RAM 13 or the like.

[0096] Here, the change operation receiving unit 103 changes the display of the product registration screen B according to the content of the change operation, without changing the actual registered product storage unit 131 registered in the product registration process. For example, when a deletion instruction is given for the registered product with ID "1" shown in FIG. 11, the change operation receiving unit 103 displays the registered product with ID "1" with a strikethrough added, as shown in FIG. 12. Also, for example, when an instruction is given to decrease the score of the registered product with ID "4" shown in FIG. 11 by 1, the score of ID "4" is displayed with 1 subtracted, as shown in FIG. 12. Note that when an instruction is given to increase or decrease the score, it is preferable to highlight the changed score so that it is easy to identify that the product has been subject to a change operation.

[0097] Then, the change operation receiving unit 103 updates and displays the total points and the total amount in the total display area Bb according to the content of the change operation. This allows the operator of the POS terminal 10 to visually confirm the content of the change operation that he or she has performed.

[0098] In addition, a checkout button Bc for instructing the user to end the product registration process (or start the payment process) is provided on the product registration screen B. The checkout button Bc is an example of a second reception means.

[0099] In a self-service product sales data processing device in which the customer himself performs product registration operations, the customer has the initiative in operations, so in addition to changes made for legitimate reasons, there is a possibility that changes may be made for illegitimate reasons.

[0100] For example, in package recognition, products are recognized based on a probability value called similarity, so there is a possibility that the wrong product may be registered. In such a case, a legitimate change operation may be performed to delete the erroneously registered product in order to register the correct product. On the other hand, a change operation may be performed for an illegitimate reason, such as deleting a registered product for the purpose of shoplifting, and then taking the deleted product home.

[0101] For the reasons mentioned above, when a change is made to a registered product on a self-service product sales data processing device, it is preferable for a store clerk to check it. However, calling a store clerk every time a change is made is a heavy burden on the store clerk, and there is room for improvement.

[0102] Therefore, in the POS terminal 10 of this embodiment, when a registered product change operation is performed, the store clerk checks it once, thereby reducing the burden on the store. Specifically, when the end of the product registration process or the start of the payment process is instructed by operating the checkout button Bc or the like, the notification processing unit 104 determines whether or not a registered product change operation has been performed before the operation is performed based on the presence or absence of change content information.

[0103] The notification processor 104 is an example of a notification means. When change content information exists, the notification processor 104 determines that a change operation of a registered product has been performed, and transmits information notifying that a change operation has been performed (hereinafter, also referred to as notification information) to the store clerk terminal 40. The notification information includes, for example, terminal information such as a terminal identifier capable of identifying the POS terminal 10, as well as change content information.

[0104] The notification information may also include information on products registered in the registered product storage unit 131. For example, the notification information may include information on registered products corresponding to the ID included in the change content information, that is, information on the registered products before the change operation, among the registered products registered in the registered product storage unit 131. This allows the store clerk terminal 40 receiving the notification information to check the information and status of the registered products before and after the change operation, thereby improving the convenience of checking the change content.

[0105] The notification information may also include image data of the product registration screen B displayed on the display unit 16 of the POS terminal 10. In this case, the image data may be a snapshot of the product registration screen B, or may be a stream of the product registration screen B displayed on the display unit 16 in real time. This allows the store clerk terminal 40 receiving the notification information to check the state of the product registration screen B showing the contents of the change operation, thereby improving the convenience of checking the contents of the change.

[0106] The change processing unit 105 is an example of a third receiving means, a reflecting means, and a suppressing means. The change processing unit 105 performs a process to reflect the contents of the registration process changed by the customer in the registered product storage unit 131 based on the change content information. Specifically, when the end of the product registration process or the start of the payment process is instructed by operating the checkout button Bc or the like while change content information is present, the change processing unit 105 suppresses the execution of the checkout process, thereby maintaining a state in which the registered product storage unit 131 can be operated.

[0107] In addition, the change processing unit 105 performs a process of reflecting the change contents of the registered product recorded in the change contents information in the registered product storage unit 131 in response to a confirmation operation by the store clerk. Here, the confirmation operation by the store clerk may be performed via the operation unit 15 of the POS terminal 10, or may be performed remotely from the store clerk terminal 40. In the former case, the change processing unit 105 reflects the change contents in the registered product storage unit 131 on the condition that, for example, after the store clerk ID of the store clerk is read via the card reader 17 or the imaging device 20, an operation instructing acceptance of the change contents is performed. In the latter case, the change processing unit 105 reflects the change contents in the registered product storage unit 131 on the condition that, for example, information instructing acceptance of the change contents is received from the store clerk terminal 40.

[0108] The store clerk may check the changes for each registered product that has been changed, or may check all registered products at once. Furthermore, the change processing unit 105 may cancel the changes or make further changes in response to a store clerk operation.

[0109] When all the changes have been reflected in the registered product storage unit 131, the change processing unit 105 notifies the start of payment processing to the payment processing unit 106 by permitting transition to payment processing. Note that if there is no change information and an instruction is given to end the product registration processing or to start the payment processing, the payment processing unit 106 will immediately start the payment processing.

[0110] The payment processing unit 106 is an example of a payment means. The payment processing unit 106 executes a payment process for the products registered in the product registration process. Specifically, the payment processing unit 106 calculates the total amount of the registered products based on the information registered in the registered product storage unit 131, and executes a payment process in which the total amount is settled with the payment amount paid by the customer.

[0111] The payment process may be performed by the POS terminal 10 alone, or in cooperation with other devices (for example, an accounting device, a payment server, etc.).

[0112] For example, when the payment processing unit 106 receives an operation to start payment via the operation unit 15, it displays a screen on the display unit 16 that allows the user to select a payment method such as cash payment, electronic money payment, or credit card payment. When cash payment is selected, the payment processing unit 106 executes a payment process to settle the total amount of the registered products with the payment amount deposited via a deposit device (not shown). When electronic money payment or credit card payment is selected, the payment processing unit 106 also receives input of payment information for making a payment using the selected payment method, and cooperates with an external payment server to execute a payment process to settle the total amount of the registered products based on the input payment information. The payment information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, the input may be received via the imaging device 20.

[0113] As shown in FIG. 9, the store clerk terminal 40 further includes a notification receiving section 401 and a confirmation operation receiving section 402 as functional components.

[0114] The functional configuration of the clerk terminal 40 is a software configuration realized by the processor of the clerk terminal 40 operating in accordance with the control program 441 stored in the ROM 42 or memory unit 44, but is not limited to this. For example, some or all of the functional configuration of the clerk terminal 40 may be realized by hardware such as a dedicated circuit.

[0115] The notification reception unit 401 issues a notification based on notification information transmitted from the POS terminal 10. Specifically, when the notification reception unit 401 receives notification information from the POS terminal 10, the notification reception unit 401 notifies the store clerk that a change operation for a registered product has been performed on the POS terminal 10 by displaying the terminal information and the contents of the change operation included in the notification information on the display unit 46. Furthermore, when the notification information includes the states of the registered product before and after the change operation, the notification reception unit 401 may display the states of the registered product before and after the change operation so that they can be compared. Note that the method of notification is not limited to display, and notification may also be performed using sound.

[0116] The confirmation operation receiving unit 402 receives an operation such as approval for the change operation performed in the POS terminal 10 via the operation unit 45. For example, when the confirmation operation receiving unit 402 receives an approval operation for the change operation that specifies terminal information, the confirmation operation receiving unit 402 transmits instruction information for instructing approval of the change contents to the POS terminal 10 corresponding to the terminal information.

[0117] In addition, in the case where the confirmation of the change operation is performed by the POS terminal 10, the confirmation operation receiving unit 402 may be configured to be removed from the store clerk terminal 40.

[0118] Next, a description will be given of the flow of processing executed by the product sales data processing system 1. Fig. 13 is a sequence diagram showing an example of the flow of processing executed by the product sales data processing system 1.

[0119] First, a customer performs a registration operation in the POS terminal 10 by holding a product to be purchased over the imaging device 20. The image acquisition unit 201 of the imaging device 20 acquires an image captured by the imaging unit 25 (step S1). Next, the image transmission unit 202 of the imaging device 20 transmits the image obtained by imaging to the edge device 30 (step S2). In addition, the code recognition unit 203 of the imaging device 20 executes code recognition on the image obtained by imaging (step S3).

[0120] When an image is transmitted from the imaging device 20, the package recognition unit 302 of the edge device 30 performs package recognition on the image (step S4). When the product is uniquely recognized or multiple candidate products are recognized by the package recognition, the recognition result output unit 303 of the edge device 30 transmits the recognition result including the product code of the recognized product to the imaging device 20 (step S5).

[0121] When the recognition result output unit 204 of the imaging device 20 receives the recognition result from the edge device 30, it outputs the recognition result to the POS terminal 10 (step S6). Furthermore, when the product is uniquely recognized by the code recognition, the code recognition unit 203 outputs the recognition result including the product code of the recognized product to the POS terminal 10 (step S6).

[0122] When an image of a product is captured by a customer's registration operation, the product will enter the imaging range of the imaging device 20 starting from the edge of the product. Therefore, there is a high possibility that the recognition result of the package recognition will be output first, and the recognition result of the code recognition will be output later, to the POS terminal 10. However, the output order of the recognition results of the code recognition and the package recognition is not limited to this.

[0123] When the recognition result is input from the imaging device 20, the product registration processing unit 102 of the POS terminal 10 executes a product registration process based on the recognition result (step S7). Moreover, the POS terminal 10, the imaging device 20, and the edge device 30 execute the processes of steps S1 to S7 described above every time a customer performs a product registration operation.

[0124] Furthermore, when the change operation receiving unit 103 of the POS terminal 10 receives a change operation for a registered product (step S8), it generates change content information and updates the display of the product registration screen B according to the content of the change operation (step S9).

[0125] Next, when the POS terminal 10 receives an operation to end the product registration (step S10), the notification processor 104 transmits notification information to the store clerk terminal 40 (step S11) because a change operation for the registered product has been performed in step S8.

[0126] On the other hand, when the clerk terminal 40 receives the notification information from the POS terminal 10, the notification receiving unit 401 notifies the clerk that a change operation for the registered product has been performed on the POS terminal 10 based on the notification information (step S12). Here, for example, when a confirmation operation is performed on the clerk terminal 40 to instruct acceptance of the changes, the confirmation operation receiving unit 402 receives the confirmation operation (step S13) and transmits instruction information to the POS terminal 10 to instruct acceptance of the changes (step S14).

[0127] When the change processing unit 105 of the POS terminal 10 receives instruction information from the store clerk terminal 40, it determines that the change contents have been confirmed, and reflects the change contents of the registered products recorded in the change contents information in the registered product memory unit 131 (step S15).

[0128] Next, the payment processing unit 106 of the POS terminal 10 calculates the total price of the registered products registered in the registered product storage unit 131, and executes the payment process based on the total price (step S16).

[0129] Next, the process performed by the POS terminal 10 will be described with reference to Fig. 14. Here, Fig. 14 is a flow chart showing an example of the process performed by the POS terminal 10. Note that this process corresponds to the processes of steps S7 to S15 in Fig. 13. Also, as a premise of this process, it is assumed that the product registration screen B is displayed on the display unit 16.

[0130] First, the recognition result acquisition unit 101 determines whether or not the recognition result has been acquired from the imaging device 20 (step S21). If the recognition result has not been acquired (step S21; No), the process proceeds to step S26. If the recognition result acquisition unit 101 has acquired the recognition result from the imaging device 20 (step S21; Yes), the process proceeds to step S22.

[0131] The product registration processing unit 102 determines whether or not the product can be uniquely identified based on the recognition result acquired in step S21 (step S22). Specifically, the product registration processing unit 102 determines whether or not one product code recognized by code recognition or package recognition is included in the recognition result.

[0132] Here, if it is determined that the product can be uniquely identified (step S22; Yes), the product registration processing unit 102 executes product registration processing based on the product code included in the recognition result (step S25), and proceeds to step S26.

[0133] When the product registration processing unit 102 executes the product registration processing, the product information of the registered products, such as product names and prices, is displayed on the product registration screen B in the order in which the products were registered.

[0134] On the other hand, if it is determined that it is impossible to uniquely identify the product (step S22; No), that is, if the recognition result includes multiple product codes of the candidate products, the product registration processing unit 102 displays a product selection screen A on the display unit 16, which allows one product to be selected from each of the candidate products (step S23). Next, the product registration processing unit 102 waits until one product is selected from the product selection screen A (step S24; No).

[0135] In step S24, when one product is selected from the product selection screen A (step S24; Yes), the product registration processing unit 102 executes the product registration process based on the product code of the selected product (step S25), and proceeds to step S26. Note that, in response to the selection of the product in step S15, the product registration processing unit 102 erases the product selection screen A and displays the product registration screen B.

[0136] Next, the change operation receiving unit 103 determines whether or not a change operation for any of the registered products has been received (step S26). If a change operation has not been received (step S26; No), the process proceeds to step S29.

[0137] Furthermore, when the change operation acceptance unit 103 determines that the change operation has been accepted (step S26; Yes), it updates the product registration screen B based on the accepted change content (step S27). Next, the change operation acceptance unit 103 stores change content information indicating the change content (step S28) and proceeds to step S29.

[0138] Next, the control unit 100 of the POS terminal 10 judges whether or not an operation to instruct the end of the product registration process (or an operation to instruct the start of the completion process) has been accepted (step S29). If an instruction to end the product registration process has not been accepted (step S29; No), the control unit 100 returns to step S21 to prepare for obtaining a new recognition result.

[0139] Also, when an operation to instruct the end of the product registration process is received (step S29; Yes), the notification processing unit 104 (and the change processing unit 105) judges whether or not a change operation of the registered product has been performed before the end instruction (step S30). If a change operation has not been performed (step S30; No), the process immediately proceeds to step S16 in FIG.

[0140] If it is determined in step S30 that a change operation has been performed (step S30; Yes), the notification processor 104 transmits notification information to the store clerk terminal 40 to notify that a change operation of the registered product has been performed in the POS terminal 10 (step S31). The change processor 105 waits until a confirmation operation is performed by the store clerk (step S32; No).

[0141] In addition, while waiting for the store clerk to confirm the change, the change processing unit 105 may display on the display unit 16 a message informing the store clerk that confirmation will be performed, or a message advising the user to wait for payment to be made until confirmation of the changes has been completed.

[0142] When the change processing unit 105 receives a confirmation operation by the store clerk, either directly via the operation unit 15 or remotely from the store clerk terminal 40 (step S32; Yes), it reflects the changes recorded in the change information in the registered product memory unit 131 (step S33), and then transitions to step S16 in Figure 13.

[0143] As described above, the POS terminal 10 obtains the recognition results of the code recognition and package recognition based on the captured image of the product, and executes the product registration process for the product recognized in the recognition process. The POS terminal 10 also displays a product registration screen showing each registered product, and when a change operation for a registered product is accepted, the content of the change operation is reflected in the target product shown on the product registration screen. When the POS terminal 10 accepts an operation to end the product registration process or start the payment process, the POS terminal 10 transmits notification information including the content of the change operation performed up to the operation to the store clerk terminal 40. This allows the POS terminal 10 to have the store clerk perform the confirmation work associated with the change operation for the registered product in one go, thereby reducing the burden on the store clerk.

[0144] Furthermore, the POS terminal 10 reflects the contents of the change operation only on the product registration screen until the store clerk performs a confirmation operation, and, on the condition that the confirmation operation has been performed, reflects the contents of the change operation in the registered product storage unit 131. This allows the POS terminal 10 to present the contents of the change operation to the customer while protecting the registered product storage unit 131. Furthermore, the POS terminal 10 can improve the identifiability of the registered product that has been the subject of the change operation when the store clerk checks the product registration screen, thereby improving the efficiency of the confirmation work.

[0145] Furthermore, the POS terminal 10 accepts a confirmation operation from the clerk terminal 40, and reflects the contents of the change operation by the confirmation operation in the registered product storage unit 131. This allows the POS terminal 10 to reduce the travel time of the clerk operating the clerk terminal 40 to the POS terminal 10, thereby improving the efficiency of the confirmation work.

[0146] Furthermore, when a change operation is performed, the POS terminal 10, even if it receives an operation to end the product registration process or start the payment process, refrains from executing the payment process until the store clerk confirms. This allows the POS terminal 10 to prevent a payment from being made in an unconfirmed state where the store clerk has not confirmed it, thereby preventing additional work such as redoing the transaction or collecting additional money. Therefore, the POS terminal 10 can facilitate transactions.

[0147] In addition, the POS terminal 10 notifies the store clerk terminal 40 of information indicating the state of the product that is the target of the change operation before and after the change as notification information. This allows the POS terminal 10 to support the store clerk terminal 40 in checking the state before and after the change, thereby making the checking work more efficient.

[0148] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each of the above-described devices. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. Note that the following mainly describes the differences from the above-described embodiment, and a detailed description of the common points with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0149] (Variation 1) In the above embodiment, a configuration has been described in which a change operation for a registered product is accepted regardless of whether the product was registered by code recognition or package recognition (hereinafter, also referred to as a product identification method). However, the present invention is not limited to this configuration, and the content of the process related to the change operation may be changed depending on the product identification method.

[0150] Specifically, when executing the product registration process, the product registration processing unit 102 according to this modification registers the product identification method used to identify the product together with the product information, etc. Then, the change operation receiving unit 103, the notification processing unit 104, and the change processing unit 105 execute processing according to the product identification method of the registered product that is the target of the change operation.

[0151] For example, the change operation receiving unit 103 may be configured to permit a change operation of a registered product registered using package recognition, and to prohibit a change operation of a registered product registered using code recognition. Also, for example, the notification processing unit 104 may transmit notification information including a product identification method of a product that is the target of a change operation. Also, for example, the change processing unit 105 may use different methods of confirmation operation for code recognition and package recognition. As an example, when a registered product registered using package recognition is included in the target of a change operation, confirmation by remote operation may be prohibited, and confirmation operation at the user's own POS terminal 10 may be required.

[0152] In this way, the POS terminal 10 according to this modification can change the content of the process for the change operation depending on the product specification method, and therefore can accommodate various operational forms.

[0153] (Variation 2) In the above-described embodiment, when a change operation is performed on a registered product, a confirmation operation by a store clerk is mandatory, but this is not limited to the above, and the configuration may be such that confirmation operation by a store clerk is not required when the registered product that is the subject of the change operation meets specified conditions.

[0154] For example, the similarity calculated during package recognition may be used as an index to switch whether or not confirmation operation by a store clerk is required. In this case, when executing the product registration process, the product registration processing unit 102 registers the similarity calculated during package recognition of the product to be registered together with the product information, etc. When the product is identified by code recognition, the similarity may be set to, for example, null or 100%, etc.

[0155] However, products that are registered due to erroneous recognition tend to have a low degree of similarity. Therefore, a threshold value may be set to the degree of similarity at which erroneous recognition may occur, and the necessity for confirmation by a store clerk may be switched based on the threshold value. For example, if the threshold value is less than the threshold value, it is determined that a legitimate change operation (deletion operation) has been performed to register the correct product, and confirmation by a store clerk is not required. On the other hand, if the threshold value is equal to or greater than the threshold value, it is determined that a change operation for the product has been performed even though the correct product has been registered, and therefore it is possible that an unauthorized change operation has been performed, and confirmation by a store clerk is required.

[0156] In this case, if the similarity of all registered products that are the subject of the change operation is less than the threshold, the notification processor 104 does not notify the store clerk terminal 40. If the similarity of some registered products that are the subject of the change operation is equal to or greater than the threshold, the notification processor 104 executes a notification to the store clerk terminal 40 as in the above-described embodiment. The notification processor 104 may be configured to exclude information on registered products whose similarity is less than the threshold from the notification information.

[0157] Furthermore, when the similarities of all registered products that are the subject of the change operation are less than the threshold, the change processing unit 105 does not require a confirmation operation by a store clerk and reflects the changes recorded in the change content information in the registered product storage unit 131. Note that when the similarities of some (or all) of the registered products that are the subject of the change operation are equal to or greater than the threshold, the change processing unit 105 deems it necessary and suspends reflecting the changes in the registered product storage unit 131 until the confirmation operation is performed.

[0158] In this way, the POS terminal 10 according to this modified example can change whether or not confirmation by the store clerk is required and the confirmation method depending on the registered product that is the subject of the change operation, making it possible to accommodate a variety of operational forms.

[0159] (Variation 3) In the above-described embodiment, confirmation by a store clerk is mandatory regardless of the content of the change operation, but this is not limited to the above, and the configuration may be such that confirmation by a store clerk is not required if the content of the change operation meets specified conditions.

[0160] Specifically, if the change operation instructs an increase in the number of points, it is assumed that a legitimate change operation has been performed to increase the number of points of a previously registered registered product in order to avoid repeated registration operations for multiple products of the same type. Therefore, for example, if the change operation instructs an increase in the number of points, a confirmation operation by a store clerk may be unnecessary.

[0161] In this case, if the change content of all registered products that are the subject of the change operation indicates an increase in points, the notification processing unit 104 does not notify the store clerk terminal 40. Also, if the change content of some (or all) registered products that are the subject of the change operation indicates deletion or a decrease in points, the notification processing unit 104 executes a notification to the store clerk terminal 40 as in the above-mentioned embodiment. Note that the notification processing unit 104 may be configured to exclude information on registered products that are changed to an increase in points from the notification information.

[0162] Furthermore, when the change content of all registered products that are the subject of the change operation indicates an increase in points, the change processing unit 105 does not require a confirmation operation by the store clerk and reflects the change content recorded in the change content information in the registered product storage unit 131. Furthermore, when the change content of some (or all) registered products that are the subject of the change operation indicates a deletion or a decrease in points, the change processing unit 105 requires a confirmation operation by the store clerk as in the above embodiment, and suspends the reflection in the registered product storage unit 131 until the confirmation operation is performed.

[0163] In this way, the POS terminal 10 according to this modification can change whether or not confirmation by a store clerk is required and the confirmation method depending on the content of the change operation, making it possible to accommodate various operational forms.

[0164] (Variation 4) In the above embodiment, when a change operation is performed on a registered product, notification information is sent to the store clerk terminal 40 at the timing when an instruction to end the product registration process (or an instruction to start the payment process) is given. However, the timing of notifying the store clerk terminal 40 is not limited to this, and the notification may be sent at the timing when the change operation is performed.

[0165] In this case, the notification processor 104 includes change content information indicating the changes made in the most recent change operation in the notification information and sends it to the store clerk terminal 40. The change processor 105 also reflects the changes made in the change operation in the registered product storage unit 131 in response to the store clerk's confirmation operation. In this case, the change processor 105 may be able to register a new product or perform a change operation even if the store clerk has not yet performed a confirmation operation. However, it is preferable to prevent the payment process from being executed when the store clerk has not yet performed a confirmation operation.

[0166] In addition, the notification processing unit 104 may notify the store clerk terminal 40 each time a change operation is performed, or may control whether or not to notify the store clerk terminal 40 depending on the content of the change operation, the similarity of the registered products that are the subject of the change operation, etc.

[0167] For example, the notification processor 104 may be configured not to notify the store clerk terminal 40 when the change operation instructs an increase in points. The notification processor 104 may also be configured not to notify the store clerk terminal 40 when a change operation is performed on a registered product whose similarity is less than a threshold. In this way, by excluding a registered product from the targets of notification when there is a high possibility that a legitimate change operation has been performed, it is possible to reduce the opportunities for the store clerk to check the change operation and reduce the burden on the store clerk.

[0168] Furthermore, the notification processor 104 may use both a notification when an instruction to end the product registration process (or an instruction to start the payment process) is given and a notification when a change operation is performed. For example, when the content of the change operation is to delete a registered product and the similarity of the registered product is equal to or greater than a threshold, and there is a possibility that an unauthorized change operation has been performed, the notification processor 104 gives a notification when the change operation is performed. On the other hand, when a change operation is performed under other conditions, a notification is given when an instruction to end the product registration process (or an instruction to start the payment process) is given.

[0169] As a result, the POS terminal 10 according to this modification can notify the store clerk at the timing when an unauthorized change operation is performed if there is a possibility that the change operation has been performed, and therefore the change contents can be promptly confirmed. Also, the POS terminal 10 notifies the store clerk at the timing when an instruction to end the product registration process is given for a change operation that may have been performed legitimately, such as a change operation for a registered product whose similarity is less than a threshold value, so that a single confirmation operation is sufficient. Therefore, the POS terminal 10 can reduce the number of times the store clerk needs to perform a confirmation operation, and the burden on the store clerk can be reduced.

[0170] (Variation 5) In the above embodiment, the state of the registered product before and after the change operation is included in the notification information and transmitted. In this modified example, in order to further improve the efficiency of the confirmation operation, the actual state of the registered product that has been changed is included in the notification information and transmitted.

[0171] In this case, the POS terminal 10 further includes an imaging device such as a camera (hereinafter, also referred to as a surveillance camera) capable of capturing images of merchandise brought by a customer to the POS terminal 10. The surveillance camera may be realized using the imaging device 20.

[0172] Furthermore, the change operation receiving unit 103 of this modified example, when triggered by receiving a change operation for a registered product, acquires surveillance images (still images or motion) captured by a surveillance camera for a predetermined period of time, and stores the images in association with the change content information of the registered product. Furthermore, when transmitting notification information to the store clerk terminal 40, the notification processing unit 104 of this modified example transmits the surveillance images associated with the change content information together with the notification information.

[0173] As a result, the clerk terminal 40 that receives the notification information can check the state of the product displayed in the surveillance image along with the contents of the change operation. Specifically, the clerk at the clerk terminal 40 can check the actual state of the product and the customer's actions from the surveillance image when the change operation is performed. Therefore, the POS terminal 10 according to this modification can provide the clerk with information to determine whether or not the operation is unauthorized, thereby supporting the confirmation work when a change operation is performed.

[0174] (Variation 6) In the above embodiment, when a change operation of a registered product is performed, the sales clerk is notified of the change operation by sending notification information to the sales clerk terminal 40. However, the notification method is not limited to this, and the sales clerk may be notified by other methods.

[0175] For example, if the POS terminal 10 is equipped with a notification device such as a warning light, the notification processing unit 104 may control the notification device to notify that a change operation has been performed. In this case, the store clerk will recognize that a change operation has been performed by looking at the state of the notification device such as a warning light, and will head to the POS terminal 10 where the change operation has been performed.

[0176] As a result, even if the POS terminal 10 is configured in a system that does not use the store clerk terminal 40, it can achieve the same effects as the above-described embodiment.

[0177] (Variation 7) In the above embodiment, the imaging device 20 performs code recognition, but the present invention is not limited to this. For example, the POS terminal 10 or the edge device 30 may perform the code recognition.

[0178] For example, when code recognition is performed in the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) an image obtained by imaging to the POS terminal 10. The POS terminal 10 also includes a functional configuration of a code recognition unit 203, and the code recognition unit 203 executes code recognition on the image transmitted from the imaging device 20.

[0179] Furthermore, for example, when code recognition is performed by the edge device 30, the edge device 30 has a functional configuration of a code recognition unit 203, and the code recognition unit 203 executes code recognition on the image transmitted from the imaging device 20. In this case, the recognition result output unit 303 of the edge device 30 outputs the recognition results of the code recognition and package recognition in order from the first recognized one.

[0180] In the above embodiment, the edge device 30 performs package recognition. However, the present invention is not limited to this, and the POS terminal 10 or the imaging device 20 may perform the package recognition.

[0181] For example, when package recognition is performed at the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) an image obtained by imaging to the POS terminal 10. The POS terminal 10 also includes a functional configuration of a package recognition unit 302, and the package recognition unit 302 executes package recognition on the image transmitted from the imaging device 20.

[0182] Furthermore, for example, when package recognition is performed by the imaging device 20, the imaging device 20 has a functional configuration of a package recognition unit 302, and executes package recognition on an image obtained by imaging by the package recognition unit 302. In this case, the recognition result output unit 204 of the imaging device 20 outputs the recognition results of the code recognition and package recognition in order from the one recognized first.

[0183] In the above embodiment, the edge device 30 is described as a single device, but the present invention is not limited to this. For example, the edge device 30 may be realized by a cloud (cloud computing) composed of a plurality of information processing devices. In addition, the store server 50 may have the functions of the edge device 30.

[0184] (Variation 8) In the above embodiment, the POS terminal 10 has been described as a device operated by the customer himself / herself. However, the present invention is not limited to this, and the POS terminal 10 may be a device operated by a clerk of the store T. In this case, the imaging device 20 captures an image of a commodity based on a registration operation by the clerk.

[0185] (Variation 9) In the above embodiment, the POS terminal 10 is described as having one imaging device 20 (or imaging section 25). However, this is not limiting, and the POS terminal 10 may be configured to have a plurality of imaging devices 20 (or imaging sections 25). In this case, code recognition may be performed by each imaging device 20, or may be performed by a single imaging device 20. It is preferable that the plurality of imaging devices 20 (or imaging sections 25) are configured to image merchandise from different imaging positions. This allows the merchandise to be imaged from multiple angles, thereby improving the recognition speed and recognition rate.

[0186] The programs executed by each device in the above-described embodiments (and modified examples) are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0187] Furthermore, the programs executed by each device of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0188] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. These new embodiments and their modifications can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0189] 1 System 10. POS terminals 20 Imaging device 30 Edge Devices 40 Store clerk terminal 50 Store Server 101 Recognition result acquisition unit 102 Product registration processing unit 103 Change operation reception section 104 Notification processing unit 105 Change Processing Section 106 Payment processing unit 201 Image Acquisition Unit 202 Image transmission unit 203 Code Recognition Unit 204 Recognition result output unit 301 Image Acquisition Unit 302 Package Recognition Unit 303 Recognition result output unit 401 Notification Reception Department 402 Confirmation Operation Reception Section [Prior art documents] [Patent documents]

[0190] [Patent Document 1] JP 2015-22367 A

Claims

1. An acquisition means for acquiring a recognition result of a recognition process for recognizing the commodity based on an image of the commodity; a registration means for executing a registration process for registering a product recognized in the recognition process as a purchase target based on the recognition result acquired by the acquisition means; A display means for displaying a screen showing each of the registered products; a first reception means for receiving a change operation for a registered product and adding information corresponding to the content of the change operation to the product to be changed that is displayed on the screen; A second reception means for receiving an end operation for instructing the end of the registration process; a notification means for notifying a store clerk that the change operation performed up until the end operation has been performed when the end operation is performed; A product sales data processing device comprising:

2. A third reception means for receiving a confirmation operation by a store clerk; a reflecting means for reflecting the content of the change operation in the registered product when the confirmation operation is performed; The product sales data processing device according to claim 1 , further comprising:

3. The notification means notifies a terminal device operated by a store clerk, The third reception means receives the confirmation operation from the terminal device. The product sales data processing device according to claim 2 .

4. The notification means notifies the terminal device of information indicating states of the product that is the subject of the change operation before and after the change operation. The product sales data processing device according to claim 3.

5. a payment means for performing payment processing for the registered product in response to the completion operation; a suppression means for suppressing the payment process until the confirmation operation is performed; The product sales data processing device according to claim 2 , further comprising:

6. A computer of a product sales data processing device, An acquisition means for acquiring a recognition result of a recognition process for recognizing the commodity based on an image of the commodity; a registration means for executing a registration process for registering a product recognized in the recognition process as a purchase target based on the recognition result acquired by the acquisition means; A display means for displaying a screen showing each of the registered products; a first reception means for receiving a change operation for a registered product and reflecting the content of the change operation in the product to be changed that is displayed on the screen; A second reception means for receiving an end operation for instructing the end of the registration process; a notification means for notifying a store clerk that the change operation performed up until the end operation has been performed when the end operation is performed; A program that can be used to

Images