Product registration device and program

Abstract

To provide a commodity registration device and a program that can increase efficiency of processing related to product registration.SOLUTION: A commodity registration device includes: first acquisition means for acquiring a recognition result of first recognition processing of reading from an image of a target commodity, captured by an imaging device, a symbol including identification information that can identify the target commodity; second acquisition means for, based on an exterior feature of the target commodity represented on the image, acquiring a recognition result of second recognition processing of estimating a product similar to the feature from among a plurality of predetermined products; and registration means for, in a case in which the target commodity can be identified by a recognition result acquired earlier between the recognition result of the first recognition processing and the recognition result of the second recognition processing, registering the target commodity as a purchase target. The registration means waits for new acquisition of the recognition result in a case of determining that the recognition result of the second recognition processing includes a plurality of candidate commodities to be candidates for the target commodity and that the symbol is attached to any of the candidate commodities based on another type of information other than the image.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product registration device and a program. [Background technology]

[0002] Conventionally, there are POS systems that perform product registration and processing of products sold in stores such as shopping centers, mass retailers, department stores, etc. Some of these POS systems recognize products from the characteristics of images of the products captured by an imaging device and perform product registration processing based on the recognition results.

[0003] The above-mentioned product recognition process is performed by comparing the external characteristics of each product sold in the store with the external characteristics of the product extracted from the image. In this case, the recognition result is output as a similarity between each product sold in the store and the captured product, and a product with a similarity equal to or greater than a threshold (e.g., 95% or greater) is uniquely identified as the product that was the subject of the image.

[0004] In addition, if there are no products that meet or exceed the threshold value, a predetermined number of products with the highest similarity are extracted as candidate products, and a screen (hereinafter referred to as a product selection screen) is displayed on which the operator can select one product from the candidate products, leaving it to the operator to select the correct product.

[0005] However, the operation of selecting a correct product from among multiple candidate products that have similar appearances is cumbersome and time-consuming for operators who are unfamiliar with the operation. For these reasons, it is desirable to reduce the number of times the product selection screen is displayed in order to improve the efficiency of the product registration process. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide a product registration device and a program capable of improving the efficiency of processes relating to product registration. [Means for solving the problem]

[0007] A product registration device of an embodiment includes a first acquisition means for acquiring a recognition result of a first recognition process that reads a symbol including identification information capable of identifying a target product from an image of a target product captured by an imaging device; a second acquisition means for acquiring a recognition result of a second recognition process that estimates a product similar to the characteristics of the target product from a predetermined number of products based on the external characteristics of the target product shown in the image; and a registration means for registering the target product as a purchase target if the target product can be identified by the recognition result acquired earlier between the recognition result of the first recognition process and the recognition result of the second recognition process. When the recognition result of the second recognition process includes a number of candidate products that are candidates for the target product and the registration means determines based on information other than the image that the symbol is affixed to any of the candidate products, the registration means waits for acquisition of a new recognition result. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a product sales data processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of package recognition according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the POS terminal according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data configuration of the product master according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a hardware configuration of the imaging device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of the edge device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of a store server according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of functional configurations of the POS terminal, the imaging device, and the edge device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a product selection screen displayed on the POS terminal of the embodiment. [Figure 10] FIG. 10 is a sequence diagram illustrating an example of a flow of processing executed in the product sales data processing system according to the embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a product registration process performed by the POS terminal of the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a confirmation screen displayed on the POS terminal of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, a product sales data processing device in which a customer himself performs the operation of registering products to be purchased will be described as an example. Note that the present invention is not limited to the embodiment described below.

[0010] Fig. 1 is a diagram showing an example of a product sales data processing system according to an embodiment. In Fig. 1, the product sales data processing system 1 is installed in a store such as a supermarket, a mass retailer, a convenience store, or a specialty store. The product sales data processing system 1 includes one or more POS (Point of Sales) terminals 10 (product sales data processing devices), an imaging device 20, an edge device 30, and a store server 40.

[0011] The POS terminal 10, the edge device 30, and the store server 40 are connected to a communication line L such as a LAN (Local Area Network) that allows mutual notification. One or more imaging devices 20 are connected to each POS terminal 10 by, for example, a USB (Universal Serial Bus) cable U. The imaging devices 20 are also connected to the communication line L. The imaging devices 20 may be configured to be connected to the communication line L via the POS terminal 10.

[0012] The imaging device 20 is a device equipped with an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imaging device 20 is provided, for example, facing the customer who operates the POS terminal 10, and captures images of products purchased by the customer. Specifically, in this embodiment, in order to perform registration processing, the customer who operates the POS terminal 10 holds each of the products he or she wishes to purchase over the imaging device 20, causing the imaging device 20 to capture an image of the product.

[0013] When the imaging device 20 captures an image (hereinafter also referred to as a captured image) by imaging, the imaging device 20 outputs the captured image to the edge device 30. Specifically, the imaging device 20 outputs an image captured at a predetermined frame rate to the edge device 30.

[0014] The imaging device 20 also performs code recognition on the image obtained by imaging, attempting to detect symbols such as barcodes and two-dimensional codes from the image. In code recognition, symbols such as barcodes and two-dimensional codes attached to products are detected from the image, and the detected symbols are decoded to obtain decoded information, which is output to the POS terminal 10. The decoded information includes product identification information that identifies the product to which the symbol is attached, so that the imaged product can be identified based on the product identification information. In this embodiment, a product code capable of identifying each product is used as the product identification information, but information other than the product code may be used as long as it is capable of identifying the product.

[0015] The imaging device 20 may extract images that include or may include a commodity from the images obtained by imaging, and output the images to the edge device 30 or use known techniques for symbol extraction. In this case, the image extraction method is not particularly limited, and known techniques can be used. For example, the imaging device 20 may detect the movement of an object included in an image from the difference between multiple temporally consecutive frames, and extract an image in which the movement is detected, and use the extracted image as a processing target.

[0016] The edge device 30 is an information processing device that performs predetermined processing such as image processing, image recognition, etc. Specifically, the edge device 30 performs package recognition to recognize a product from an image input from the imaging device 20.

[0017] Package recognition is a technology for identifying a product imaged by the imaging device 20 based on the external characteristics of the product depicted in the image. Here, the external characteristics of the product include, for example, the design, pattern, color, texture, shape, etc. of the package.

[0018] Package recognition can use known technologies such as general object recognition and image recognition using models such as deep learning. In package recognition, based on a feature amount indicating the external characteristics of a product extracted from an image, one product whose similarity (also expressed as likelihood) is equal to or exceeds a threshold value among products similar to the feature amount is identified as the product that is the subject of the image.

[0019] Fig. 2 is a diagram for explaining an example of package recognition. As shown in Fig. 2, in package recognition, a product is identified from an image captured by the imaging device 20 using three types of trained models or dictionary data (hereinafter, also collectively referred to as dictionaries), which are an object detection model 51, a feature extraction model 52, and a matching dictionary 53.

[0020] The object detection model 51 is a dictionary for detecting an area of ​​an object appearing in a captured image. When a captured image captured by the imaging device 20 is input, the object detection model 51 detects and outputs an area showing a commodity (hereinafter also referred to as an object image) from the captured image.

[0021] The feature extraction model 52 is a dictionary for extracting feature amounts indicating the external characteristics of a product from the product depicted in the area detected by the object detection model 51. When an object image or a captured image including an object image is input, the feature extraction model 52 extracts and outputs feature amounts indicating the external characteristics of the product depicted in the object image.

[0022] The matching dictionary 53 is a dictionary for identifying products similar to the features of a product based on the features of the product. For example, the matching dictionary 53 stores a plurality of products sold in a store in association with features indicating the characteristics of the product for each product code that identifies the product. The matching dictionary 53 also compares the input features with the stored features to calculate a similarity that quantitatively indicates how similar the many stored products are to the imaged product. The higher the similarity of a product, the higher the degree of similarity of the product to the imaged product (i.e., the probability that it is the imaged product). As an example, a product with a similarity of 95% or more is highly likely to be the same product as the imaged product.

[0023] The edge device 30 uses an object detection model 51 to detect an area in which a product is captured from an image captured by the imaging device 20. Next, the edge device 30 uses a feature extraction model 52 to extract features that quantitatively represent the external features of the product captured in that area (object image) using a multidimensional vector or the like. Then, the edge device 30 uses a comparison dictionary 53 to compare the extracted features with features of many products stored in the dictionary to calculate a similarity, and recognizes the product according to the similarity.

[0024] Specifically, when there is one product whose similarity is equal to or greater than a threshold (e.g., 95% or greater), the edge device 30 specifies the product whose similarity is equal to or greater than the threshold as a product captured by the imaging device 20. When there is no product whose similarity is equal to or greater than the threshold (e.g., when the similarity of all products is less than 95%), the edge device 30 does not specify one product, but specifies multiple products whose similarity is high as candidate products. For example, when the similarity of all products is less than the threshold, the edge device 30 extracts a predetermined number (e.g., four) of candidate products from those whose similarity is high. Then, the edge device 30 transmits the product code of the specified product or the candidate product as the recognition result of the package recognition to the imaging device 20 that transmitted the captured image. When the edge device 30 recognizes multiple candidate products, it transmits the similarity calculated for the candidate product together with the recognition result. When the edge device 30 can uniquely identify a product, it may transmit the similarity together with the recognition result. In addition, the edge device 30 has a lower threshold for similarity determination, and when the similarity falls below the lower threshold, the edge device 30 excludes the product having the similarity from the candidate products.

[0025] When the imaging device 20 receives the recognition result of the package recognition from the edge device 30, it outputs the recognition result to the POS terminal 10. Note that the imaging device 20 may output the recognition result of the code recognition in its own device and the recognition result of the package recognition received from the edge device 30 to the POS terminal 10 in a distinguishable state. For example, the imaging device 20 may output a flag value that distinguishes between the recognition methods of code recognition and package recognition together with the recognition result, thereby outputting both recognition methods in a distinguishable manner.

[0026] In this embodiment, the edge device 30 outputs the package recognition result to the POS terminal 10 via the imaging device 20, but the present invention is not limited to this, and the edge device 30 may output the package recognition result directly to the POS terminal 10. In this case, the edge device 30 stores, for example, setting information that associates the IP addresses of the paired POS terminal 10 and imaging device 20, and transmits the package recognition result to the POS terminal 10 corresponding to the imaging device 20 that is the transmission source of the captured image.

[0027] The store server 40 is installed, for example, in the back yard of a store. The store server 40 stores a product master (not shown) that associates the product name, price, product image, etc. of each product sold in the store with the product code of the product. The store server 40 also provides the product master to each POS terminal 10. For example, the store server 40 transmits the product master to each POS terminal 10, and stores the product master locally in the POS terminal 10. For example, the store server 40 also stores the product master so that each POS terminal 10 can refer to it.

[0028] The store server 40 also collects sales data received from the POS terminal 10. The store server 40 manages the sales of products in the store based on the collected sales data.

[0029] The POS terminal 10 cooperates with other devices to perform registration processing and payment processing for products sold in a store. Specifically, the POS terminal 10 is operated by a customer and executes a registration processing to register a product as a purchase target, the product being identified based on an image captured by the imaging device 20 when the customer holds the product over the imaging device 20. In addition, in response to an operation to end the registration processing, the POS terminal 10 executes a payment processing to settle the price of the product registered up to that point with the amount to be paid by the customer.

[0030] One or more POS terminals 10 (three in FIG. 1) are installed in the checkout area of ​​a store, for example. The POS terminal 10 is a product sales data processing device that is operated by the customer himself. When a customer performs a registration operation by holding the product to be purchased over the imaging device 20, the POS terminal 10 executes a product registration process for the product to be purchased by the customer, based on the product code of the product recognized by the above-mentioned code recognition or package recognition. The product that is the subject of the registration operation is an example of a target product. The product registration process refers to a process of reading product information (product name, product price, etc.) such as the product name and price of the product corresponding to the product code based on the product code from the product master, and storing it in RAM 13, etc.

[0031] In addition, the POS terminal 10 executes a payment process for the products registered in the product registration process in response to an operation to instruct the end of the product registration process or an operation to instruct the start of a payment process. The payment process refers to a closing process for the transaction based on the product information registered in the product registration process, specifically, displaying the total amount on the display unit 16, payment processing by media such as cash, credit card, and electronic money, calculating the change based on the deposit in the case of a cash payment and displaying it on the display unit 16, and issuing a receipt from the printer 18 described later on, on which product information and accounting information (total amount, deposit amount, change amount, point information, etc.) of the products for which payment has been processed are printed.

[0032] The POS terminal 10 also transmits product information and transaction information (collectively referred to as sales data) of the products for which payment has been processed to the store server 40. In this embodiment, the timing of transmission of sales data from the POS terminal 10 to the store server 40 is when the payment process is completed, but this is not limited thereto, and for example, a day's worth of sales data may be transmitted collectively when the store closes.

[0033] As described above, in the product sales data processing system 1 of this embodiment, product identification based on captured images is performed by combining code recognition and package recognition. Each POS terminal 10 performs product registration processing using the product code identified first among the results of code recognition and package recognition performed using the captured image of the imaging device 20 connected to the POS terminal 10 itself.

[0034] Next, the configuration of each device included in the product sales data processing system 1 will be described.

[0035] Fig. 3 is a diagram showing an example of a hardware configuration of the POS terminal 10. As shown in Fig. 3, the POS terminal 10 includes a CPU (Central Processing Unit) 11, which is an example of a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a memory unit 14, and the like.

[0036] The CPU 11 has one or more processing circuits and controls the overall operation of the POS terminal 10. The ROM 12 stores various programs that the CPU 11 can execute. The RAM 13 is a work area for temporarily storing programs and various data. For example, the RAM 13 stores the recognition results of code recognition or package recognition, product codes of products that have been registered, etc.

[0037] The memory unit 14 is composed of a non-volatile memory such as a hard disk drive (HDD) or a flash memory that retains stored information even when the power is turned off. The memory unit 14 stores various programs that the CPU 11 can execute, setting information, and the like. For example, the memory unit 14 stores a control program 141 that the CPU 11 can execute, a product master 142, and the like. The product master 142 stores product information, etc., of a product corresponding to a product code in association with the product code of each product sold in the store.

[0038] Fig. 4 is a diagram showing an example of the data configuration of the product master 142. As shown in Fig. 4, the product master 142 stores, in association with a product code of each product, product information of the product corresponding to the product code and a symbol flag.

[0039] The product information includes various information related to the product. For example, the product information includes the product name, price, and product image. The product image is image data representing the appearance of the product. The symbol flag is binary flag information indicating whether or not a symbol is attached to the product.

[0040] Products with symbols are subject to code recognition and may also be subject to package recognition. On the other hand, products without symbols are not subject to code recognition and can only be recognized by package recognition. Examples of products without symbols include fresh foods and prepared foods produced and sold in stores.

[0041] 3, the CPU 11, ROM 12, RAM 13, and memory unit 14 are connected to one another via a bus or the like. Here, the CPU 11, ROM 12, and RAM 13 constitute a control unit 100 of the POS terminal 10. The control unit 100 realizes the functions of the POS terminal 10 by the CPU 11 operating in accordance with a control program 141 stored in the ROM 12 and memory unit 14 and expanded in the RAM 13.

[0042] In addition, the control unit 100 is connected to an operation unit 15, a display unit 16, a card reader 17, a printer 18, etc. via a bus or the like.

[0043] The operation unit 15 has various operation keys that allow the operator to input operations. The operation unit 15 has a closing key as an example of an operation key. The closing key is an operation key that is operated when terminating the product registration process in the POS terminal 10 and proceeding to the payment process. The display unit 16 is, for example, a liquid crystal display, and displays information to the customer who is the operator of the POS terminal 10. Under the control of the control unit 100, the display unit 16 displays, for example, product information (product name, product price, etc.) of the product that has been registered, and accounting information related to the payment process. The operation unit 15 may be a touch panel provided on the display unit 16.

[0044] The card reader 17 is a reading device for reading card information related to a payment from a card, for example, a credit card, a debit card, an electronic money card, etc. The printer 18 is a printing device, for example, a thermal printer, etc. Under the control of the control unit 100, the printer 18 issues a receipt on which product information related to the product for which payment has been processed, payment information, etc. are printed.

[0045] The control unit 100 is also connected to a communication unit 19 via a bus or the like. The communication unit 19 is a variety of communication interfaces for connecting to external devices. The communication unit 19 is communicatively connected to the imaging device 20 via a cable U. The communication unit 19 is also communicatively connected to the edge device 30, the store server 40, and the like via a communication line L.

[0046] Next, a description will be given of a hardware configuration of the imaging device 20. Fig. 5 is a diagram showing an example of a hardware configuration of the imaging device 20. As shown in Fig. 5, the imaging device 20 includes a CPU 21, which is an example of a processor, a ROM 22, a RAM 23, a memory unit 24, and the like.

[0047] The CPU 21 has one or more processing circuits and comprehensively controls the operation of the imaging device 20. The ROM 22 stores various programs that the CPU 21 can execute. The RAM 23 is a work area for temporarily storing programs and various data. For example, the RAM 23 stores images captured by the imaging unit 25 (described later), intermediate data of code recognition, processing results (recognition results), etc.

[0048] The memory unit 24 is composed of a non-volatile memory such as a HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 24 stores various programs and setting information that can be executed by the CPU 21. For example, the memory unit 24 stores a control program 241 that can be executed by the CPU 21.

[0049] The CPU 21, the ROM 22, the RAM 23, and the memory unit 24 are connected to one another via a bus or the like. Here, the CPU 21, the ROM 22, and the RAM 23 constitute a control unit 200 of the imaging device 20. The control unit 200 realizes the functions of the imaging device 20 by the CPU 21 operating in accordance with a control program 241 stored in the ROM 22 or the memory unit 24 and expanded in the RAM 23.

[0050] The control unit 200 is also connected to an imaging unit 25 and a communication unit 26 via a bus or the like. The imaging unit 25 is a camera module equipped with an imaging element such as a CCD. The communication unit 26 is a variety of communication interfaces for connecting to external devices. The communication unit 26 is connected to the POS terminal 10 via a cable U so as to be able to communicate with the POS terminal 10.

[0051] Next, a description will be given of a hardware configuration of the edge device 30. Fig. 6 is a diagram illustrating an example of a hardware configuration of the edge device 30. As illustrated in Fig. 6, the edge device 30 includes a CPU 31, which is an example of a processor, a ROM 32, a RAM 33, a memory unit 34, and the like.

[0052] The CPU 31 has one or more processing circuits and comprehensively controls the operation of the edge device 30. The ROM 32 stores various programs that the CPU 31 can execute. The RAM 33 is a work area for temporarily storing programs and various data. For example, the RAM 33 stores images transmitted from the imaging device 20, intermediate data of package recognition, processing results (recognition results), etc.

[0053] The memory unit 34 is composed of a non-volatile memory such as an HDD or a flash memory that retains stored information even when the power is turned off. The memory unit 34 stores various programs and setting information that can be executed by the CPU 31. For example, the memory unit 34 stores a control program 341 that can be executed by the CPU 31. The memory unit 34 also stores the above-mentioned object detection model 51, feature extraction model 52, matching dictionary 53, and the like, as well as threshold setting information 342 that sets a threshold that serves as an index for uniquely identifying a product based on the similarity.

[0054] The CPU 31, the ROM 32, the RAM 33, and the memory unit 34 are connected to one another via a bus or the like. Here, the CPU 31, the ROM 32, and the RAM 33 constitute a control unit 300 of the edge device 30. The control unit 300 realizes the functions of the edge device 30 by the CPU 31 operating in accordance with a control program 341 stored in the ROM 32 and the memory unit 34 and loaded in the RAM 33.

[0055] The control unit 300 is connected to a communication unit 35 via a bus or the like. The communication unit 35 is a variety of communication interfaces for connecting to external devices. The communication unit 35 is connected to the POS terminal 10, the imaging device 20, and the store server 40 via a communication line L so as to be able to communicate with them.

[0056] Next, a description will be given of the hardware configuration of the store server 40. Fig. 7 is a diagram showing an example of the hardware configuration of the store server 40. As shown in Fig. 7, the store server 40 includes a CPU 41, which is an example of a processor, a ROM 42, a RAM 43, a memory unit 44, and the like.

[0057] The CPU 41 has one or more processing circuits and generally controls the operation of the store server 40. The ROM 42 stores various programs that the CPU 41 can execute. The RAM 43 is a work area for temporarily storing programs and various data. For example, the RAM 43 temporarily stores accounting data sent from the POS terminal 10.

[0058] The memory unit 44 is composed of a non-volatile memory such as a HDD or flash memory that retains stored information even when the power is turned off. The memory unit 44 stores various programs that the CPU 41 can execute, setting information, and the like. For example, the memory unit 44 stores a control program 441 that the CPU 41 can execute. The memory unit 44 also stores a product master 442, a transaction history DB (database) 443 for storing and managing accounting data, and the like. The product master 442 is the same as the product master 142. The product master 442 is provided to each of the POS terminals 10, and the product master 442 is used as the product master 142 in each of the POS terminals 10.

[0059] The CPU 41, ROM 42, RAM 43, and memory unit 44 are connected to one another via a bus or the like. Here, the CPU 41, ROM 42, and RAM 43 constitute a control unit 400 of the store server 40. The control unit 400 realizes the functions of the store server 40 by the CPU 41 operating in accordance with a control program 441 stored in the ROM 42 and memory unit 44 and loaded into the RAM 43.

[0060] The control unit 400 is connected to a communication unit 45 via a bus or the like. The communication unit 45 is a variety of communication interfaces for connecting to external devices. The communication unit 45 is connected to the POS terminal 10 or the like via a communication line L so as to be able to communicate with the POS terminal 10 or the like.

[0061] Next, a description will be given of the functional configurations of the POS terminal 10, the imaging device 20, and the edge device 30. FIG.

[0062] As shown in FIG. 8, the imaging device 20 includes an image acquisition unit 201, an image transmission unit 202, a code recognition unit 203, and a recognition result output unit 204 as functional components.

[0063] The functional configuration of the imaging device 20 is a software configuration realized by the processor of the imaging device 20 operating in accordance with the control program 241 stored in the ROM 22 or the memory unit 24, but is not limited to this. For example, a part or all of the functional configuration of the imaging device 20 may be realized by hardware such as a dedicated circuit.

[0064] The image acquisition unit 201 acquires images captured by the imaging unit 25. The image acquisition unit 201 may extract and acquire images in which a commodity is shown or images in which a commodity may be shown, from images captured by the imaging unit 25, using a known technique such as motion detection.

[0065] The image transmission unit 202 transmits the image acquired by the image acquisition unit 201 to the edge device 30. The code recognition unit 203 attempts to detect a symbol such as a barcode from the image acquired by the image acquisition unit 201 by executing code recognition on the image acquired by the image acquisition unit 201. When the code recognition unit 203 detects a symbol from the image, it decodes the symbol and acquires decoded information including a product code as the recognition result of the code recognition.

[0066] The recognition result output unit 204 outputs the recognition result obtained by the code recognition by the code recognition unit 203 to the POS terminal 10. In addition, the recognition result output unit 204 outputs the recognition result of the package recognition received from the edge device 30 to the POS terminal 10. Note that the recognition result output unit 204 may output the recognition results of the code recognition and the package recognition together with a flag value indicating the recognition method, so as to be identifiable.

[0067] As shown in FIG. 8, the edge device 30 includes an image acquisition unit 301, a package recognition unit 302, and a recognition result output unit 303 as functional components.

[0068] The functional configuration of the edge device 30 is a software configuration realized by the processor of the edge device 30 operating in accordance with the control program 341 stored in the ROM 32 or the memory unit 34, but is not limited to this. For example, some or all of the functional configuration of the edge device 30 may be realized by hardware such as a dedicated circuit.

[0069] The image acquisition unit 301 acquires images transmitted from each of the imaging devices 20 via the communication unit 45. The package recognition unit 302 performs the above-mentioned package recognition on the image acquired by the image acquisition unit 301, thereby attempting to detect an object image from the image and extract features. When the package recognition unit 302 confirms that there is one product in the matching dictionary 53 whose similarity is equal to or greater than the threshold based on the features extracted from the image, it outputs the product code of the product as a recognition result. Furthermore, when there is no product whose similarity is equal to or greater than the threshold, the package recognition unit 302 extracts a predetermined number of products with high similarity from the matching dictionary 53, and outputs a recognition result in which the similarity of the product is associated with the product code of each extracted product.

[0070] If there are multiple products whose similarity is equal to or greater than the threshold, the package recognition unit 302 outputs a recognition result that associates the product codes of those products with the similarity of the products.

[0071] The recognition result output unit 303 outputs the recognition result of the package recognition executed on the image to an external device related to the transmission source of the image. Specifically, the recognition result output unit 303 transmits the recognition result of the package recognition executed on the image to the imaging device 20 that transmitted the image via the communication unit 45. Note that the recognition result output unit 303 may transmit the recognition result to the POS terminal 10 corresponding to the imaging device 20 that transmitted the image.

[0072] As shown in FIG. 8, the POS terminal 10 includes a recognition result acquisition unit 101, a product registration processing unit 102, and a payment processing unit 103.

[0073] The functional configuration of the POS terminal 10 is a software configuration realized by the processor of the POS terminal 10 operating in accordance with the control program 141 stored in the ROM 12 or the memory unit 14, but is not limited to this. For example, a part or all of the functional configuration of the POS terminal 10 may be realized by hardware such as a dedicated circuit.

[0074] The recognition result acquisition unit 101 is an example of a first acquisition unit and a second acquisition unit. The acquisition unit 101 acquires the recognition results of the code recognition and package recognition for the POS terminal 10 itself, which are performed by the imaging device 20 and the edge device 30. Specifically, the recognition result acquisition unit 101 acquires the recognition results of the code recognition and package recognition transmitted from the imaging device 20 via the communication unit 19. Note that in the case of a configuration in which the recognition result of the package recognition is directly transmitted from the edge device 30 (recognition result output unit 303) to the POS terminal 10 (see the dashed line in the figure), the recognition result acquisition unit 101 may acquire the recognition result of the package recognition from the edge device 30.

[0075] The product registration processing unit 102 is an example of a registration means and a display means. The product registration processing unit 102 executes a product registration process for a product purchased by a customer, based on the recognition result acquired by the recognition result acquisition unit 101. In other words, the product registration processing unit 102 executes a product registration process based on the recognition result of the code recognition of the image of the product and the recognition result of the package recognition.

[0076] Specifically, when a product is uniquely identified by the recognition result of code recognition or package recognition, that is, when the recognition result includes the product code of a product, the product registration processing unit 102 executes product registration processing for the product. Note that the product registration processing unit 102 speeds up the processing by executing the product registration processing using the product code of the recognition result obtained first from the code recognition or package recognition.

[0077] In addition, when the recognition result of the code recognition is not obtained but the recognition result of the package recognition including multiple product codes and their similarities is obtained, the product registration processing unit 102 attempts to perform product registration processing based on the recognition result.

[0078] In the conventional product registration process using generic object recognition, when multiple product codes are recognized as candidate products, a screen on which these candidate products can be selected (hereinafter, also referred to as a product selection screen) is displayed on a display device, and the operator is left to select the correct product. However, the operation of selecting the correct product from multiple candidate products that have similar appearances is troublesome, and is a time-consuming task, especially for operators who are not familiar with the operation. For these reasons, it is desirable to reduce the number of times the product selection screen is displayed in order to improve the efficiency and speed of the product registration process.

[0079] Therefore, the product registration processing unit 102 of the present embodiment aims to speed up the product registration process by reducing the number of times the product selection screen is displayed, even when multiple candidate products are recognized.

[0080] Specifically, the product registration processing unit 102 determines whether or not a symbol is attached to any of the candidate products based on information other than the image captured by the imaging device 20, and controls whether to continue code recognition and package recognition or display a product selection screen depending on the determination result.

[0081] In this embodiment, the product registration processing unit 102 judges whether or not a symbol is attached to any of the candidate products by referring to the symbol flag in the product master 142. Here, the candidate products to be judged for the presence or absence of a symbol can be arbitrarily set, but the candidate products to be judged may be determined based on the similarity included in the recognition result of the package recognition.

[0082] For example, the product registration processing unit 102 refers to the similarity corresponding to each of the product codes and identifies the product code with the highest similarity. Next, the product registration processing unit 102 refers to the symbol flag of the identified product code in the product master 142 and determines whether a symbol is attached to the product corresponding to the product code.

[0083] Here, if it is determined that a symbol is attached, the product registration processing unit 102 waits until the product is uniquely identified by code recognition (or package recognition) without displaying a product selection screen. In this case, the product registration processing unit 102 may display, on the display unit 16, a message prompting the customer to check whether a symbol is attached to the product or a message instructing the customer to point the symbol toward the imaging device 20. This makes it possible to prompt the customer to read the symbol when a symbol is attached to the product held by the customer, thereby speeding up the product registration process.

[0084] On the other hand, if it is determined that no symbol is attached, the product registration processing unit 102 causes the display unit 16 to display a product selection screen based on the product information corresponding to each of the product codes included in the recognition result.

[0085] Fig. 9 is a diagram showing an example of a product selection screen A displayed on the POS terminal 10. As shown in Fig. 9, the product selection screen A has a display area Aa for displaying products corresponding to each of the product codes included in the recognition result as candidate products.

[0086] In the display area Aa, the product name, price, product image, etc. of the candidate product are displayed based on the product information of the candidate product. In FIG. 9, four candidate products are shown, and a display area Aa is provided for each candidate product. Note that, although the arrangement position of the candidate products on the product selection screen A is not particularly important, it is preferable to arrange them according to the degree of similarity. For example, the products may be arranged in the order of the highest degree of similarity, from top left → top right → bottom left → bottom right.

[0087] The display area Aa also functions as an operator for selecting a product to be registered. When a customer selects one of the display areas Aa, the product registration processing unit 102 executes a product registration process based on the product code corresponding to the candidate product in the selected display area Aa. Then, the product registration processing unit 102 erases the product selection screen A.

[0088] Furthermore, the product selection screen A has a back button Ab. When the product registration processing unit 102 accepts the operation of the back button Ab, it erases the product selection screen A and waits until a new recognition result is acquired.

[0089] It should be noted that the candidate products for which a symbol flag indicating a symbol is registered may be configured not to be displayed on the product selection screen A. This allows the number of candidate products displayed on the product selection screen A to be reduced, thereby improving the recognition speed for recognizing candidate products and improving operability, thereby making the product registration process more efficient.

[0090] Furthermore, the product registration processing unit 102 may wait for the recognition result by code recognition even while displaying the product selection screen A. This allows the product registration processing unit 102 to continue the product registration process by code recognition even while the candidate products recognized by the package recognition are displayed on the product registration processing unit 102, so that when a product is uniquely identified by the code recognition, the product registration process can be performed promptly. In this case, it is preferable to display a message or the like on the product selection screen A notifying the user that the symbol can also be read.

[0091] Returning to Fig. 8, the payment processing unit 103 executes processing related to payment for the product registered in the product registration process (payment processing) in response to a customer's operation. The payment processing may be performed by the POS terminal 10 alone, or may be performed in cooperation with other devices (e.g., an accounting device, a payment server, etc.).

[0092] For example, when the payment processing unit 103 receives an operation to start payment via the operation unit 15, it displays a screen on the display unit 16 that allows the user to select a payment method such as cash payment, electronic money payment, or credit card payment. When cash payment is selected, the payment processing unit 103 executes a payment process to settle the total amount of the registered products with the payment amount deposited via a deposit device (not shown). When electronic money payment or credit card payment is selected, the payment processing unit 103 also receives input of payment information for making the payment using the selected payment method, and cooperates with an external payment server to execute a payment process to settle the total amount of the registered products based on the input payment information. Such payment information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, the input may be received via the imaging device 20.

[0093] Next, a description will be given of the flow of processing executed by the product sales data processing system 1. Fig. 10 is a sequence diagram showing an example of the flow of processing executed by the product sales data processing system 1.

[0094] First, a customer performs a registration operation in the POS terminal 10 by holding a product to be purchased over the imaging device 20. The image acquisition unit 201 of the imaging device 20 acquires an image captured by the imaging unit 25 (step S1). Next, the image transmission unit 202 of the imaging device 20 transmits the image obtained by imaging to the edge device 30 (step S2). In addition, the code recognition unit 203 of the imaging device 20 executes code recognition on the image obtained by imaging (step S3).

[0095] When an image is transmitted from the imaging device 20, the package recognition unit 302 of the edge device 30 performs package recognition on the image (step S4). When the product is uniquely recognized or multiple candidate products are recognized by the package recognition, the recognition result output unit 303 of the edge device 30 transmits the recognition result including the product code of the recognized product to the imaging device 20 (step S5).

[0096] When the recognition result output unit 204 of the imaging device 20 receives the recognition result from the edge device 30, it outputs the recognition result to the POS terminal 10 (step S6). Furthermore, when the product is uniquely recognized by the code recognition, the code recognition unit 203 outputs the recognition result including the product code of the recognized product to the POS terminal 10 (step S6).

[0097] When an image of a product is captured by a customer's registration operation, the product will enter the imaging range of the imaging device 20 starting from the edge of the product. Therefore, there is a high possibility that the recognition result of the package recognition will be output first, and the recognition result of the code recognition will be output later, to the POS terminal 10. However, the output order of the recognition results of the code recognition and the package recognition is not limited to this.

[0098] When the recognition result is input from the imaging device 20, the product registration processing unit 102 of the POS terminal 10 executes product registration processing based on the recognition result (step S7). Furthermore, the POS terminal 10, the imaging device 20, and the edge device 30 execute the above-mentioned processes of steps S1 to S7 every time a customer performs a product registration operation. Then, when the customer instructs to end the product registration, the payment processing unit 103 of the POS terminal 10 executes payment processing (step S8).

[0099] Next, the process performed by the POS terminal 10 will be described with reference to Fig. 11. Fig. 11 is a flow chart showing an example of a product registration process performed by the POS terminal 10. This process corresponds to step S7 in Fig. 10.

[0100] First, the recognition result acquisition unit 101 determines whether or not the recognition result has been acquired from the imaging device 20 (step S11). If the recognition result has not been acquired (step S11; No), the process proceeds to step S18. If the recognition result acquisition unit 101 has acquired the recognition result from the imaging device 20 (step S11; Yes), the process proceeds to step S12.

[0101] The product registration processing unit 102 determines whether or not the product can be uniquely identified based on the recognition result acquired in step S11 (step S12). Specifically, the product registration processing unit 102 determines whether or not one product code recognized by code recognition or package recognition is included in the recognition result.

[0102] Here, if it is determined that the product can be uniquely identified (step S12; Yes), the product registration processing unit 102 executes a product registration process based on the product code included in the recognition result (step S17), and proceeds to step S 18. When the product registration processing unit 102 executes the product registration process, it causes the display unit 16 to display a screen (also called a product registration screen) (not shown) showing information such as the product name and price of the registered products in the order of registration.

[0103] On the other hand, if it is determined that the product cannot be uniquely identified (step S12; No), that is, if the recognition result includes product codes of multiple candidate products, the process proceeds to step S13. In step S13, the product registration processing unit 102 extracts the product code of the candidate product with the highest similarity based on the similarity included in the recognition result (step S13). Next, the product registration processing unit 102 refers to the product master 142 and determines whether the symbol flag of the product code extracted in step S13 indicates the symbol "present" (step S14).

[0104] Here, if the symbol flag indicates that the symbol is "present" (step S14; Yes), the product registration processing unit 102 returns the process to step S11, thereby making it possible to obtain the recognition result of the code recognition.

[0105] On the other hand, if the symbol flag indicates the "absence" symbol (step S14; No), the product registration processing unit 102 causes the display unit 16 to display a product selection screen A on which one product can be selected from each of the candidate products based on the product information of each candidate product stored in the product master 142 (step S15). Next, the product registration processing unit 102 waits until one product is selected from the product selection screen A (step S15; No).

[0106] In addition, in the case where the product selection screen A is configured not to display candidate products with symbols, a process of acquiring the recognition result of the code recognition may be executed in the background of step S16. In this case, when the recognition result of the code recognition is acquired while the product selection screen A is displayed, the product registration processing unit 102 may immediately proceed to step S17 to execute the product registration process. In addition, when transitioning from the product selection screen A to step S17, it is preferable to inform the customer of the reason for the screen transition by displaying, for example, "Symbol has been recognized."

[0107] In step S16, when one product is selected from the product selection screen A (step S16; Yes), the product registration processing unit 102 executes the product registration process based on the product code of the selected product (step S17), and proceeds to step S18.

[0108] Next, the product registration processing unit 102 determines whether or not an operation to end the product registration process, in other words, an operation to start the payment process, has been performed (step S18). If no operation has been performed (step S18; No), the product registration processing unit 102 returns the process to step S11 to prepare for obtaining a new recognition result. If an operation has been received (step S18; Yes), the product registration processing unit 102 ends this process and transitions to the payment process (step S8 in FIG. 10).

[0109] As described above, when the POS terminal 10 cannot uniquely identify a product based on the recognition results of the code recognition and package recognition, the POS terminal 10 switches between continuing the code recognition and package recognition and displaying the product selection screen A depending on whether a symbol is attached to the candidate product with the highest similarity. This allows the POS terminal 10 to reduce the number of times that the product selection screen A is displayed during the product registration process, thereby reducing the number of times that the operator selects a candidate product, thereby improving the efficiency of the product registration process.

[0110] Furthermore, when a symbol is attached to the candidate product with the highest similarity, the POS terminal 10 continues the code recognition and package recognition and does not display the product selection screen A. As a result, when there is a high possibility that the product can be identified by the code recognition, the POS terminal 10 can continue the code recognition (and package recognition), thereby making it possible to efficiently recognize products and speed up the product registration process.

[0111] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each of the above-described devices. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. Note that the following mainly describes the differences from the above-described embodiment, and a detailed description of the common points with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0112] (Variation 1) In the above-described embodiment, the display of the product selection screen A is controlled based on the symbol flag of the candidate product with the highest similarity among the candidate products recognized by package recognition, but the conditions for this control are not limited to this.

[0113] For example, when the symbol flag of at least one candidate product recognized in the package recognition indicates the symbol "present," the product registration processing unit 102 may hide the product selection screen A and continue the code recognition and package recognition.

[0114] Also, for example, when the symbol flags of all candidate commodities recognized by package recognition indicate the "present" symbol, the product selection screen A may be hidden and the code recognition and package recognition may be continued. Also, for example, when the number of commodities with the symbol flags indicating the "present" symbol among the candidate commodities recognized by package recognition exceeds a predetermined threshold, such as when the number of commodities with the symbol flags indicating the "absent" symbol exceeds the number of commodities with the symbol flags indicating the "absent" symbol, the product registration processing unit 102 may hide the product selection screen A and continue the code recognition and package recognition.

[0115] In this way, the POS terminal 10 switches between continuing the code recognition and package recognition and displaying the product selection screen A depending on the state of the symbol flag of the candidate product. Therefore, the POS terminal 10 can flexibly switch the product registration method depending on the state of the symbol of the candidate product recognized in the package recognition, and can reduce the frequency of displaying the product selection screen A.

[0116] (Variation 2) In the above embodiment, a configuration has been described in which, when a candidate product is recognized by package recognition, it is determined whether or not a symbol is attached by referring to the symbol flag in the product master 142. In this modified example, a configuration will be described in which it is determined whether or not a symbol is attached based on information input by the operator as an example of information other than the image captured by the imaging device 20.

[0117] Specifically, when a plurality of candidate products are recognized by package recognition, the product registration processing unit 102 according to this modification displays a message or a confirmation screen for the customer to confirm whether or not the product has a symbol on the display unit 16. For example, the product registration processing unit 102 displays a confirmation screen shown in FIG.

[0118] Fig. 12 is a diagram showing an example of a confirmation screen displayed on the POS terminal 10. As shown in Fig. 12, the confirmation screen B has a message Ba for confirming whether or not a symbol is attached to the product, and a YES button Bb and a No button Bc for inputting the confirmation result.

[0119] The operator of the POS terminal 10 checks whether or not a symbol is attached to the product held over the imaging device 20 based on the confirmation screen B, and inputs the confirmation result via the YES button Bb or the No button Bc. When the product registration processing unit 102 receives the operation of the YES button Bb or the No button Bc, it acquires the operation content as the confirmation result.

[0120] Here, if the confirmation result indicates that the symbol is "present," the product registration processing unit 102 hides the product selection screen A and waits until the product is uniquely identified by code recognition (or package recognition). On the other hand, if the confirmation result indicates that the symbol is "absent," the product registration processing unit 102 displays the product selection screen A on the display unit 16 based on the product information of the candidate product. Note that in this case, the product registration processing unit 102 may be configured not to display candidate products with symbols on the product selection screen A.

[0121] As a result, the POS terminal 10 can reduce the number of times the product selection screen A is displayed, as in the above embodiment, even when a candidate product is recognized. In this modified example, when a candidate product is recognized, the confirmation screen B is displayed, but since the confirmation screen B is a binary selection operation of YES or NO, the operation is simpler than the selection operation performed on the product selection screen A. Even when the product selection screen A is displayed, the candidate products are narrowed down to those with the symbol "none", so the operation of selecting a product can be easily performed. Therefore, the POS terminal 10 of this modified example can improve the efficiency of the product registration process.

[0122] If the confirmation result indicates that the symbol is "present", the product registration processing unit 102 may refer to the symbol flag of each candidate product and determine whether there is only one candidate product with the symbol "present". In this case, when the product registration processing unit 102 determines that there is only one candidate product with the symbol "present", it may immediately execute product registration processing based on the product code of the candidate product, without waiting for the recognition result of code recognition (or package recognition).

[0123] Furthermore, even if the confirmation result indicates the "absent" symbol, the product registration processing unit 102 may refer to the symbol flag of each candidate product and determine whether there is only one candidate product with the "absent" symbol. In this case, when the product registration processing unit 102 determines that there is only one candidate product with the "absent" symbol, it may immediately execute product registration processing based on the product code of the candidate product without displaying the product selection screen A.

[0124] (Variation 3) In the above embodiment, the imaging device 20 performs code recognition, but the present invention is not limited to this. For example, the POS terminal 10 or the edge device 30 may perform the code recognition.

[0125] For example, when code recognition is performed in the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) an image obtained by imaging to the POS terminal 10. The POS terminal 10 also includes a functional configuration of a code recognition unit 203, and the code recognition unit 203 executes code recognition on the image transmitted from the imaging device 20.

[0126] Furthermore, for example, when code recognition is performed by the edge device 30, the edge device 30 has a functional configuration of a code recognition unit 203, and the code recognition unit 203 executes code recognition on the image transmitted from the imaging device 20. In this case, the recognition result output unit 303 of the edge device 30 outputs the recognition results of the code recognition and package recognition in order from the first recognized one.

[0127] In the above embodiment, the edge device 30 performs package recognition. However, the present invention is not limited to this, and the POS terminal 10 or the imaging device 20 may perform the package recognition.

[0128] For example, when package recognition is performed at the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) an image obtained by imaging to the POS terminal 10. The POS terminal 10 also includes a functional configuration of a package recognition unit 302, and the package recognition unit 302 executes package recognition on the image transmitted from the imaging device 20.

[0129] Furthermore, for example, when package recognition is performed by the imaging device 20, the imaging device 20 has a functional configuration of a package recognition unit 302, and executes package recognition on an image obtained by imaging by the package recognition unit 302. In this case, the recognition result output unit 204 of the imaging device 20 outputs the recognition results of the code recognition and package recognition in order from the one recognized first.

[0130] In the above embodiment, the edge device 30 is described as a single device, but the present invention is not limited to this. For example, the edge device 30 may be realized by a cloud (cloud computing) composed of a plurality of information processing devices. In addition, the store server 40 may have the functions of the edge device 30.

[0131] (Variation 4) In the above embodiment, the POS terminal 10 has been described as a device operated by the customer himself. However, the present invention is not limited to this, and the POS terminal 10 may be a device operated by a store clerk. In this case, the imaging device 20 captures images of products based on a registration operation by the store clerk.

[0132] (Variation 5) In the above embodiment, the POS terminal 10 is described as having one imaging device 20 (or imaging section 25). However, this is not limiting, and the POS terminal 10 may be configured to have a plurality of imaging devices 20 (or imaging sections 25). In this case, code recognition may be performed by each imaging device 20, or may be performed by a single imaging device 20. It is preferable that the plurality of imaging devices 20 (or imaging sections 25) are configured to image merchandise from different imaging positions. This allows the merchandise to be imaged from multiple angles, thereby improving the recognition speed and recognition rate.

[0133] The programs executed by each device in the above-described embodiments (and modified examples) are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0134] Furthermore, the programs executed by each device of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by each device of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0135] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. These new embodiments and their modifications can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0136] 1 System 10. POS terminals 20 Imaging device 30 Edge Devices 40 Store Server 101 Recognition result acquisition unit 102 Product registration processing unit 103 Payment processing unit 201 Image Acquisition Unit 202 Image transmission unit 203 Code Recognition Unit 204 Recognition result output unit 301 Image Acquisition Unit 302 Package Recognition Unit 303 Recognition result output unit [Prior art documents] [Patent documents]

[0137] [Patent Document 1] JP 2013-89258 A

Claims

1. a first acquisition means for acquiring a recognition result of a first recognition process in which a symbol including identification information capable of identifying a target commodity is read from an image of the target commodity captured by an imaging device; A second acquisition means for acquiring a recognition result of a second recognition process for estimating a product similar to the characteristics of the target product from a plurality of predetermined products based on the characteristics of the target product shown in the image; a registration means for registering the target product as a purchase target when the target product can be identified by the recognition result acquired earlier out of the recognition result of the first recognition process and the recognition result of the second recognition process; Equipped with The registration means, when it determines that the recognition result of the second recognition process includes a plurality of candidate products that are candidates for the target product and that the symbol is affixed to any of the candidate products based on information other than the image, waits to obtain a new recognition result.

2. The product registration device according to claim 1 , wherein the registration means determines whether or not the symbol is affixed to the candidate product based on setting information in which a setting value indicating whether or not the symbol is affixed to each of the plurality of products is set.

3. a display unit that displays a confirmation screen for an operator to confirm whether or not the target commodity has the symbol when a recognition result of the second recognition process includes a plurality of the candidate commodities, The product registration device according to claim 1 , wherein the registration means determines whether or not the symbol is affixed to the candidate product based on information input by the operator on the confirmation screen.

4. the recognition result of the second recognition process includes a similarity indicating a degree of similarity of the candidate product to the target product; The product registration device according to claim 1 , wherein the registration means waits for acquisition of a new recognition result when the symbol is affixed to a candidate product having the highest similarity among the candidate products included in the recognition result of the second recognition process.

5. 2. The product registration device according to claim 1, wherein, when it is determined that the symbol is not affixed to all of the candidate products or to the candidate products that meet a predetermined condition, the registration means displays a selection screen on which one product can be selected from among the candidate products, and registers the product selected from the selection screen as the product to be purchased.

6. The computer of the product registration device, a first acquisition means for acquiring a recognition result of a first recognition process in which a symbol including identification information capable of identifying a target commodity is read from an image of the target commodity captured by an imaging device; A second acquisition means for acquiring a recognition result of a second recognition process for estimating a product similar to the characteristics of the target product from a plurality of predetermined products based on the characteristics of the target product shown in the image; a registration means for registering the target product as a purchase target when the target product can be identified by the recognition result acquired earlier out of the recognition result of the first recognition process and the recognition result of the second recognition process; and make it work. The registration means is a program that waits for acquisition of a new recognition result when the recognition result of the second recognition process includes a plurality of candidate products that are candidates for the target product and when it is determined that the symbol is affixed to any of the candidate products based on information other than the image.

Images