Product registration device and program
The product registration device enhances POS system efficiency by using code and package recognition to automatically register products when possible, and displaying a selection screen only when necessary, addressing the inefficiency of manual selection from similar candidates.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOSHIBA TEC KK
- Filing Date
- 2023-07-07
- Publication Date
- 2026-06-16
AI Technical Summary
Existing POS systems require operators to manually select the correct product from multiple similar candidates, which is time-consuming and inefficient, especially for those unfamiliar with the system.
A product registration device and program that utilizes a combination of code recognition and package recognition, where the system registers the product if it can be uniquely identified, otherwise, it displays a selection screen allowing the user to choose, reducing the need for manual selection when multiple candidates are present.
Improves the efficiency of product registration by minimizing the frequency of manual selection screens, thereby speeding up the process and reducing operator workload.
Smart Images
Figure 0007874587000001 
Figure 0007874587000002 
Figure 0007874587000003
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to a product registration device and a program.
Background Art
[0002] Conventionally, in stores such as shopping centers, supermarkets, department stores, etc., there is a POS system that performs product registration processing and processing of products sold in the store. Some such POS systems recognize products from the features of images of products captured by an imaging device and perform product registration processing based on the recognition results.
[0003] [[ID=1十六条]]The above-described product recognition processing is performed by comparing the appearance features of each product sold in the store with the appearance features of the product extracted from the image. In this case, the recognition result is output as the similarity between each product sold in the store and the captured product, and the product with a similarity of a threshold value or more (for example, 95% or more) is uniquely identified as the product that was the target of imaging.
[0004] Also, when there is no product with a similarity of a threshold value or more, a predetermined number of products with the highest similarity are extracted as candidate products, and a screen (hereinafter, also referred to as a product selection screen) that allows one product to be selected from the candidate products is displayed, leaving the selection of the correct product to the operator.
[0005] However, the operation of selecting the correct product from among a plurality of candidate products with similar appearances is troublesome and time-consuming for an operator who is not used to the operation. Under such circumstances, in order to improve the efficiency of product registration processing, it is desired to reduce the opportunity to display the product selection screen.
Summary of the Invention
Problems to be Solved by the Invention
[0006] The problem to be solved by the present invention is to provide a product registration device and a program capable of improving the efficiency of processing related to product registration.
Means for Solving the Problems
[0007] The product registration device of the embodiment includes: a first acquisition means for acquiring the recognition result of a first recognition process which reads a symbol containing identification information that can identify the target product from an image of the target product captured by an imaging device; a second acquisition means for acquiring the recognition result of a second recognition process which estimates a product similar to the external features of the target product shown in the image from a predetermined number of products; and a registration means for registering the target product as a purchase target if the target product can be identified by the recognition result obtained earlier from the recognition result of the first recognition process and the recognition result of the second recognition process, wherein the registration means determines that the symbol is attached to one of the candidate products based on the recognition result of the second recognition process which includes a plurality of candidate products that are candidates for the target product, The registration process for the target product is suspended until a recognition result that uniquely identifies the target product is obtained. If it is determined that none of the candidate products or candidate products that meet the specified conditions have the symbol attached, a selection screen is displayed allowing the user to select one product from the candidate products, and the product selected from the selection screen is registered as the target product for purchase. [Brief explanation of the drawing]
[0008] [Figure 1] Figure 1 shows an example of a product sales data processing system according to an embodiment. [Figure 2] Figure 2 is a diagram illustrating an example of package recognition according to this embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of a POS terminal according to the embodiment. [Figure 4] Figure 4 shows an example of the data structure of a product master according to an embodiment. [Figure 5] Figure 5 shows an example of the hardware configuration of the imaging device according to this embodiment. [Figure 6] Figure 6 shows an example of the hardware configuration of an edge device according to the embodiment. [Figure 7] Figure 7 shows an example of the hardware configuration of a store server according to this embodiment. [Figure 8] Figure 8 shows an example of the functional configuration of a POS terminal, imaging device, and edge device according to an embodiment. [Figure 9] Figure 9 shows an example of a product selection screen displayed on a POS terminal according to the embodiment. [Figure 10] Figure 10 is a sequence diagram showing an example of the processing flow performed in the product sales data processing system of the embodiment. [Figure 11] Figure 11 is a flowchart showing an example of the product registration process performed by the POS terminal of the embodiment. [Figure 12] Figure 12 shows an example of a confirmation screen displayed on the POS terminal of the embodiment. [Modes for carrying out the invention]
[0009] Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below will be described using a product sales data processing device in which the customer themselves performs the registration operation of the product to be purchased. However, the present invention is not limited to the embodiments described below.
[0010] Figure 1 shows an example of a product sales data processing system according to an embodiment. In Figure 1, the product sales data processing system 1 is installed in a store such as a supermarket, mass retailer, convenience store, or specialty store. The product sales data processing system 1 comprises one or more POS (Point of Sales) terminals 10 (product sales data processing devices), an imaging device 20, an edge device 30, and a store server 40.
[0011] The POS terminal 10, edge device 30, and store server 40 are connected to a communication line L, such as a LAN (Local Area Network), which allows for mutual notification. One or more imaging devices 20 are connected to each POS terminal 10, for example, via a USB (Universal Serial Bus) cable U. The imaging devices 20 are also connected to the communication line L. Alternatively, the imaging devices 20 may be connected to the communication line L via the POS terminal 10.
[0012] The imaging device 20 is a device equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging device 20 is, for example, positioned facing the customer operating the POS terminal 10 and captures images of the products the customer purchases. Specifically, in this embodiment, the customer operating the POS terminal 10 performs a registration process by holding each product they purchase up to the imaging device 20, causing the imaging device 20 to capture an image of the product.
[0013] When the imaging device 20 acquires an image (hereinafter also referred to as the acquired image) through imaging, it outputs the acquired image to the edge device 30. Specifically, the imaging device 20 outputs the image, which was captured at a predetermined frame rate, to the edge device 30.
[0014] Furthermore, the imaging device 20 attempts to detect symbols such as barcodes and two-dimensional codes from the image obtained by imaging by performing code recognition on the image. Code recognition detects symbols such as barcodes and two-dimensional codes attached to the product from the image, and outputs the decoded information obtained by decoding the detected symbols to the POS terminal 10. The decoded information includes product identification information that identifies the product to which the symbol is attached, so the imaged product can be identified based on this product identification information. In this embodiment, a product code that can identify each product is used as the product identification information, but other information that can identify the product may be used instead of a product code.
[0015] Furthermore, the imaging device 20 may extract images containing or potentially containing the product from the images obtained through imaging, and use them for output to the edge device 30 or for symbol extraction processing. In this case, the image extraction method is not particularly limited, and known techniques can be used. For example, the imaging device 20 may detect the movement of an object contained in an image from the difference between multiple temporally consecutive frames, and extract the image in which movement has been detected for processing.
[0016] The edge device 30 is an information processing device that performs predetermined processes such as image processing and image recognition. Specifically, the edge device 30 performs package recognition to recognize a product from an image input from the imaging device 20.
[0017] Package recognition is a technology for identifying a product imaged by the imaging device 20 based on the external features of the product represented in the image. Here, the external features of the product include, for example, the package design, pattern, color tone, texture, shape, and the like.
[0018] Package recognition can use known technologies such as general object recognition and image recognition using a model such as deep learning. In package recognition, based on the feature amount indicating the external features of the product extracted from the image, among the products similar to the feature amount, one product whose similarity (also expressed as likelihood or the like) is equal to or greater than a threshold value is identified as the product that is the subject.
[0019] FIG. 2 is a diagram for explaining an example of package recognition. As shown in FIG. 2, in package recognition, three types of learned models or dictionary data (hereinafter also collectively referred to as a dictionary), namely, an object detection model 51, a feature extraction model 52, and a reference dictionary 53, are used to identify a product from the image captured by the imaging device 20.
[0020] The object detection model 51 is a dictionary for detecting the region of an object shown in the captured image. When the captured image captured by the imaging device 20 is input, the object detection model 51 detects and outputs the region (hereinafter also referred to as an object image) in which the product is represented from the captured image.
[0021] The feature extraction model 52 is a dictionary for extracting, from the product represented in the region detected by the object detection model 51, a feature amount indicating the external features of the product. When an object image or a captured image including the object image is input, the feature extraction model 52 extracts and outputs a feature amount indicating the external features of the product represented in the object image.
[0022] The matching dictionary 53 is a dictionary used to identify products similar to a given product's features. For example, the matching dictionary 53 stores the features that represent the characteristics of each product, associated with the product code that identifies each product, for multiple products sold in a store. The matching dictionary 53 also calculates a similarity score, which quantitatively indicates how similar the stored products are to the photographed product, by comparing the input features with the stored features. Products with a higher similarity score are more similar to the photographed product (i.e., have a higher probability of being the photographed product). For example, products with a similarity score of 95% or higher have a very high probability of being the same product as the photographed product.
[0023] The edge device 30 uses an object detection model 51 to detect the region containing the product from the image captured by the imaging device 20. Next, the edge device 30 uses a feature extraction model 52 to extract feature quantities that quantitatively represent the external features of the product within that region (object image) using multidimensional vectors or the like. Then, the edge device 30 uses a matching dictionary 53 to calculate the similarity by comparing the extracted feature quantities with the feature quantities of numerous products stored in the dictionary, and recognizes the product according to the similarity.
[0024] Specifically, if there is one product whose similarity is above a threshold (for example, 95% or more), the edge device 30 identifies that product as the product captured by the imaging device 20. If there is no product whose similarity is above the threshold (for example, if the similarity of any product is less than 95%), the edge device 30 does not identify a single product, but instead selects multiple products with high similarity as candidate products. For example, if the similarity of any product is below the threshold, the edge device 30 extracts a predetermined number (for example, 4) of candidate products with high similarity. The edge device 30 then uses the product codes of the identified product or candidate products as the recognition result for package recognition and transmits them to the imaging device 20 that transmitted the captured image. If the edge device 30 recognizes multiple candidate products, it will include the calculated similarity for each candidate product in the recognition result and transmit it. The edge device 30 may also include the similarity of a uniquely identified product in the recognition result and transmit it if it is able to uniquely identify a product. Furthermore, the edge device 30 has a lower threshold for similarity determination, and if the similarity falls below the lower threshold, it excludes the product with that similarity from the list of candidate products.
[0025] When the imaging device 20 receives the package recognition result from the edge device 30, it outputs the recognition result to the POS terminal 10. The imaging device 20 may also output to the POS terminal 10 in a way that allows it to distinguish between the code recognition result from its own device and the package recognition result received from the edge device 30. For example, the imaging device 20 may output both recognition methods in an identifiable way by outputting a flag value that allows it to distinguish between the code recognition and package recognition methods along with the recognition result.
[0026] In this embodiment, the edge device 30 outputs the package recognition result to the POS terminal 10 via the imaging device 20, but it is not limited to this configuration, and may also output directly to the POS terminal 10. In this case, the edge device 30 stores, for example, configuration information associating the IP addresses of the paired POS terminal 10 and imaging device 20, and sends the package recognition result to the POS terminal 10 corresponding to the imaging device 20 that transmitted the captured image.
[0027] The store server 40 is installed, for example, in the back room of the store. The store server 40 stores a product master (not shown) associated with the product code of each product sold in the store, along with the product name, price, product image, etc. The store server 40 also provides the product master to each of the POS terminals 10. For example, the store server 40 sends the product master to each of the POS terminals 10, causing the POS terminals 10 to store the product master locally. Alternatively, for example, the store server 40 stores the product master so that each of the POS terminals 10 can access it.
[0028] The store server 40 also collects sales data received from the POS terminal 10. Based on the collected sales data, the store server 40 manages the sales of the store's products.
[0029] The POS terminal 10 works in cooperation with other devices to register and process payments for products sold in the store. Specifically, the POS terminal 10 performs a registration process to register products as items to be purchased, based on images captured by the imaging device 20 when the customer holds the product over the imaging device 20. In response to an operation to indicate the end of the registration process, the POS terminal 10 performs a payment process to settle the price of the registered products up to that point with the amount payable by the customer.
[0030] POS terminals 10 are installed, for example, in the checkout area of a store, one or more units (three units in Figure 1). POS terminals 10 are product sales data processing devices operated by the customer themselves. When a customer performs a registration operation by holding the product to be purchased over the imaging device 20, POS terminals 10 execute product registration processing for the product purchased by the customer based on the product code of the product recognized by the code recognition or package recognition described above. Note that the product subject to registration is just an example of a target product. Product registration processing refers to the process of reading product information (product name, price, etc.) corresponding to the product code from the product master and storing it in RAM 13, etc.
[0031] Furthermore, the POS terminal 10 executes the payment processing for the products registered in the product registration process in response to operations that instruct the completion of the product registration process or operations that instruct the start of the payment processing. Payment processing refers to closing processing related to the transaction based on the product information registered in the product registration process, specifically, displaying the total amount on the display unit 16, processing payment using media such as cash, credit card, or electronic money, calculating the change based on the deposit and displaying it on the display unit 16 in the case of cash payment, and issuing a receipt printed with product information and accounting information (total amount, deposit amount, change amount, point information, etc.) for the products that have been paid for using the printer 18, which will be described later.
[0032] The POS terminal 10 also transmits product information and accounting information (collectively referred to as sales data) of the processed payment items to the store server 40. In this embodiment, the transmission of sales data from the POS terminal 10 to the store server 40 is performed when the payment processing is completed, but it is not limited to this, and for example, the data for the day may be transmitted all at once when the store closes.
[0033] As described above, in the product sales data processing system 1 of this embodiment, product identification based on captured images is performed by using a combination of code recognition and package recognition. Each POS terminal 10 performs product registration processing using the product code identified first from the results of code recognition and package recognition performed using the captured image from the imaging device 20 connected to its own POS terminal 10.
[0034] Next, the configuration of each device in the product sales data processing system 1 will be described.
[0035] Figure 3 shows an example of the hardware configuration of a POS terminal 10. As shown in Figure 3, the POS terminal 10 includes a CPU (Central Processing Unit) 11, which is an example of a processor, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, a memory unit 14, etc.
[0036] The CPU 11 has one or more processing circuits and comprehensively controls the operation of the POS terminal 10. The ROM 12 stores various programs that the CPU 11 can execute. The RAM 13 is a work area for temporarily storing programs and various data. For example, the RAM 13 stores recognition results from code recognition or package recognition, and product codes of products that have been registered.
[0037] The memory unit 14 consists of non-volatile memory such as an HDD (Hard Disk Drive) or flash memory, which retains stored information even when the power is turned off. The memory unit 14 stores various programs and setting information that the CPU 11 can execute. For example, the memory unit 14 stores control programs 141 and product masters 142 that the CPU 11 can execute. The product master 142 stores product information for products corresponding to product codes of each product sold in stores, in association with the product codes of those products.
[0038] Figure 4 shows an example of the data structure of the product master 142. As shown in Figure 4, the product master 142 stores product information and symbol flags for each product, associated with the product code of that product.
[0039] Product information includes various types of information about a product. For example, product information includes the product name, price, and product image. The product image is image data that represents the appearance of the product. The symbol flag is binary flag information that indicates whether or not a symbol is attached to the product.
[0040] Products with a symbol attached can be subject to both code recognition and package recognition. On the other hand, products without a symbol attached are not subject to code recognition and can only be recognized by package recognition. Examples of products without a symbol include fresh food and prepared foods manufactured and sold in stores.
[0041] Returning to Figure 3, the CPU 11, ROM 12, RAM 13, and memory unit 14 are connected to each other via a bus or the like. Here, the CPU 11, ROM 12, and RAM 13 constitute the control unit 100 of the POS terminal 10. The control unit 100 realizes the functions of the POS terminal 10 by operating according to the control program 141 stored in the ROM 12 and memory unit 14 and loaded into the RAM 13.
[0042] Furthermore, the control unit 100 is connected to the operation unit 15, display unit 16, card reader 17, printer 18, etc., via a bus or the like.
[0043] The operation unit 15 has various operation keys for the operator to input operations. The operation unit 15 has a closing key as an example of an operation key. The closing key is an operation key that is operated when the product registration process is completed on the POS terminal 10 and the process proceeds to payment processing. The display unit 16 is, for example, a liquid crystal display and displays information to the customer who is the operator of the POS terminal 10. Under the control of the control unit 100, the display unit 16 displays, for example, product information of the registered product (product name, product price, etc.) and accounting information related to payment processing. The operation unit 15 may also be a touch panel provided on the display unit 16.
[0044] The card reader 17 is a reading device for reading card information related to payment from a card, such as a credit card, debit card, or electronic money card, when processing a payment. The printer 18 is a printing device, such as a thermal printer. Under the control of the control unit 100, the printer 18 issues a receipt that prints product information and payment information related to the processed payment.
[0045] Furthermore, the control unit 100 is connected to the communication unit 19 via a bus or the like. The communication unit 19 is a variety of communication interfaces for connecting to external devices. The communication unit 19 is connected to the imaging device 20 via cable U so as to be able to communicate. The communication unit 19 is also connected to the edge device 30 and the store server 40, etc. via communication line L so as to be able to communicate.
[0046] Next, the hardware configuration of the imaging device 20 will be described. Figure 5 is a diagram showing an example of the hardware configuration of the imaging device 20. As shown in Figure 5, the imaging device 20 includes a CPU 21, which is an example of a processor, ROM 22, RAM 23, memory unit 24, etc.
[0047] The CPU 21 has one or more processing circuits and comprehensively controls the operation of the imaging device 20. The ROM 22 stores various programs that the CPU 21 can execute. The RAM 23 is a work area for temporarily storing programs and various data. For example, the RAM 23 stores images captured by the imaging unit 25 (described later), intermediate data and processing results (recognition results) of code recognition, etc.
[0048] The memory unit 24 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off. The memory unit 24 stores various programs and setting information that the CPU 21 can execute. For example, the memory unit 24 stores control programs 241 that the CPU 21 can execute.
[0049] The CPU 21, ROM 22, RAM 23, and memory unit 24 are connected to each other via a bus or the like. Here, the CPU 21, ROM 22, and RAM 23 constitute the control unit 200 of the imaging device 20. The control unit 200 realizes the functions of the imaging device 20 by operating according to the control program 241 stored in the ROM 22 and memory unit 24 and loaded into the RAM 23.
[0050] Furthermore, the control unit 200 is connected to the imaging unit 25 and the communication unit 26 via a bus or the like. The imaging unit 25 is a camera module equipped with an image sensor such as a CCD. The communication unit 26 is a variety of communication interfaces for connecting to external devices. The communication unit 26 is connected to the POS terminal 10 via a cable U so as to be able to communicate.
[0051] Next, the hardware configuration of the edge device 30 will be described. Figure 6 is a diagram showing an example of the hardware configuration of the edge device 30. As shown in Figure 6, the edge device 30 includes a CPU 31, which is an example of a processor, ROM 32, RAM 33, memory unit 34, etc.
[0052] The CPU 31 has one or more processing circuits and comprehensively controls the operation of the edge device 30. The ROM 32 stores various programs that the CPU 31 can execute. The RAM 33 is a work area for temporarily storing programs and various data. For example, the RAM 33 stores images transmitted from the imaging device 20, intermediate data and processing results (recognition results) for package recognition, etc.
[0053] The memory unit 34 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off. The memory unit 34 stores various programs and setting information that the CPU 31 can execute. For example, the memory unit 34 stores a control program 341 that the CPU 31 can execute. In addition to the object detection model 51, feature extraction model 52, and matching dictionary 53 mentioned above, the memory unit 34 also stores threshold setting information 342, which sets thresholds that serve as indicators for uniquely identifying products based on similarity.
[0054] The CPU 31, ROM 32, RAM 33, and memory unit 34 are connected to each other via a bus or the like. Here, the CPU 31, ROM 32, and RAM 33 constitute the control unit 300 of the edge device 30. The control unit 300 realizes the functions of the edge device 30 by operating according to the control program 341 stored in the ROM 32 and memory unit 34 and loaded into the RAM 33.
[0055] Furthermore, the control unit 300 is connected to the communication unit 35 via a bus or the like. The communication unit 35 is a variety of communication interfaces for connecting to external devices. The communication unit 35 is connected to the POS terminal 10, the imaging device 20, and the store server 40 via a communication line L.
[0056] Next, the hardware configuration of the store server 40 will be described. Figure 7 shows an example of the hardware configuration of the store server 40. As shown in Figure 7, the store server 40 includes a CPU 41, which is an example of a processor, ROM 42, RAM 43, memory unit 44, etc.
[0057] The CPU 41 has one or more processing circuits and comprehensively controls the operation of the store server 40. The ROM 42 stores various programs that the CPU 41 can execute. The RAM 43 is a work area for temporarily storing programs and various data. For example, the RAM 43 temporarily stores accounting data transmitted from the POS terminal 10.
[0058] The memory unit 44 is composed of non-volatile memory such as an HDD or flash memory that retains stored information even when the power is turned off. The memory unit 44 stores various programs and setting information that the CPU 41 can execute. For example, the memory unit 44 stores a control program 441 that the CPU 41 can execute. The memory unit 44 also stores a product master 442 and a transaction history DB (database) 443 for storing and managing accounting data. The product master 442 is the same as the product master 142. By providing the product master 442 to each POS terminal 10, the product master 442 is used as the product master 142 in each POS terminal 10.
[0059] The CPU 41, ROM 42, RAM 43, and memory unit 44 are connected to each other via a bus or the like. Here, the CPU 41, ROM 42, and RAM 43 constitute the control unit 400 of the store server 40. The control unit 400 realizes the functions of the store server 40 by operating according to the control program 441 stored in the ROM 42 and memory unit 44 and loaded into the RAM 43.
[0060] Furthermore, the control unit 400 is connected to the communication unit 45 via a bus or the like. The communication unit 45 is a variety of communication interfaces for connecting to external devices. The communication unit 45 is connected to the POS terminal 10 or the like via a communication line L so as to be able to communicate.
[0061] Next, the functional configuration of the POS terminal 10, imaging device 20, and edge device 30 described above will be explained. Figure 8 shows an example of the functional configuration of the POS terminal 10, imaging device 20, and edge device 30.
[0062] As shown in Figure 8, the imaging device 20 includes, functionally, an image acquisition unit 201, an image transmission unit 202, a code recognition unit 203, and a recognition result output unit 204.
[0063] The functional configuration of the imaging device 20 is a software configuration realized by the processor of the imaging device 20 operating according to a control program 241 stored in the ROM 22 or memory unit 24, but is not limited to this. For example, some or all of the functional configuration of the imaging device 20 may be realized by hardware such as dedicated circuits.
[0064] The image acquisition unit 201 acquires images captured by the imaging unit 25. The image acquisition unit 201 may also use known techniques such as motion detection to extract and acquire images that contain products or images that may contain products from the images captured by the imaging unit 25.
[0065] The image transmission unit 202 transmits the image acquired by the image acquisition unit 201 to the edge device 30. The code recognition unit 203 attempts to detect symbols such as barcodes from the image by performing code recognition on the image acquired by the image acquisition unit 201. When the code recognition unit 203 detects a symbol from the image, it decodes the symbol and obtains decoded information containing the product code as the recognition result of the code recognition.
[0066] The recognition result output unit 204 outputs the recognition result obtained from the code recognition unit 203 to the POS terminal 10. The recognition result output unit 204 also outputs the package recognition result received from the edge device 30 to the POS terminal 10. The recognition result output unit 204 may output the recognition results of code recognition and package recognition together with a flag value indicating the recognition method, so that the results are identifiable.
[0067] As shown in Figure 8, the edge device 30 includes an image acquisition unit 301, a package recognition unit 302, and a recognition result output unit 303 as its functional configuration.
[0068] The functional configuration of the edge device 30 is a software configuration realized by the processor of the edge device 30 operating according to a control program 341 stored in the ROM 32 or memory unit 34, but is not limited to this. For example, some or all of the functional configuration of the edge device 30 may be realized by hardware such as dedicated circuits.
[0069] The image acquisition unit 301 acquires images transmitted from each of the imaging devices 20 via the communication unit 45. The package recognition unit 302 attempts to detect object images and extract features from the images by performing the package recognition described above on the images acquired by the image acquisition unit 301. Based on the features extracted from the image, the package recognition unit 302 confirms that there is one product in the matching dictionary 53 whose similarity is above a threshold, and outputs the product code of that product as the recognition result. If there are no products with similarity above a threshold, the package recognition unit 302 extracts a predetermined number of products with high similarity from the matching dictionary 53 and outputs a recognition result that associates the similarity of each product with the product code of each extracted product.
[0070] If there are multiple products with a similarity score above a threshold, the package recognition unit 302 outputs a recognition result that associates the product codes of those products with their similarity scores.
[0071] The recognition result output unit 303 outputs the recognition result of the package recognition performed on the image to an external device related to the source of the image. Specifically, the recognition result output unit 303 transmits the recognition result of the package recognition performed on the image to the image capture device 20, the source of the image, via the communication unit 45. Alternatively, the recognition result output unit 303 may transmit the recognition result to the POS terminal 10 corresponding to the source image capture device 20.
[0072] Furthermore, as shown in Figure 8, the POS terminal 10 includes a recognition result acquisition unit 101, a product registration processing unit 102, and a payment processing unit 103.
[0073] The functional configuration of the POS terminal 10 is a software configuration realized by the POS terminal 10's processor operating according to a control program 141 stored in the ROM 12 or memory unit 14, but is not limited to this. For example, some or all of the functional configuration of the POS terminal 10 may be realized by hardware such as dedicated circuits.
[0074] The recognition result acquisition unit 101 is an example of the first and second acquisition means. The acquisition unit 101 acquires the recognition results of code recognition and package recognition related to its own POS terminal 10, which are performed by the imaging device 20 and the edge device 30. Specifically, the recognition result acquisition unit 101 acquires the code recognition and package recognition results transmitted from the imaging device 20 via the communication unit 19. In the case where the package recognition result is transmitted directly from the edge device 30 (recognition result output unit 303) to the POS terminal 10 (see dashed line in the figure), the recognition result acquisition unit 101 may acquire the package recognition result from the edge device 30.
[0075] The product registration processing unit 102 is an example of a registration and display means. Based on the recognition results obtained by the recognition result acquisition unit 101, the product registration processing unit 102 performs product registration processing for the products purchased by the customer. In other words, the product registration processing unit 102 performs product registration processing based on the recognition results of code recognition of the image captured of the product and the recognition results of package recognition.
[0076] Specifically, the product registration processing unit 102 executes the product registration process for a product when the product is uniquely identified by the recognition result of code recognition or package recognition, that is, when the product code of a product is included in the recognition result. Furthermore, the product registration processing unit 102 aims to speed up the process by using the product code obtained first from the recognition result of code recognition or package recognition when performing the product registration process.
[0077] Furthermore, if the code recognition result is not obtained, but the package recognition result, which includes multiple product codes and their similarity scores, is obtained, the product registration processing unit 102 attempts to perform product registration processing based on this recognition result.
[0078] Incidentally, in conventional product registration processes using general object recognition, when multiple product codes are recognized as candidate products, a screen where these candidate products can be selected (hereinafter also referred to as the product selection screen) is displayed on the display device, leaving the selection of the correct product to the operator. However, selecting the correct product from multiple candidate products with similar appearances is a cumbersome operation, and it is a time-consuming task, especially for operators unfamiliar with the system. For these reasons, in order to improve the efficiency and speed of the product registration process, it is desirable to reduce the frequency of displaying the product selection screen.
[0079] Therefore, the product registration processing unit 102 of this embodiment aims to speed up the product registration process by reducing the number of times the product selection screen is displayed, even when multiple candidate products are recognized.
[0080] Specifically, the product registration processing unit 102 determines whether a symbol is attached to any of the candidate products based on information other than the image captured by the imaging device 20, and controls whether to continue code recognition and package recognition or to display the product selection screen according to the determination result.
[0081] In this embodiment, the product registration processing unit 102 refers to the symbol flag in the product master 142 and determines whether or not a symbol is attached to any of the candidate products. Here, the candidate products to be determined for the presence or absence of a symbol can be arbitrarily set, but the candidate products to be determined may be determined based on the similarity included in the recognition result of package recognition.
[0082] For example, the product registration processing unit 102 refers to the similarity score corresponding to each product code and identifies the product code with the highest similarity score. Next, the product registration processing unit 102 refers to the symbol flag of the identified product code from the product master 142 and determines whether or not a symbol is attached to the product corresponding to that product code.
[0083] If the product registration processing unit 102 determines that a symbol is attached, it waits until the product is uniquely identified by code recognition (or package recognition) without displaying the product selection screen. In this case, the product registration processing unit 102 may also display a message on the display unit 16 prompting the customer to confirm whether a symbol is attached to the product, or a message instructing the customer to point the symbol towards the imaging device 20. This allows the product registration process to be sped up by prompting the customer to read the symbol if the product they are holding has a symbol attached.
[0084] On the other hand, if it is determined that no symbol is attached, the product registration processing unit 102 displays a product selection screen on the display unit 16 based on the product information corresponding to each of the product codes included in the recognition result.
[0085] Figure 9 shows an example of product selection screen A displayed on the POS terminal 10. As shown in Figure 9, product selection screen A has a display area Aa for displaying products corresponding to each of the product codes included in the recognition result as candidate products.
[0086] Display area Aa displays the product name, price, product image, etc., of the candidate product based on the product information of the candidate product. Figure 9 shows four candidate products, and a display area Aa is provided for each candidate product. In product selection screen A, the placement of candidate products is not particularly restricted, but it is preferable to place them according to their similarity. For example, they may be arranged in the order of top left → top right → bottom left → bottom right, from most similar to least similar.
[0087] Furthermore, display area Aa also functions as an operator for selecting the product to be registered. When a customer selects any of the display areas Aa, the product registration processing unit 102 executes the product registration process based on the product code corresponding to the candidate product in the selected display area Aa. Then, the product registration processing unit 102 clears the product selection screen A.
[0088] Furthermore, product selection screen A has a back button Ab. When product registration processing unit 102 receives an operation on the back button Ab, it clears product selection screen A and waits until a new recognition result is obtained.
[0089] Furthermore, in product selection screen A, candidate products with a symbol flag indicating "with symbol" may be hidden. This reduces the number of candidate products displayed on product selection screen A, thereby improving the recognition speed and usability of the product registration process.
[0090] Furthermore, the product registration processing unit 102 may wait for the recognition result from code recognition even while the product selection screen A is displayed. This allows the product registration processing unit 102 to continue the product registration process by code recognition even while candidate products recognized by package recognition are displayed on the product registration processing unit 102. Therefore, if a product is uniquely identified by code recognition, the product registration process can be carried out quickly. In this case, it is preferable to display a message on the product selection screen A indicating that symbol reading is also possible.
[0091] Returning to Figure 8, the payment processing unit 103 executes the payment processing (settlement processing) for the products registered in the product registration process, in response to customer operations. The settlement processing may be performed by the POS terminal 10 alone, or it may be performed in cooperation with other devices (for example, accounting equipment or a settlement server).
[0092] For example, when the settlement processing unit 103 receives an operation to initiate settlement via the operation unit 15, it displays a screen on the display unit 16 that allows the user to select a payment method such as cash payment, electronic money payment, or credit card payment. If cash payment is selected, the settlement processing unit 103 executes a settlement process to settle the total amount of registered goods with the payment amount received via a deposit device (not shown). If electronic money payment or credit card payment is selected, the settlement processing unit 103 accepts input of settlement information for settlement using the selected payment method and, in cooperation with an external settlement server, executes a settlement process to settle the total amount of goods registered for sale based on the input settlement information. Such settlement information may be, for example, a credit card number or a two-dimensional code for code payment. In the case of a two-dimensional code for code payment, input may be received via the imaging device 20.
[0093] Next, we will explain the processing flow performed by the product sales data processing system 1. Figure 10 is a sequence diagram showing an example of the processing flow performed by the product sales data processing system 1.
[0094] First, at the POS terminal 10, the customer performs a registration operation by holding the product to be purchased over the imaging device 20. The image acquisition unit 201 of the imaging device 20 acquires the image captured by the imaging unit 25 (step S1). Next, the image transmission unit 202 of the imaging device 20 transmits the image obtained by imaging to the edge device 30 (step S2). In addition, the code recognition unit 203 of the imaging device 20 performs code recognition on the image obtained by imaging (step S3).
[0095] When an image is transmitted from the imaging device 20, the package recognition unit 302 of the edge device 30 performs package recognition on the image (step S4). If a product is uniquely recognized or multiple candidate products are recognized through package recognition, the recognition result output unit 303 of the edge device 30 transmits the recognition result, including the product code of the recognized product, to the imaging device 20 (step S5).
[0096] When the recognition result output unit 204 of the imaging device 20 receives a recognition result from the edge device 30, it outputs the recognition result to the POS terminal 10 (step S6). Also, when the code recognition unit 203 uniquely recognizes a product through code recognition, it outputs the recognition result, including the product code of the recognized product, to the POS terminal 10 (step S6).
[0097] When a product is imaged during customer registration, the edges of the product will be entered into the imaging range of the imaging device 20 sequentially. Therefore, it is highly likely that the package recognition result will be output to the POS terminal 10 first, followed by the code recognition result. However, the output order of the code recognition and package recognition results is not limited to this.
[0098] When the product registration processing unit 102 of the POS terminal 10 receives recognition results from the imaging device 20, it executes the product registration process based on those recognition results (step S7). Furthermore, the POS terminal 10, imaging device 20, and edge device 30 execute the processes described in steps S1 to S7 each time a customer performs a product registration operation. Then, when the customer instructs the POS terminal 10 to finish product registration, the payment processing unit 103 of the POS terminal 10 executes the payment process (step S8).
[0099] Next, the process performed by the POS terminal 10 will be explained with reference to Figure 11. Here, Figure 11 is a flowchart showing an example of the product registration process performed by the POS terminal 10. Note that this process corresponds to step S7 in Figure 10.
[0100] First, the recognition result acquisition unit 101 determines whether or not it has acquired a recognition result from the imaging device 20 (step S11). If it has not acquired a recognition result (step S11; No), it proceeds to step S18. If the recognition result acquisition unit 101 has acquired a recognition result from the imaging device 20 (step S11; Yes), it proceeds to step S12.
[0101] The product registration processing unit 102 determines whether or not the product can be uniquely identified based on the recognition result obtained in step S11 (step S12). Specifically, the product registration processing unit 102 determines whether or not a product code recognized by code recognition or package recognition is included in the recognition result.
[0102] If it is determined that the product can be uniquely identified (step S12; Yes), the product registration processing unit 102 executes the product registration process based on the product code included in the recognition result (step S17), and proceeds to step S18. When the product registration processing unit 102 executes the product registration process, it displays information such as the product name and price of the registered product on the display unit 16 (a screen not shown, also called the product registration screen), in the order of registration.
[0103] On the other hand, if it is determined that a product cannot be uniquely identified (Step S12; No), that is, if the recognition result includes product codes for multiple candidate products, the process proceeds to Step S13. In Step S13, the product registration processing unit 102 extracts the product code of the candidate product with the highest similarity based on the similarity included in the recognition result (Step S13). Next, the product registration processing unit 102 refers to the product master 142 and determines whether the symbol flag of the product code extracted in Step S13 indicates that the symbol "is present" (Step S14).
[0104] If the symbol flag indicates that the symbol "exists" (step S14; Yes), the product registration processing unit 102 returns to step S11, making it possible to obtain the code recognition result.
[0105] On the other hand, if the symbol flag indicates "no symbol" (step S14; No), the product registration processing unit 102 displays a product selection screen A on the display unit 16, allowing the user to select one product from each of the candidate products, based on the product information of each candidate product stored in the product master 142 (step S15). Next, the product registration processing unit 102 waits until a product is selected from the product selection screen A (step S15; No).
[0106] If the system is configured not to display candidate products with symbols on the product selection screen A, the process of obtaining the code recognition result may be executed in the background of step S16. In this case, if the code recognition result is obtained while the product selection screen A is being displayed, the product registration processing unit 102 may immediately proceed to step S17 to execute the product registration process. When transitioning from the product selection screen A to step S17, it is preferable to inform the customer of the reason for the screen transition by displaying a message such as "Symbol recognized."
[0107] In step S16, if a product is selected from product selection screen A (step S16; Yes), the product registration processing unit 102 executes the product registration process based on the product code of the selected product (step S17), and then proceeds to step S18.
[0108] Next, the product registration processing unit 102 determines whether an operation to instruct the end of the product registration process, in other words, an operation to instruct the start of the payment process, has been performed (step S18). If the operation is not performed (step S18; No), the product registration processing unit 102 returns to step S11 to prepare for obtaining new recognition results. If the operation is accepted (step S18; Yes), the product registration processing unit 102 terminates this process and moves on to the payment process (step S8 in Figure 10).
[0109] As described above, if the POS terminal 10 cannot uniquely identify a product based on the code recognition and package recognition results, it switches between continuing code recognition and package recognition or displaying product selection screen A, depending on whether the candidate product with the highest similarity has a symbol attached. This reduces the opportunities for the POS terminal 10 to display product selection screen A during the product registration process, thereby reducing the opportunities for the operator to select candidate products and improving the efficiency of the product registration process.
[0110] Furthermore, if the candidate product with the highest similarity has a symbol attached, the POS terminal 10 continues code recognition and package recognition, and hides the product selection screen A. This allows the POS terminal 10 to continue code recognition (and package recognition) when there is a high probability that the product can be identified by code recognition, thereby enabling efficient product recognition and speeding up the product registration process.
[0111] The embodiments described above can also be modified and implemented as appropriate by changing some of the configurations or functions of each of the devices described above. Therefore, several modifications of the embodiments described above will be described below as other embodiments. In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common with what has already been described. Furthermore, the modifications described below may be implemented individually or in combination as appropriate.
[0112] (Variation 1) In the above embodiment, the display of product selection screen A is controlled based on the symbol flag of the candidate product with the highest similarity among the candidate products recognized by package recognition; however, the conditions related to this control are not limited to this.
[0113] For example, if the product registration processing unit 102 finds that the symbol flag of at least one candidate product recognized by package recognition indicates that the symbol "is present", it may hide the product selection screen A and continue with code recognition and package recognition.
[0114] Furthermore, for example, if the symbol flag of all candidate products recognized by package recognition indicates "symbol present," the product selection screen A may be hidden, and code recognition and package recognition may be continued. Also, for example, if the number of candidate products recognized by package recognition that have a symbol flag of "symbol present" exceeds the number of products that do not have a symbol, or if a predetermined threshold is exceeded, the product registration processing unit 102 may hide the product selection screen A, and code recognition and package recognition may be continued.
[0115] In this way, the POS terminal 10 switches between continuing code recognition and package recognition, or displaying product selection screen A, depending on the state of the symbol flag of the candidate product. Therefore, the POS terminal 10 can flexibly switch the product registration method depending on the state of the symbol of the candidate product recognized by package recognition, thereby reducing the opportunities to display product selection screen A.
[0116] (Modification 2) In the above-described embodiment, when a candidate product is recognized by package recognition, a configuration was described in which the symbol flag of the product master 142 is referenced to determine whether or not a symbol is attached. In this modification, a configuration is described in which, as an example of information other than the image captured by the imaging device 20, a configuration is described in which a symbol is attached based on information input by the operator.
[0117] Specifically, in this modified example, when multiple candidate products are recognized by package recognition, the product registration processing unit 102 displays a message or confirmation screen on the display unit 16 to confirm with the customer whether or not a symbol is attached to the product. For example, the product registration processing unit 102 displays the confirmation screen shown in Figure 12 on the display unit 16.
[0118] Figure 12 shows an example of a confirmation screen displayed on the POS terminal 10. As shown in Figure 12, the confirmation screen B has a message Ba for confirming whether or not a symbol is attached to the product, and a YES button Bb and a No button Bc for entering the confirmation result.
[0119] The operator of the POS terminal 10 checks whether a symbol is attached to the product held up to the imaging device 20 based on confirmation screen B, and inputs the confirmation result via the YES button Bb or the No button Bc. When the product registration processing unit 102 receives the operation of the YES button Bb or the No button Bc, it obtains the operation details as the confirmation result.
[0120] If the confirmation result indicates that the symbol is present, the product registration processing unit 102 hides the product selection screen A and waits until the product is uniquely identified by code recognition (or package recognition). On the other hand, if the confirmation result indicates that the symbol is absent, the product registration processing unit 102 displays the product selection screen A on the display unit 16 based on the product information of the candidate product. In this case, the product registration processing unit 102 may be configured not to display candidate products with the symbol present on the product selection screen A.
[0121] As a result, even when a candidate product is recognized, the POS terminal 10 can reduce the number of times the product selection screen A is displayed, similar to the embodiment described above. In this modified example, when a candidate product is recognized, a confirmation screen B is displayed, but since the operation on the confirmation screen B is a binary selection operation of YES or NO, the operation is simpler compared to the selection operation on the product selection screen A. Furthermore, even when the product selection screen A is displayed, the candidate products are narrowed down to those with the symbol "None", so the operation of selecting a product can be easily performed. Therefore, the POS terminal 10 of this modified example can improve the efficiency of the product registration process.
[0122] If the confirmation result indicates that the symbol "is present", the product registration processing unit 102 may refer to the symbol flag of each candidate product and determine whether there is only one candidate product with the symbol "is present". In this case, if the product registration processing unit 102 determines that there is only one candidate product with the symbol "is present", it may immediately execute the product registration process based on the product code of that candidate product without waiting for the recognition result of code recognition (or package recognition).
[0123] Furthermore, even if the confirmation result indicates "No symbol", the product registration processing unit 102 may refer to the symbol flag of each candidate product and determine whether there is only one candidate product with the symbol "No". In this case, if the product registration processing unit 102 determines that there is only one candidate product with the symbol "No", it may immediately execute the product registration process based on the product code of that candidate product without displaying the product selection screen A.
[0124] (Variation 3) In the embodiment described above, the imaging device 20 performs code recognition, but the configuration is not limited to this, and for example, the POS terminal 10 or the edge device 30 may perform the recognition.
[0125] For example, when performing code recognition at the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) the image obtained through imaging to the POS terminal 10. The POS terminal 10 also has a code recognition unit 203, which performs code recognition on the image transmitted from the imaging device 20.
[0126] Furthermore, for example, when performing code recognition on the edge device 30, the edge device 30 is equipped with a code recognition unit 203, which performs code recognition on the image transmitted from the imaging device 20. In this case, the recognition result output unit 303 of the edge device 30 outputs the recognition results of code recognition and package recognition in the order in which they were recognized.
[0127] Furthermore, in the above-described embodiment, the edge device 30 performs package recognition, but this is not limited to this configuration; for example, the POS terminal 10 or the imaging device 20 may perform the recognition.
[0128] For example, when package recognition is performed at the POS terminal 10, the image transmission unit 202 of the imaging device 20 transmits (outputs) the image obtained by imaging to the POS terminal 10. The POS terminal 10 also has a package recognition unit 302, which performs package recognition on the image transmitted from the imaging device 20.
[0129] Furthermore, for example, when performing package recognition with the imaging device 20, the imaging device 20 is equipped with a package recognition unit 302, which performs package recognition on the image obtained by imaging. In this case, the recognition result output unit 204 of the imaging device 20 outputs the recognition results of code recognition and package recognition in the order in which they were recognized.
[0130] Furthermore, although the edge device 30 was described as a single device in the above embodiment, it is not limited to this, and for example, the edge device 30 may be implemented as a cloud (cloud computing) composed of multiple information processing devices. Also, the store server 40 may have the functions of the edge device 30.
[0131] (Modification 4) In the above-described embodiment, the POS terminal 10 was described as a device operated by the customer themselves. However, the POS terminal 10 is not limited to this and may be operated by a store employee. In this case, the imaging device 20 takes images of the products based on the registration operation performed by the store employee.
[0132] (Variation 5) In the above-described embodiment, the POS terminal 10 was configured to have one imaging device 20 (or imaging unit 25). However, it is not limited to this configuration, and may also have multiple imaging devices 20 (or imaging units 25). In this case, code recognition may be performed by each of the imaging devices 20, or by a single imaging device 20. It is preferable that the multiple imaging devices 20 (or imaging units 25) are configured to capture images of the product from different imaging positions. This allows for imaging of the product from multiple angles, thereby improving recognition speed and recognition rate.
[0133] The programs executed in each of the above-described embodiments (and modifications) are provided pre-installed in ROM, storage units, etc. The programs executed in each of the above-described embodiments may also be provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).
[0134] Furthermore, the programs executed by each of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs executed by each of the above-described embodiments may be provided or distributed via a network such as the Internet.
[0135] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and their variations can be implemented in a variety of other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]
[0136] 1 System 10 POS terminals 20 Imaging device 30 Edge Devices 40 Store Servers 101 Recognition result acquisition unit 102 Product Registration Processing Unit 103 Payment Processing Unit 201 Image Acquisition Unit 202 Image Transmission Unit 203 Code Recognition Unit 204 Recognition Result Output Unit 301 Image acquisition unit 302 Package Recognition Unit 303 Recognition Result Output Unit [Prior art documents] [Patent Documents]
[0137] [Patent Document 1] Japanese Patent Publication No. 2013-89258
Claims
1. A first acquisition means for acquiring the recognition result of a first recognition process, which reads a symbol containing identification information that can identify the target product from an image of the target product captured by the imaging device, A second acquisition means for acquiring the recognition result of a second recognition process that estimates products similar to the aforementioned characteristics from a predetermined number of products based on the external characteristics of the target product shown in the aforementioned image, If the target product can be identified by the recognition result obtained earlier from the recognition result of the first recognition process and the recognition result of the second recognition process, a registration means for registering the target product as a purchase target, Equipped with, The registration means, if the recognition result of the second recognition process includes multiple candidate products that are candidates for the target product, and if it is determined that the symbol is attached to any of the candidate products based on information other than the image, the registration process for the target product is suspended until a recognition result that uniquely identifies the target product is obtained. If it is determined that the symbol is not attached to all of the candidate products or to any of the candidate products that meet predetermined conditions, the registration means displays a selection screen in which one product can be selected from the candidate products, and registers the product selected from the selection screen as the target for purchase.
2. The product registration device according to claim 1, wherein the registration means determines whether the symbol is attached to each of the plurality of products based on setting information in which a setting value indicating whether or not the symbol is attached to each of the plurality of products is set.
3. If the recognition result of the second recognition process includes multiple candidate products, the system further includes a display means for displaying a confirmation screen to allow the operator to confirm whether or not the target product has the symbol attached. The product registration device according to claim 1, wherein the registration means determines whether or not the candidate product has the symbol attached to it, based on the information entered by the operator on the confirmation screen.
4. The recognition result of the second recognition process includes a similarity score indicating the degree of similarity between the candidate product and the target product. The product registration device according to claim 1, wherein the registration means waits to acquire a new recognition result if the candidate product with the highest similarity among the candidate products included in the recognition result of the second recognition process has the symbol attached to it.
5. The computer of the product registration device A first acquisition means for acquiring the recognition result of a first recognition process, which reads a symbol containing identification information that can identify the target product from an image of the target product captured by the imaging device, A second acquisition means for acquiring the recognition result of a second recognition process that estimates products similar to the aforementioned characteristics from a predetermined number of products based on the external characteristics of the target product shown in the aforementioned image, If the target product can be identified by the recognition result obtained earlier from the recognition result of the first recognition process and the recognition result of the second recognition process, a registration means for registering the target product as a purchase target, and make it work The registration means, if the recognition result of the second recognition process includes multiple candidate products that are candidates for the target product, and if it is determined that the symbol is attached to any of the candidate products based on information other than the image, the registration process for the target product is suspended until a recognition result that uniquely identifies the target product is obtained. If it is determined that the symbol is not attached to all of the candidate products or to any of the candidate products that meet the predetermined conditions, the program displays a selection screen in which one product can be selected from the candidate products, and registers the product selected from the selection screen as the target for purchase.