Commodity information registration system, information processing apparatus, commodity image recognition apparatus, and commodity image recognition program

Abstract

To prevent the same commodity from being registered redundantly even when code recognition and image recognition are used simultaneously.SOLUTION: A commodity information registration system includes a decoding unit, a recognition unit, and a registration unit. The decoding unit decodes a code symbol from an image of the code symbol included in an image of a commodity to be registered, to obtain a code value represented by the code symbol. The recognition unit recognizes the appearances of the commodity from an appearance image of the commodity included in the image of the commodity to be registered, to obtain a code value. The registration unit registers information on a commodity identified by a commodity code corresponding to the code value, based on the code value acquired by at least one of the decoding unit and the recognition unit.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] An embodiment of the present invention relates to a product information registration system, an information processing device, a product image recognition device, and a product image recognition program. [Background technology]

[0002] POS (Point Of Sales) systems are used in stores that sell products and services (hereinafter collectively referred to as "products"). This type of POS system is capable of registering information about products that customers are about to purchase by reading code symbols (hereinafter referred to as "product barcodes") attached to the products with a scanner. Product barcodes include one-dimensional codes and two-dimensional codes. In one example, the product barcode is printed on a sticker and affixed to the product. In another example, the product barcode is printed on the packaging that contains the product.

[0003] The POS system operator must search for the product barcode on the product and hold it up to the scanner, making the operation of reading the product barcode cumbersome. As a result, the operator must register product information (hereinafter also referred to as "product registration") in a time-consuming manner. In particular, in so-called self-service POS systems, customers who are not familiar with product registration must scan product barcodes, so there is a demand for improving the operability of reading product codes, in other words, the operability of the operator to identify products.

[0004] In view of this, a method has been considered in which images captured by a camera are used to recognize images of products and register the products.

[0005] When implementing both the code recognition function for reading product barcodes and this type of image recognition function in a single device (system), if the output codes for each function are different, the same product will be registered twice. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2014-021915 A Summary of the Invention [Problem to be solved by the invention]

[0007] The problem that the present invention aims to solve is to provide a product information registration system, an information processing device, a product image recognition device, and a product image recognition program that are capable of preventing duplicate registration of the same product even when code recognition and image recognition are used in combination. [Means for solving the problem]

[0008] According to an embodiment, the product information registration system includes a decoding unit, a recognition unit, and a registration unit. The decoding unit obtains a code value represented by a code symbol from an image of the code symbol included in an image of a product to be registered by decoding the code symbol. The recognition unit obtains the code value by recognizing the appearance of the product from an appearance image of the product included in the image of the product to be registered. The registration unit registers information related to a product identified by a product code corresponding to the code value based on the code value obtained by at least one of the decoding unit and the recognition unit. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a product information management system including a product information registration system according to the first embodiment. [Diagram 2] FIG. 2 is a perspective view showing the appearance of a self-service POS system as an example of the product information registration system according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram showing a hardware configuration of a POS terminal in the product information registration system according to the first embodiment. [Figure 4]FIG. 4 is a block diagram showing the hardware configuration of the scanner in the product information registration system according to the first embodiment. [Diagram 5] FIG. 5 is a block diagram showing a hardware configuration of an image recognition device as an example of the commodity image recognition device according to the first embodiment in the commodity information registration system according to the first embodiment. [Figure 6] FIG. 6 is a block diagram showing a hardware configuration of a server included in the product information management system. [Figure 7] FIG. 7 is a diagram showing a data configuration of the product master stored in the storage unit of the POS terminal of FIG. [Figure 8] FIG. 8 is a diagram showing the data configuration of the mark type setting table and the transmission mode setting table stored in the storage unit of the scanner of FIG. [Figure 9] FIG. 9 is a diagram showing a data structure of the comparison dictionary stored in the storage unit of the image recognition device of FIG. [Figure 10] FIG. 10 is a diagram showing the data configuration of the recognition flag setting table and the mark type setting table stored in the storage unit of the image recognition device of FIG. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of the control unit of the POS terminal of FIG. [Figure 12] FIG. 12 is a block diagram showing the functional configuration of the control unit of the scanner of FIG. [Figure 13] FIG. 13 is a block diagram showing a functional configuration of the control unit of the image recognition device of FIG. [Figure 14] FIG. 14 is a block diagram showing a functional configuration of the control unit of the server in FIG. [Figure 15] FIG. 15 is a sequence diagram showing an outline of the operation of the product information management system. [Figure 16] FIG. 16 is a flowchart showing the flow of operations performed when registering one product using the scanner and image recognition device of the embodiment. [Figure 17] FIG. 17 is a flowchart showing details of the output code generation process in FIG. [Figure 18]FIG. 18 is a block diagram showing a hardware configuration of a product information registration system according to the second embodiment. [Figure 19] FIG. 19 is a block diagram showing a functional configuration of the control unit of the POS terminal of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0011] [First embodiment] (Product Information Management System) First, a product information management system MS according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the product information management system MS according to the first embodiment. The product information management system MS has a product information registration system RS according to the first embodiment and a server 40.

[0012] The product information registration system RS includes a POS terminal 10, a scanner 20, and an image recognition device 30 as an example of a product image recognition device according to the first embodiment. The POS terminal 10 and the scanner 20 are communicatively connected via a LAN (Local Area Network) or a USB (Universal Serial Bus) cable. The scanner 20 and the image recognition device 30 are communicatively connected via a LAN. In the embodiment, a combination of Ethernet (registered trademark) and TCP (Transmission Control Protocol) is used for LAN communication, but the communication protocol is not limited. In addition, the LAN is not limited to a wired LAN, and may be a wireless LAN, or may be a mixture of wired and wireless.

[0013] The POS terminal 10 and the image recognition device 30 are communicatively connected to a LAN hub LH via a LAN. The LAN hub LH is communicatively connected to a router RT via a LAN. The router RT is communicatively connected to a server 40 via a network NW. The router RT may be wired and connected to the network NW for wired communication, or wirelessly and connected to the network NW for wireless communication.

[0014] In addition, although only one product information registration system RS is shown for one LAN hub LH in FIG. 1, there may be multiple product information registration systems RS. Although only one LAN hub LH is shown for one router RT, there may be multiple LAN hubs LH. Furthermore, there may be multiple routers RT. For example, one or more LAN hubs LH under one router RT and one or more product information registration systems RS under each LAN hub LH may be disposed in one store. For example, the server 40 may be a management server disposed in the headquarters of a company that manages and supervises the operation of multiple stores.

[0015] The scanner 20 captures an image of a product and obtains an image showing the appearance of the product (hereinafter, referred to as a product image). The scanner 20 transmits the obtained product image to the image recognition device 30 via a LAN. The scanner 20 also detects and decodes a product barcode (one-dimensional code and two-dimensional code), which is a code symbol included in the obtained product image. That is, the scanner 20 reads information held by the product barcode. The decoded result of the product barcode includes a code value, which is a barcoded numerical value, and a code type mark indicating the code type of the barcode. The code type is a type of barcode, such as an EAN code or a JAN code. The code value is a numerical value for uniquely identifying a product to which the product barcode is attached. For example, in the case of a JAN code, the code value is a 13-digit numerical value including a business code (9 or 7 digits) indicating the manufacturer of the product, a product item code (3 or 5 digits) identifying the product, and a check digit (1 digit). The store handles products that do not have product barcodes from the manufacturer, such as prepared foods and fresh produce, and for such products, an in-store code exclusive to the store is assigned to the product item code, and a product barcode unique to the store is created along with the store's business code and attached to the products. Some stores also introduce private brand products, and similarly use product barcodes unique to the store for such products. The scanner 20 transmits the decoded results to the image recognition device 30 via a LAN.

[0016] The image recognition device 30 receives the product image from the scanner 20 via a LAN. The image recognition device 30 has an image recognition function using a trained model by machine learning, for example, and performs image recognition on the product image by the image recognition function. In detail, the image recognition device 30 acquires a feature amount included in the product image by the image recognition function, compares the acquired feature amount with a matching dictionary in which a plurality of feature amounts are associated with code values ​​in a product barcode attached to the appearance of the product included in the product image, and acquires a code value associated with the feature amount closest to the acquired feature amount. The matching dictionary includes a code value in the product barcode of each product as correct answer data for image recognition based on the product feature amount. In other words, the matching dictionary defines the same information as the product barcode of the product as correct answer data for image recognition. Therefore, the image recognition device 30 acquires a code value corresponding to the appearance of the product, such as a package, by using such a matching dictionary. The image recognition device 30 determines the code type of the acquired code value and converts it into a code type mark indicating the code type. Then, the image recognition device 30 adds flag information indicating that the code type mark and the code value are image recognition results to make them the final image recognition results. The image recognition device 30 may perform image recognition on the product images using a method such as deep learning. The image recognition device 30 transmits the image recognition results to the scanner 20 via a LAN.

[0017] The scanner 20 receives the image recognition result from the image recognition device 30 via the LAN. The scanner 20 creates an output code including the decoded result and the image recognition result. At this time, the scanner 20 creates an output code in a format according to a transmission mode that has been set in advance by an instruction from the POS terminal 10. The transmission mode includes, for example, a distinction mode that includes the decoded result and the image recognition result as they are, thereby making it possible to distinguish between the two results, and a common mode that does not distinguish between the two results by deleting flag information in the image recognition result. The scanner 20 transmits the created output code to the POS terminal 10 via the LAN or a USB cable.

[0018] The POS terminal 10 has a product master in which an output code, a product name, and a price are registered in association with a product code. The product code is an example of product identification information that uniquely identifies each product. The output code in the product master includes information corresponding to a decoded result and information corresponding to a recognition result. Therefore, the POS terminal 10 identifies a record of the corresponding product code in the product master based on the output code received from the scanner 20, and registers the purchased product based on the product name and price registered in that record.

[0019] Furthermore, the image recognition device 30 determines whether the product image received from the scanner 20 is to be used for additional learning. For example, if the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is equal to or greater than a predetermined amount, the image recognition device 30 determines that the product image received from the scanner 20 is a product image to be used for additional learning. The image recognition device 30 transmits the product image determined to be used for additional learning, and the code value and code type mark, which are the decoded results received from the scanner 20, to the server 40 via the LAN.

[0020] The server 40 receives the product images and the decoded results from the image recognition device 30 via the LAN. The server 40 stores the received product images and the decoded results in association with each other. The server 40 performs additional learning of a trained model for product image recognition using the stored product images and decoded results. The server 40 updates the trained model through the additional learning. The server 40 also updates the matching dictionary in conjunction with the additional learning. Updating the matching dictionary includes correcting existing records and adding new records. The server 40 transmits the updated trained model and the updated matching dictionary to the image recognition device 30 via the LAN.

[0021] The image recognition device 30 receives the updated trained model and the updated matching dictionary from the server 40 via a LAN. The image recognition device 30 updates the currently held trained model with the received updated trained model. The image recognition device 30 also updates the currently held matching dictionary with the received updated matching dictionary.

[0022] In this product information management system MS, if the product packaging is changed, it is possible to eliminate the need to collect new product images for additional training of the trained model for product image recognition.

[0023] The server 40 also manages a transaction database and a product master. The POS terminal 10 transmits transaction processing information (transaction information related to a transaction that has been processed) via the LAN through the LAN hub LH to the router RT. The router RT transmits the received transaction processing information to the server 40 through the network NW. The server 40 updates the transaction database based on the received transaction processing information. For example, the server 40 receives transaction processing information from multiple POS terminals 10, and records transaction information related to transactions, such as what time, which product, how much yen, and how many units were sold for each of the multiple POS terminals 10, in the transaction database. The server 40 also manages the product master held by each POS terminal 10. The management of the product master includes modifying existing records and adding new records. The server 40 also transmits product master suitable for each POS terminal 10 to the POS terminal 10 through the network NW, the router RT, and the LAN hub LH. The management of such a transaction database and product master may be performed by a store server (not shown) arranged in each store, rather than by the server 40.

[0024] The image recognition device 30 may transmit the product images received from the scanner 20 to the server 40 regardless of whether they are used for additional learning. In this case, the product images stored in the server 40 may be used, for example, to analyze the state (event) of image recognition in the image recognition device 30.

[0025] (Product Information Registration System) Hereinafter, as an example of a product information registration system, a self-service POS system operated by a customer who purchases a product will be described. However, the product information registration system is not limited to a self-service POS system. The product information registration system may be a POS system operated by a salesperson at a store that sells the product.

[0026] 2 is a perspective view showing the appearance of the self-service POS system 1. The self-service POS system 1 has a POS terminal 10, a scanner 20 built into the POS terminal 10, and an image recognition device 30 built into the POS terminal 10. Here, the self-service POS system 1 is an example of a product information registration system, the POS terminal 10 is an example of a registration device, and the scanner 20 and the image recognition device 30 are examples of an information processing device according to the first embodiment.

[0027] The scanner 20 decodes the barcode attached to the product purchased by the customer, obtains the coded code value, determines the barcode type from this code value, and obtains a decoded result including the code value and a code type mark indicating the barcode type. The scanner can transmit this decoded result as an output code to the POS terminal 10. The POS terminal 10 identifies the product code of the purchased product from the product master based on the output code input from the scanner 20, executes product registration, and executes accounting for the registered product. The accounting is a process executed by the POS terminal 10 to allow the customer to pay for the product.

[0028] 2, the POS terminal 10 includes a main body 50, a first basket mounting stand 70, and a second basket mounting stand 80. The main body 50 includes a display 51 and a touch panel 52 on an upper portion, and also houses a printer 53, a card reader 54, and a change dispenser 55 (see FIG. 3 for each).

[0029] Display 51 is a display device such as an LCD (Liquid Crystal Display) and displays various information. For example, display 51 displays product information such as the product name and price of a product registered by a customer's registration operation. Display 51 also displays accounting information such as the total amount of the transaction and the amount of change in accordance with the customer's accounting operation.

[0030] The touch panel 52 is provided on the surface of the display 51, and inputs information according to the touched position into the control unit 100 (see FIG. 3) of the POS terminal 10. A customer using the POS terminal 10 can operate the touch panel 52 by touching various buttons and operation keys displayed on the display 51.

[0031] As shown in FIG. 2, below the display 51, a reading window 230 for the scanner 20, an issuing port 531 for receipts printed by the printer 53, and a card insertion port 541 for the card reader 54 are formed facing the direction in which the customer is positioned.

[0032] The reading window 230 is a window through which the scanner 20 captures an image of the exterior of a commodity. A customer who is an operator operating the self-service POS system 1 performs an operation related to commodity registration by holding the commodity to be purchased over the reading window 230. The customer holds the commodity over the reading window 230 while changing the orientation of the commodity with respect to the reading window 230 until the self-service POS system 1 executes the commodity registration and generates a registration completion sound.

[0033] The issuing port 531 is an opening through which a receipt printed by the printer 53 is ejected. Information (transaction processing information) related to a transaction processed by the POS terminal 10 is printed on the receipt. The POS terminal 10 may be configured to issue an electronic receipt instead of a paper receipt printed by the printer 53.

[0034] Card insertion slot 541 is an opening for inserting a credit card or the like to be read by card reader 54. A card is inserted through card insertion slot 541 when a customer makes a payment with a credit card or when a customer has a store membership card or the like read.

[0035] The main body 50 is also formed with a coin insertion slot 551, a coin ejection slot 552, a coin receiving tray 553, a bill insertion slot 554, and a bill ejection slot 555 in correspondence with the position of the change machine 55 housed therein.

[0036] The coin slot 551 is an opening through which a customer inserts coins to be paid. Coins inserted into the coin slot 551 are stored in the change machine 55. The coin outlet 552 is an opening through which coins to be dispensed from the change machine 55 are discharged. The coin outlet 552 discharges change or coins inserted into the coin slot 551 but not accepted by the change machine 55. The coin tray 553 is a tray that receives coins discharged from the coin outlet 552. The banknote slot 554 is an opening through which a customer inserts banknotes to be paid. Banknotes inserted into the banknote slot 554 are stored in the change machine 55. The banknote outlet 555 is an opening through which banknotes that are change to be dispensed from the change machine 55 are discharged.

[0037] The first basket placement table 70 is a table on which a shopping basket (not shown) containing products that a customer wishes to purchase is placed. The first basket placement table 70 includes a first weighing unit 71. The first weighing unit 71 weighs the total weight of a shopping basket placed on the first basket placement table 70 and the products contained in the shopping basket.

[0038] The second basket placement table 80 is a table for placing shopping baskets (not shown) that contain products that have been removed from the shopping basket placed on the first basket placement table 70 and whose product barcodes have been read, i.e., registered products. The second basket placement table 80 includes a second weighing unit 81, a pair of support rods 82, and a temporary placement table 83.

[0039] The second weighing unit 81 weighs the total weight of a shopping basket placed on the second basket placement table 80 and the products contained in the shopping basket, or the total weight of a shopping bag (not shown) supported by a pair of support rods 82 and the products contained in the shopping bag. The support rods 82 are rods for hanging and supporting a shopping bag containing registered products. The support rods 82 support the shopping bag in an open state, making it easier to put products into the shopping bag. The temporary placement table 83 is a table on which registered products that are easily broken or deformed, such as tofu and eggs, are temporarily placed. The customer can finally place the products placed on the temporary placement table 83 into a shopping basket or shopping bag.

[0040] The customer registers the product by holding the product taken out of the shopping basket placed on the first basket placement stand 70 over the reading window 230. The customer then places the registered product in a shopping basket placed on the second basket placement stand 80 or in a shopping bag supported by the support rod 82.

[0041] The control unit 100 (see FIG. 3) recognizes the weight change when the first weighing unit 71 weighs the weight before and after an item is removed from the shopping basket placed on the first basket placement table 70. The control unit 100 recognizes the weight change when the second weighing unit 81 weighs the weight before and after an item is placed in the shopping basket placed on the second basket placement table 80 or a shopping bag supported by the support rod 82. Depending on whether these two weight changes are equal, the control unit 100 can determine whether all of the items the customer wishes to purchase have been registered.

[0042] The main body 50 also includes a cylindrical pole 57 extending upward. The pole 57 has a light emitting section 571 at its upper end that emits light when a store clerk is called.

[0043] Next, the hardware configurations of the POS terminal 10, the scanner 20, the image recognition device 30 and the server 40 will be described with reference to Fig. 3 to Fig. 6. Fig. 3 is a block diagram showing the hardware configuration of the POS terminal 10, Fig. 4 is a block diagram showing the hardware configuration of the scanner 20, Fig. 5 is a block diagram showing the hardware configuration of the image recognition device 30, and Fig. 6 is a block diagram showing the hardware configuration of the server 40.

[0044] (POS terminal hardware configuration) 3, the POS terminal 10 includes a control unit 100, a memory unit 110, a display 51, a touch panel 52, a printer 53, a card reader 54, a change machine 55, a first weighing unit 71, a second weighing unit 81, a light-emitting unit 571, an audio output unit 120, a connection interface 121, and a LAN interface 122. The control unit 100, the memory unit 110, the display 51, the touch panel 52, the printer 53, the card reader 54, the change machine 55, the first weighing unit 71, the second weighing unit 81, the light-emitting unit 571, the audio output unit 120, the connection interface 121, and the LAN interface 122 are connected to each other via a bus 123 or the like.

[0045] The control unit 100 is configured as a computer including a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 123.

[0046] The CPU 101 controls the entire POS terminal 10. The CPU 101 is an example of a processor, and instead of the CPU 101, an MPU (Micro Processing Unit) may be used. If the processor is a multi-core / multi-thread processor, it can execute multiple processes in parallel. The processor may be realized in various other forms including integrated circuits such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), a SoC (System on a Chip), and a PLD (Programmable Logic Device), or may be a combination of two or more of these.

[0047] The ROM 102 stores various programs and various data, such as a program used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101, and loads various programs and data stored in the ROM 102 and the storage unit 110. The control unit 100 executes various control processes of the POS terminal 10, for example, by the CPU 101 operating in accordance with a control program 111 stored in the storage unit 110 and loaded in the RAM 103.

[0048] RAM 103 has a product information section 1031. Product information section 1031 stores product information read from product master 112 (described later) based on an output code obtained from scanner 20. A product is registered by storing (registering) product information in product information section 1031. Product information stored in product information section 1031 includes a product code, product name, price, etc. Product information section 1031 stores product information related to one transaction, and is cleared when the accounting process for that transaction is completed.

[0049] The storage unit 110 is configured with a storage medium such as a hard disk drive (HDD) or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 110 stores a control program 111 and a product master 112.

[0050] The control program 111 is a program for realizing a function of receiving an output code from the scanner 20, a function of executing product registration based on the received output code, a function of executing accounting processing for the registered product, a function of changing the setting values ​​of the scanner 20 and the image recognition device 30, etc. The control program 111 may be stored in the ROM 102 instead of being stored in the storage unit 110.

[0051] The product master 112 is a master file that stores product information about products handled in the store. Since the products handled in the store change daily, the product master 112 is updated as appropriate by the server 40 or a store server (not shown) connected via the LAN interface 122. Fig. 7 is a diagram showing the data configuration of the product master 112. The product master 112 includes product codes, output codes associated with the product codes, product categories, product names, and prices.

[0052] The product code is information that identifies a product. The product code can be set independently by each store, for example. The product classification is information that classifies the products sold in the store. The product name is information that indicates the name of the product. The price is information that indicates the price (unit price) of the product. The product master 112 may also include other information in association with the product code.

[0053] Returning to Fig. 3, the description will continue of the hardware configuration of the POS terminal 10. The configurations and functions of the display 51 and the touch panel 52 are as described above.

[0054] The printer 53 is, for example, a thermal printer. The printer 53 prints information about the transaction on a roll of receipt paper and issues the paper as a receipt from an issuing port 531.

[0055] Card reader 54 reads credit information required for payment from a credit card inserted through card insertion slot 541. Card reader 54 also reads membership information identifying a member from a medium such as a membership card or a point card inserted through card insertion slot 541. Card reader 54 may magnetically read information from a card, or may be an IC card reader that reads information from an IC card with a built-in IC (Integrated Circuit) chip.

[0056] The change machine 55 accepts bills and coins paid by a customer when the customer pays for the purchase price of a product in cash, and dispenses change as necessary. Bills are inserted through a bill insertion slot 554, and coins are inserted through a coin insertion slot 551. Bills as change are discharged from a bill discharge slot 555, and coins as change are discharged from a coin discharge slot 552.

[0057] The configurations and functions of first weighing unit 71, second weighing unit 81, and light-emitting unit 571 are as described above. Audio output unit 120 generates sound as necessary to notify the customer of information. For example, audio output unit 120 outputs an alarm sound that notifies the customer that product registration has been completed. The audio output unit is an example of an alarm unit.

[0058] The connection interface 121 is an interface for communicating with the scanner 20 via a LAN or a USB cable. The control unit 100 transmits and receives information (data) to and from the scanner 20 via the connection interface 121, via the LAN or the USB cable. For example, the control unit 100 receives an output code from the scanner 20 via the connection interface 121, via the LAN or the USB cable.

[0059] The LAN interface 122 is an interface for communicating with the LAN hub LH via the LAN. The control unit 100 transmits and receives information (data) to and from the LAN hub LH via the LAN interface 122. For example, the control unit 100 transmits and receives the product master 112 to and from the server 40 via the LAN, via the LAN hub LH and the router RT.

[0060] The LAN interface 122 is also an interface for communicating with a higher-level device such as a store server (not shown) managed by the store. The control unit 100 transmits and receives information to and from the higher-level device via the LAN interface 122 and the LAN.

[0061] (Scanner hardware configuration) The scanner 20 captures an image of a product purchased by a customer to obtain the product image. The scanner 20 then detects a product barcode included in the product image, i.e., a barcode attached to the captured product, and decodes the detected product barcode. The decoded result of the product barcode includes a code value held by the product barcode and a code type mark indicating a code type determined from the code value. The scanner 20 also transmits the product image to the image recognition device 30 via a LAN. The scanner 20 can also transmit the decoded result to the image recognition device 30 via a LAN.

[0062] 4, the scanner 20 includes a control unit 200, a storage unit 210, an imaging unit 220, a connection interface 221, and a LAN interface 222. The control unit 200, the storage unit 210, the imaging unit 220, the connection interface 221, and the LAN interface 222 are connected to each other via a bus 223 or the like.

[0063] The control unit 200 is configured as a computer including a CPU 201, a ROM 202, and a RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 223.

[0064] The CPU 201 controls the entire scanner 20. It may be a processor other than the CPU 201. The ROM 202 stores various programs, such as a program used to drive the CPU 201, and various data. The RAM 203 is used as a work area for the CPU 201, and loads various programs and data stored in the ROM 202 and the storage unit 210. The control unit 200 executes various control processes for the scanner 20, for example, by the CPU 201 operating in accordance with a control program 211 stored in the storage unit 210 and loaded in the RAM 203.

[0065] The storage unit 210 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 210 stores a control program 211, a setting table 212, and a processing result 213.

[0066] The control program 211 is a program for realizing functions such as a function of capturing an image of a commodity by the imaging unit 220 and storing it in the RAM 203, a function of acquiring a frame image (commodity image) held in the RAM 203, a function of detecting a commodity barcode included in the commodity image, a function of decoding the commodity barcode, a function of transmitting the commodity image to the image recognition device 30, a function of receiving an image recognition result from the image recognition device 30, a function of creating an output code from the decoded result and the image recognition result, and a function of transmitting the output code to the POS terminal 10. The control program 211 may be stored in the ROM 202 instead of being stored in the storage unit 210.

[0067] The setting table 212 includes data tables for managing various settings for the scanner 20, which are set from the POS terminal 10 via the LAN. For example, the setting table 212 includes a mark type setting table 2121 and a transmission mode setting table 2122 as shown in FIG. 8. The mark type setting table 2121 is a data table for managing code type marks for each code type assumed as a product barcode. For example, if the code type is "JAN-13", "F" is set as the code type mark. The transmission mode setting table 2122 is a data table for managing a setting for whether or not to distinguish between a decoded result and an image recognition result in an output code transmitted from the scanner 20 to the POS terminal 10.

[0068] The processing result 213 includes the decoded result of the product barcode, the image recognition result by the image recognition device 30, and an output code created from the decoded result and the image recognition result.

[0069] The imaging unit 220 is composed of, for example, a color CCD image sensor or a color CMOS image sensor. The imaging unit 220 images a product through the reading window 230 under the control of the control unit 200, and outputs the acquired product image. For example, the imaging unit 220 captures a moving image at a predetermined frame rate (e.g., 30 fps). The imaging unit 220 stores the captured frame images in sequence in the RAM 203. The frame images (product images) stored in the RAM 203 are used to identify the code value of the product depicted in the frame image.

[0070] The connection interface 221 is an interface for communicating with the POS terminal 10 via a LAN or a USB cable. The control unit 200 transmits and receives information (data) to and from the POS terminal 10 via the connection interface 221, the LAN, or the USB cable.

[0071] The LAN interface 222 is an interface for communicating with the image recognition device 30 via a LAN. The control unit 200 transmits and receives information (data) to and from the image recognition device 30 via the LAN interface 222 and the LAN.

[0072] The control unit 200 detects a product barcode included in a frame image (product image) held by the RAM 203, decodes the product barcode, and determines the code type based on the number of digits of the code value indicated by the product barcode. The control unit 200 acquires the code value and the code type mark representing the code type as a decoded result by specifying a code type mark corresponding to the determined code type using the mark type setting table 2121. The control unit 200 also transmits the frame image (product image) held by the RAM 203 and the acquired decoded result to the image recognition device 30 via the LAN interface 222, and receives the image recognition result by the image recognition device 30 from the image recognition device 30. Furthermore, the control unit 200 creates an output code from the decoded result and the image recognition result. Specifically, the control unit 200 checks whether or not to distinguish between the decoded result and the image recognition result using the transmission mode setting table 2122, and creates an output code according to the check result. The method of creating this output code will be described in more detail in the operation description to be described later. The control unit 200 transmits the generated output code to the POS terminal 10 via the connection interface 221 .

[0073] (Hardware configuration of image recognition device) The image recognition device 30 receives the product image from the scanner 20 via the LAN. The image recognition device 30 performs image recognition on the received product image. That is, the image recognition device 30 acquires a code type and a code value, which are contents corresponding to the code value held by the product barcode of the product shown in the product image, based on the feature amount of the product image. The image recognition device 30 acquires a code value to which a code type mark indicating the code type and flag information indicating that it is an image recognition result are added as an image recognition result. The image recognition device 30 transmits the image recognition result to the scanner 20 via the LAN. In addition, the image recognition device 30 receives a decoded result from the scanner 20 via the LAN, and transmits the product image and the decoded result to the server 40 via the LAN as necessary. Furthermore, the image recognition device 30 receives an updated trained model and a matching dictionary from the server 40 via the LAN, and updates the currently held trained model and matching dictionary.

[0074] 5, the image recognition device 30 includes a control unit 300, a storage unit 310, and a LAN interface 321. The control unit 300, the storage unit 310, and the LAN interface 321 are connected to each other via a bus 322 or the like.

[0075] The control unit 300 is configured as a computer including a CPU 301, a ROM 302, and a RAM 303. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 322.

[0076] The CPU 301 controls the entire image recognition device 30. It may be a processor other than the CPU 301. The ROM 302 stores various programs, such as a program used to drive the CPU 301, and various data. The RAM 303 is used as a work area for the CPU 301, and loads various programs and data stored in the ROM 302 and the storage unit 310. The control unit 300 executes various control processes of the image recognition device 30, for example, by the CPU 301 operating in accordance with a control program 311 stored in the storage unit 310 and loaded in the RAM 303.

[0077] The storage unit 310 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 310 stores a control program 311, a trained model 312, a comparison dictionary 313, and a setting table 314.

[0078] The control program 311 is a program for realizing functions such as a function of receiving a product image from the scanner 20, a function of performing image recognition on the product image, a function of determining whether the image recognition is successful, a function of transmitting the image recognition result to the scanner 20, a function of receiving the decoded result from the scanner 20, a function of determining whether the product image is used for additional learning, a function of transmitting the product image and the decoded result to the server 40, a function of receiving an updated trained model and an updated matching dictionary from the server 40, a function of updating the currently held trained model with the updated trained model, and a function of updating the currently held matching dictionary with the updated matching dictionary. The control program 311 may be stored in the ROM 302 instead of being stored in the storage unit 310.

[0079] The trained model 312 is an image recognition model for performing image recognition on product images. The trained model 312 has been trained by machine learning and is additionally trained by machine learning.

[0080] The matching dictionary 313 is a data table that manages products and their appearance features in association with each other. FIG. 9 is a diagram showing the data structure of the matching dictionary 313. The matching dictionary 313 stores the appearance features of each of a plurality of products to be registered in association with the code value represented by the product barcode placed on the appearance of each product. The matching dictionary 313 may have a plurality of features for one code value. The code value of the matching dictionary 313 is the same as the code value included in the output code of the product master 112 held by the storage unit 110 of the POS terminal 10. Thus, the storage unit 310 functions as a memory that stores the matching dictionary 313. The matching dictionary 313 may store code types and code values ​​in association with the features, instead of the code values. In other words, the matching dictionary 313 may include the code value of the product code of each product and the code type of the code value as the correct answer data for image recognition based on the product features.

[0081] The feature amount is, for example, a parameterized feature of the appearance of the product, such as the standard shape of the product, the color tone, the pattern, and the unevenness of the surface of the product. In addition to the feature amount of the product appearance, the feature amount may also include a parameterized feature of the shape of a sticker label on which a product barcode is printed and the character string or number string written thereon, which is affixed to the product. The control unit 300 acquires the feature amount included in the product image, compares the acquired feature amount with the collation dictionary, and acquires a code value associated with the acquired feature amount. In other words, the control unit 300 acquires a code value of the product shown in the product image from the products pre-recorded in the collation dictionary 313 based on the feature amount acquired from the product image. In addition, when the collation dictionary 313 stores a code type and a code value associated with the feature amount, the control unit 300 can acquire the code type and the code value of the product shown in the product image.

[0082] The setting table 314 includes a data table for managing various settings for the image recognition device 30, which are set from the POS terminal 10 via the LAN via the LAN hub LH or via the LAN via the scanner 20. For example, the data tables included in the setting table 314 include a recognition flag setting table 3141 and a mark type setting table 3142 as shown in FIG. 10. The recognition flag setting table 3141 manages the setting of flag information to be added to the image recognition result as an identifier for distinguishing the decoded result from the image recognition result. The mark type setting table 3142 is similar to the mark type setting table 2121 of the scanner 20. This is a data table for managing code type marks for each code type assumed as a product barcode. When setting via the LAN via the scanner 20, the mark type setting table 3142 may be a copy of the mark type setting table 2121.

[0083] Returning to FIG. 5, the description of the hardware configuration of the image recognition device 30 will be continued. The LAN interface 321 is an interface for communicating with the scanner 20 and the LAN hub LH. The control unit 300 transmits and receives information (data) to and from the scanner 20 and the LAN hub LH via the LAN interface 321.

[0084] For example, the control unit 300 receives a frame image (product image) from the scanner 20 via the LAN interface 321. The control unit 300 also acquires the code value of the product shown in the product image using the trained model 312 and the matching dictionary 313 as described above, and determines the code type in the same manner as the control unit 200 of the scanner 20. If the matching dictionary 313 stores the code type and the code value in association with the feature, the control unit 300 can acquire the code type and the code value from the product image. The control unit 300 identifies the code type mark of the code type using the mark type setting table 3142. The control unit 300 then adds the flag information managed in the recognition flag setting table 3141 and the identified code type mark to the acquired code value to obtain an image recognition result. The control unit 300 transmits the image recognition result, which is made up of the flag information, the code type mark, and the code value, to the scanner 20 via the LAN interface 321.

[0085] Furthermore, the control unit 300 transmits the product image that has been subjected to image recognition and the code value of the decoded result received from the scanner 20 to the server 40 via the LAN interface 321 as necessary. The control unit 300 may also transmit the code type mark of the decoded result. Furthermore, the control unit 300 receives the updated trained model 412 (see FIG. 6) and the updated matching dictionary 413 (see FIG. 6) from the server 40 via the LAN interface 321. The control unit 300 updates the currently held trained model 312 with the updated trained model 412. Furthermore, the control unit 300 updates the currently held matching dictionary 313 with the updated matching dictionary 413.

[0086] (Server hardware configuration) The server 40 receives and stores product images and code values ​​(and code type marks) from the image recognition device 30. The server 40 uses the stored product images and code values ​​(and code type marks) to perform additional learning of a trained model for product image recognition, and updates the trained model and matching dictionary. The server 40 transmits the updated trained model and updated matching dictionary to the image recognition device 30. The server 40 also manages product master data. The server 40 receives transaction processing information from the POS terminal 10, and manages the product master data based on the received transaction processing information. The server 40 transmits to each POS terminal 10 a product master data suitable for that POS terminal 10.

[0087] 6, the server 40 includes a control unit 400, a storage unit 410, and a communication unit 421. The control unit 400, the storage unit 410, and the communication unit 421 are connected to each other via a bus 422 or the like.

[0088] The control unit 400 is configured by a computer including a CPU 401, a ROM 402, and a RAM 403. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 422.

[0089] The CPU 401 controls the entire server 40. It may be a processor other than the CPU 401. The ROM 402 stores various programs, such as a program used to drive the CPU 401, and various data. The RAM 403 is used as a work area for the CPU 401, and loads various programs and data stored in the ROM 402 and the storage unit 410. The control unit 400 executes various control processes of the server 40, for example, by the CPU 401 operating in accordance with a control program 411 stored in the storage unit 410 and loaded in the RAM 403.

[0090] The storage unit 410 is configured with a storage medium such as a HDD or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 410 stores a control program 411, a trained model 412, a comparison dictionary 413, a transaction database 414, and a product master 415.

[0091] The control program 411 is a program for realizing functions such as receiving product images and decoded results from the image recognition device 30, storing product images and decoded results, additionally learning and updating the trained model 412, updating the matching dictionary 413, transmitting the updated trained model 412 and the updated matching dictionary 413 to the image recognition device 30, receiving transaction processing information from the POS terminal 10, managing the product master 415, and transmitting the product master 415 to the POS terminal 10. The control program 411 may be stored in the ROM 402 instead of in the storage unit 410.

[0092] The trained model 412 is a model trained by machine learning for performing image recognition on product images. In detail, the trained model 412 is an image recognition model that acquires a feature amount included in the product image and compares the acquired feature amount with a matching dictionary 413 to acquire a code value of the product depicted in the product image.

[0093] Similar to the matching dictionary 313 held by the image recognition device 30, the matching dictionary 413 is a data table that manages the code value of a product and the appearance feature of the product in association with each other. The matching dictionary 413 may be a data table that manages the code value and code type mark of a product and the appearance feature of the product in association with each other. The matching dictionary 413 includes the matching dictionaries 313 held by multiple image recognition devices 30. The server 40 transmits to each image recognition device 30 the matching dictionary 313 suitable for that image recognition device 30.

[0094] The transaction database 414 stores transaction information for each store, such as which POS terminal 10 sold which product, at what time, for how much, and how many units were sold. The server 40 manages the product master 415 based on the transaction processing information received from the POS terminal 10.

[0095] The product master 415, like the product master 112 held by the POS terminal 10, includes a product code, an output code associated with the product code, a product category, a product name, and a price. The product master 415 is the basis for the product masters 112 held by a plurality of POS terminals 10, and includes information on all products. The product master 112 of the POS terminal 10 includes only information on products sold in its own store. In other words, the product master 112 of the POS terminal 10 is an excerpt of the product master 415 of the server 40. The server 40 transmits to each POS terminal 10 the product master 112 suitable for that POS terminal 10.

[0096] The communication unit 421 is an interface for communicating with the router RT. The control unit 400 transmits and receives information (data) to and from the router RT via the communication unit 421.

[0097] For example, the control unit 400 receives and stores product images and decoded results from the image recognition device 30 via the communication unit 421. The control unit 400 performs additional learning of the trained model 412 using the stored product images and decoded results, and updates the trained model 412 and the matching dictionary 413. The control unit 400 transmits the updated trained model and the updated matching dictionary to the image recognition device 30 via the communication unit 421.

[0098] Additionally, server 40 receives transaction processing information from POS terminal 10 via communication unit 421. Server 40 manages transaction database 414 based on the received transaction processing information. Furthermore, server 40 transmits product master data suitable for each POS terminal 10 to each POS terminal 10 via communication unit 421.

[0099] (Functional configuration of POS terminal) Next, the functional configuration of the POS terminal 10 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10.

[0100] The control unit 100 has an input means 1001, a registration means 1002, an accounting processing means 1003, and an output means 1004. The control unit 100 functions as the input means 1001, the registration means 1002, the accounting processing means 1003, and the output means 1004 by the CPU 101 operating in accordance with a control program stored in the ROM 102 or the storage unit 110. Each of these functions may be configured by hardware such as a dedicated circuit.

[0101] Various information is input to the input means 1001 from the scanner 20, the touch panel 52, the card reader 54, the first weighing unit 71, the second weighing unit 81, etc. For example, an output code is input to the input means 1001 from the scanner 20.

[0102] Furthermore, information based on the operation of the touch panel 52 is input to the input means 1001. For example, transaction start information declaring the start of one transaction is input to the input means 1001. Furthermore, registration completion information indicating that product registration of the product related to one transaction has been completed is input to the input means 1001.

[0103] The registration means 1002 executes product registration of a product that a customer is to purchase through an operation of the customer. In other words, the registration means 1002 registers information on a product identified by product identification information input to the input means 1001. Specifically, the registration means 1002 reads product information from the product master 112 based on the output code input to the input means 1001. Then, the registration means 1002 stores the read product information in the product information section 1031. The product information is information including a product code, a product name, a price, and the like. Thus, the computer having the CPU 101 of the POS terminal 10 as its core portion functions as a registration section that registers information on a product identified by a product code corresponding to a code value based on a code value acquired by a decoding section and a recognition section, which will be described later.

[0104] The transaction processing means 1003 executes transaction processing related to the payment of the prices of the products registered by the registration means 1002. Specifically, the transaction processing means 1003 executes processing to calculate the total price (transaction amount) of the registered products, calculate the amount of change, display various information, issue receipts, etc. If the customer desires cashless payment such as credit card payment or two-dimensional code payment, the transaction processing means 1003 can execute the transaction processing necessary for payment.

[0105] The output unit 1004 outputs various information to the scanner 20, the printer 53, the change machine 55, the light emitting unit 571, the audio output unit 120, etc. For example, when transaction start information is input to the input unit 1001, the output unit 1004 outputs an imaging start signal that instructs the scanner 20 to start imaging. Also, when registration completion information is input to the input unit 1001, the output unit 1004 outputs an imaging stop signal that instructs the scanner 20 to stop imaging.

[0106] When the registration means 1002 completes the product registration for one product, the output means 1004 outputs a control signal for causing the audio output unit 120 to output an alarm sound. The output means 1004 functions as an alarm means for causing an alarm unit (audio output unit 120) to notify that the registration means 1002 has completed the registration of product information. It is also possible to provide the audio output unit 120 in the scanner 20. In this case, the audio output unit 120 may be configured to emit an alarm sound when the scanner 20 identifies the code value.

[0107] Furthermore, when the transaction processing unit 1003 has completed transaction processing, the output unit 1004 transmits transaction processing information to the server 40 via the LAN hub LH.

[0108] (Scanner Functional Configuration) Next, the functional configuration of the scanner 20 will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the functional configuration of the control unit 200 of the scanner 20.

[0109] The control unit 200 has an acquiring means 2001, a decoding means 2002, a transmitting / receiving means 2003, an output code creating means 2004, and a transmitting means 2005. The control unit 200 functions as the acquiring means 2001, the decoding means 2002, the transmitting / receiving means 2003, the output code creating means 2004, and the transmitting means 2005 by the CPU 201 operating in accordance with a control program stored in the ROM 202 or the storage unit 210. Each of these functions may be configured by hardware such as a dedicated circuit.

[0110] The acquisition means 2001 acquires one frame image as a product image from the moving image captured by the imaging unit 220. The acquisition means 2001 stores the acquired product image in the RAM 203. For example, the acquisition means 2001 acquires all or some of the multiple frame images sequentially captured by the imaging unit 220 as candidates for product images, stores all or some of the acquired frame images in the RAM 203, and selects a product image from all or some of the stored frame images. Thus, the computer with the CPU 201 of the scanner 20 as its core portion functions as an image acquisition unit that captures images of products to be registered.

[0111] In addition, the acquiring unit 2001 acquires various other information, for example, an image capture start signal and an image capture stop signal from the POS terminal 10.

[0112] The decoding means 2002 detects the product barcode included in the product image acquired by the acquisition means 2001 and extracts the product barcode from the product image. The decoding means 2002 decodes the extracted product barcode. As a result of decoding the product barcode, the decoding means 2002 acquires a code value held by the product barcode and a code type mark indicating a code type determined from the code value. Thus, the computer with the CPU 201 of the scanner 20 as its core portion functions as a decoding unit that acquires the code value represented by the code symbol by decoding the code symbol from an image of the code symbol included in a captured image of the product to be registered.

[0113] The transmitting / receiving means 2003 transmits the commodity image acquired by the acquiring means 2001 to the image recognition device 30. Thus, the computer with the CPU 201 of the scanner 20 as its central part functions as an image capturing unit that captures the commodity to be registered, and outputs the captured image to the image recognition device 30 in addition to the decoding unit. The image recognition device 30 performs image recognition on the received commodity image, and transmits the image recognition result (the code value of the commodity shown in the commodity image, the code type mark added thereto, and flag information) to the scanner 20. The transmitting / receiving means 2003 receives the image recognition result transmitted by the image recognition device 30. Thus, the computer with the CPU 201 of the scanner 20 as its central part functions as a code value receiving unit that receives the code value output from the image recognition device 30. The transmitting / receiving means 2003 also transmits the decoded result by the decoding means 2002 (the code value and the code type mark held by the commodity barcode) to the image recognition device 30.

[0114] The output code creation means 2004 creates an output code from the decoded result (the code value and code type mark held by the product barcode) and the image recognition result (the code value of the product shown in the product image and the code type mark and flag information added thereto). That is, the output code creation means 2004 deletes or maintains the flag information from the image recognition result according to the transmission mode set in the transmission mode setting table 2122, and creates an output code including the decoded result and the image recognition result. Thus, the computer with the CPU 201 of the scanner 20 as its core portion functions as a creation unit that creates an output code to be output to the registration unit from the code value acquired by the decode unit, the code value acquired by a recognition unit described later, and the flag information added by an addition unit described later.

[0115] If no decoded result is obtained, the output code generating means 2004 generates an output code including only the image recognition result. Situations in which no decoded result is obtained include when the customer does not properly hold the product barcode over the image capturing unit 220 of the scanner 20, or when the product barcode is unnaturally bent or folded.

[0116] The transmitting means 2005 transmits the output code created by the output code creating means 2004 to the POS terminal 10 via the connection interface 221. Thus, the computer having the CPU 201 of the scanner 20 as its central part functions as a code value transmitting section which transmits at least one of the code value acquired by the decoding section and the code value received by the code value receiving section to the registration device.

[0117] (Functional configuration of image recognition device) Next, the functional configuration of the image recognition device 30 will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the functional configuration of the control unit 300 of the image recognition device 30.

[0118] The control unit 300 has a receiving means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, and a transmission means 3005. The control unit 300 functions as the receiving means 3001, the image recognition means 3002, the determination means 3003, the update means 3004, and the transmission means 3005 by the CPU 301 operating in accordance with a control program stored in the ROM 302 or the storage unit 310. Each of these functions may be configured by hardware such as a dedicated circuit.

[0119] The receiving means 3001 receives the product images from the scanner 20 via the LAN interface 321. Thus, the computer having the CPU 301 of the image recognition device 30 as its core portion functions as a receiving unit that receives images of the product to be registered, which are transmitted from the scanner 20, which is a higher-level device. The receiving means 3001 stores the received product images in the RAM 303.

[0120] The image recognition means 3002 performs image recognition on the commodity image by an image recognition function using the trained model 312. In detail, the image recognition means 3002 acquires a feature amount included in the commodity image, compares the acquired feature amount with the comparison dictionary 313, and acquires a code value associated with the feature amount closest to the acquired feature amount. Thus, the computer having the CPU 301 of the image recognition device 30 as its central part functions as an acquisition unit that acquires a code value of the photographed commodity by comparing the appearance image of the commodity included in the image with the comparison dictionary 313. The image recognition means 3002 determines the code type from the acquired code value. Note that, if the comparison dictionary 313 stores the code type and the code value in association with the feature amount, the image recognition means 3002 can acquire the code type and the code value from the commodity image. The image recognition means 3002 determines the code type mark of this mark type according to the setting of the mark type setting table 3142, and adds the determined code type mark to the acquired code value. Furthermore, the image recognition means 3002 adds flag information set in the recognition flag setting table 3141 to the code value to which this code type mark has been added. The image recognition means 3002 stores the code value to which the flag information and the code type mark have been added in this way in the RAM 303 as the image recognition result. Thus, the computer having the CPU 301 of the image recognition device 30 as its core portion functions as a recognition unit that acquires the code value by recognizing the appearance of the product from the appearance image of the product included in the image obtained by photographing the product to be registered.

[0121] The transmission means 3005 transmits the image recognition result by the image recognition means 3002 stored in the RAM 303 to the transmission / reception means 2003 of the scanner 20 via the LAN interface 321. Thus, the computer having the CPU 301 of the image recognition device 30 as its central part functions as a transmission section that transmits the code value to a higher-level device.

[0122] The determination means 3003 determines whether the image recognition by the image recognition means 3002 has been successful. For example, if the difference between the acquired feature amount and the feature amount closest to the acquired feature amount is less than a predetermined amount, the determination means 3003 determines that the image recognition has been successful, and conversely, if the difference is equal to or greater than the predetermined amount, the determination means 3003 determines that the image recognition has failed. In addition, if the determination means 3003 determines that the image recognition has failed, it determines that the product image that has been subjected to image recognition is a product image to be used for additional learning.

[0123] Moreover, the receiving means 3001 receives the decoded result from the scanner 20 via the LAN interface 321. The receiving means 3001 stores the received decoded result in the RAM 303.

[0124] When the determination means 3003 determines that the product image that has been subjected to image recognition is a product image to be used for additional learning, the transmission means 3005 transmits the product image and the decoded result thereof to the server 40 via the LAN interface 321. In other words, since it has been determined that the image recognition has failed, the reliability of the image recognition result by the image recognition means 3002 is low. Therefore, the decoded result of the product barcode is transmitted to the server 40 as data for additional learning.

[0125] The server 40 performs additional learning of the trained model 412 for commodity image recognition using the received commodity image and the decoded result, and updates the trained model 412 and the matching dictionary 413. The server 40 transmits the updated trained model 412 and the updated matching dictionary 413 to the receiving means 3001 of the image recognition device 30 via the LAN interface 321.

[0126] The receiving means 3001 receives the updated trained model 412 and the updated matching dictionary 413 from the server 40 via the LAN interface 321. The receiving means 3001 stores the received updated trained model 412 and updated matching dictionary 413 in the RAM 303.

[0127] The update means 3004 updates the trained model 312 currently held in the storage unit 310 with the received updated trained model 412. In addition, the image recognition device 30 updates the matching dictionary 313 currently held in the storage unit 310 with the received updated matching dictionary 413.

[0128] (Server Functionality Configuration) Next, the functional configuration of the server 40 will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the functional configuration of the control unit 400 of the server 40.

[0129] The control unit 400 has a receiving means 4001, an additional learning means 4002, a master management means 4003, a transmitting means 4004, and a transaction management means 4005. The control unit 400 functions as the receiving means 4001, the additional learning means 4002, the master management means 4003, the transmitting means 4004, and the transaction management means 4005 by the CPU 401 operating in accordance with a control program stored in the ROM 402 or the storage unit 410. Each of these functions may be configured by hardware such as a dedicated circuit.

[0130] The receiving means 4001 receives a product image for additional learning to be used for additional learning and a decoded result of the product image for additional learning from the image recognition device 30 via the communication unit 421. The receiving means 4001 stores the received product image and the decoded result in the RAM 403. The receiving means 4001 may store and accumulate the received product image in the RAM 403 or the storage unit 410 as a product image for analyzing a state (event) of image recognition in the image recognition device 30, for example.

[0131] The additional learning means 4002 performs additional learning on the trained model 412 held in the storage unit 410 at an appropriate timing, for example, when a certain amount of product images for additional learning and decoded results have been accumulated, and updates the trained model 412. In addition, the additional learning means 4002 updates the matching dictionary 413 held in the storage unit 410 in accordance with the additional learning of the trained model 412.

[0132] The transmitting means 4004 transmits the updated trained model 412 and the matching dictionary 413 updated by the additional learning means 4002 to the receiving means 3001 of the image recognition device 30 via the communication unit 421, for example as an image file or a compressed file.

[0133] The receiving means 4001 also receives transaction processing information from the POS terminal 10 via the communication unit 421. The receiving means 4001 stores the received transaction processing information in RAM 403.

[0134] The transaction management means 4005 manages the transaction database 414 held in the memory unit 410 based on the received transaction processing information.

[0135] The master management means 4003 updates the information of the product master 415 based on update information for the product master 415 inputted from an input means (not shown) or received by the receiving means 4001 from a management terminal (not shown).

[0136] The transmission means 4004 transmits the product master 112 suitable for each store to the POS terminal 10 of the store via the communication unit 421 based on the product master 415 updated by the master management means 4003.

[0137] (Scanner and image recognition device operation) Next, the operation of the scanner 20 and the image recognition device 30 involved in the registration of purchased merchandise in the merchandise information management system MS will be described with reference to FIGS.

[0138] Fig. 15 is a sequence diagram showing an outline of the operation of the product information management system MS. As shown in Fig. 15, the process related to the registration of purchased products performed by the product information management system MS includes a product registration process PRO1 that is repeatedly performed each time a product is registered in the product information registration system RS, and a learning update process PRO2 that is performed at any time. As described above, the learning update process PRO2 is performed, for example, when a certain amount of product images and decoded results for additional learning have been accumulated.

[0139] First, the product registration process PRO1 will be described. When the scanner 20 captures an image of a product (step S11), the scanner 20 determines whether or not a product barcode has been detected from the captured product image (step S12). If the product barcode has not been detected, the process returns to step S11.

[0140] Moreover, the scanner 20 transmits the product image captured in step S11 to the image recognition device 30.

[0141] When the image recognition device 30 receives the product image, the image recognition device 30 recognizes the image of the product shown in the captured image, thereby performing a process of acquiring an image recognition result including a code value of the product (step S13).

[0142] The image recognition device 30 judges whether the image recognition has been successful, that is, whether the image recognition result has been acquired (step S14), and if the image recognition has not been successful, transmits a product image request to the scanner 20 to request transmission of a product image. As a result, a new product image is transmitted from the scanner 20, and image recognition processing can be performed on the new product image.

[0143] If the image recognition is successful, the image recognition device 30 transmits the image recognition result, that is, the code value to which the flag information and the code type mark are added, to the scanner 20.

[0144] If the scanner 20 detects a product barcode from the product image, the scanner 20 decodes the product barcode and acquires the decoded result (step S15). The scanner 20 transmits the acquired decoded result, i.e., the code value to which the code type mark is added, to the image recognition device 30.

[0145] Then, the scanner 20 creates an output code from the acquired decoded result, i.e., the code value to which the code type mark has been added, and the image recognition result transmitted from the image recognition device 30, i.e., the flag information and the code value to which the code type mark has been added (step S16). The scanner 20 transmits the created output code to the POS terminal 10.

[0146] The POS terminal 10 performs registration processing of the purchased item based on the output code transmitted from the scanner 20 (step S17). When the registration processing of the item is completed, the POS terminal 10 can output an alarm sound to notify the completion of the registration processing.

[0147] In addition, the image recognition device 30 determines whether additional learning is necessary, that is, whether there is a product image to be used for additional learning (step S18). If additional learning is necessary, the image recognition device 30 transmits learning data including the product image and the obtained decoded result to the server 40.

[0148] The product information registration system RS performs the above-described product registration process PRO1 for each product. Then, when the registration of all products is completed, that is, when the product registration of products related to one transaction is completed, the POS terminal 10 performs the transaction process (step S21).

[0149] Next, the learning update process PRO2 will be described. In Fig. 15, the learning update process PRO2 is described as being performed following the product registration process PRO1, but it can be performed simultaneously with the product registration process PRO1.

[0150] When a predetermined amount of learning data is accumulated, the server 40 uses the learning data to perform additional learning on the trained model 412 held by the server 40, thereby updating the model (step S31). In addition, the server 40 updates the matching dictionary 413 held by the server 40 in accordance with the additional learning of the trained model 412 (step S32). The server 40 then transmits the updated trained model 412 and matching dictionary 413 to the image recognition device 30, for example, as update data for an image file or compressed file.

[0151] Upon receiving this update data, the image recognition device 30 updates the trained model 312 and the matching dictionary 313 held by the image recognition device 30 (step S33). This update may be performed immediately upon receiving the update data, but since the update process requires a certain amount of time, it is not preferable to perform the update in the middle of the product registration process PRO1, since this makes customers wait. Therefore, the update is performed when the image recognition device 30 starts operating, such as when the store opens, or during batch processing at night. Alternatively, the update timing in the image recognition device 30 may be controlled by adjusting the time when the update data is transmitted from the server 40.

[0152] FIG. 16 is a flowchart showing the flow of operations of the scanner 20 and image recognition device 30 of the embodiment when registering one product. In FIG. 16, the operations of the scanner 20 are shown on the left side, and the operations of the image recognition device 30 are shown on the right side. The operations of the scanner 20 are performed by each part of the scanner 20 under the control of the control unit 200 that executes a control program 211. The operations of the image recognition device 30 are performed by each part of the image recognition device 30 under the control of the control unit 300 that executes a control program 311. Unless otherwise specified, the processing operations of the scanner 20 and the image recognition device 30 shown in FIG. 16 transition from ACTn (n is a natural number) to ACT(n+1).

[0153] In ACT11, the scanner 20 acquires one product image from a moving image captured by the imaging unit 220. For example, the scanner 20 first stores in the RAM 203 all or some of the multiple frame images captured sequentially by the imaging unit 220 as product image candidates. Next, the scanner 20 selects one frame image from the multiple frame images stored in the RAM 203, and sets the selected frame image as the product image. The scanner 20 also assigns an identification code to the product image for identifying the product image. In the following, for convenience, the product image is assumed to include the identification code.

[0154] In ACT12, the scanner 20 transmits the product image to the image recognition device 30 via the LAN, for example, by TCP communication. Thus, the CPU 201 executes information processing based on the control program, and the computer with the CPU 201 as the central part functions as an image capturing unit that outputs the captured image of the product to be registered to the image recognition device 30 in addition to the decoding unit.

[0155] In ACT21, the image recognition device 30 receives the product image and the identification code from the scanner 20 via the LAN. Thus, the CPU 301 executes information processing based on the product image recognition program, and the computer with the CPU 301 as the core part functions as a receiving unit that receives images of the product to be registered, which are transmitted from a higher-level device. The image recognition device 30 stores the received product image in the RAM 303.

[0156] In ACT22, the image recognition device 30 performs image recognition on the commodity image. For example, the image recognition device 30 performs image recognition based on the feature amount of the commodity, such as the outline, color, and character of the commodity shown in the commodity image. In this case, the image recognition device 30 first acquires the feature amount included in the commodity image. Next, the image recognition device 30 checks the acquired feature amount against the comparison dictionary 313 to search for the feature amount closest to the acquired feature amount. Then, the image recognition device 30 acquires a code value associated with the search result (the feature amount closest to the acquired feature amount). Thus, the CPU 301 executes information processing based on the commodity image recognition program, and the computer with the CPU 301 as the core part functions as an acquisition unit that acquires the code value of the photographed commodity by checking the appearance image of the commodity included in the photographed commodity of the commodity to be registered, which is transmitted from the upper device, against the comparison dictionary.

[0157] For example, consider the case where a product is registered with a product barcode having a JAN-13 code with a code value of "4912345678904". The matching dictionary 313 of the image recognition device 30 stores the code value "4912345678904" in association with the feature of the product. In this manner, in this embodiment, correct answer data of the image recognition result for each product sold by the store is specified in advance in the image recognition device 30. In this manner, the image recognition device 30 acquires a code value consisting of a 13-digit number, "4912345678904", from the product image. The image recognition device 30 identifies the code type as "JAN-13" based on the number of digits of this code value. Note that, when the matching dictionary 313 stores the code type and the code value in association with the feature, the image recognition device 30 can acquire the code type "JAN-13" and the code value "4912345678904" from the product image. The image recognition device 30 acquires a code type mark, for example, "F", preset for this code type from the mark type setting table 3142. If the code type cannot be identified, the most likely code type, for example, the code type with the closest number of digits, is assigned. In this embodiment, the format of the image recognition result of the image recognition device 30 is defined as "flag information indicating that it is an image recognition result" + "code type mark" + "code value". Then, the image recognition device 30 acquires preset flag information, for example, "@", from the recognition flag setting table 3141. Therefore, the image recognition result acquired by the image recognition device 30 is "@F4912345678904". Thus, the CPU 301 executes information processing based on the control program, and the computer with the CPU 301 as the core part functions as an adding unit that adds flag information to distinguish the code value acquired by the recognition unit from the code value acquired by the decoding unit.

[0158] In ACT23, the image recognition device 30 determines whether the image recognition has been successful. For example, the image recognition device 30 first obtains a difference between the acquired feature amount and a feature amount closest to the acquired feature amount. Next, the image recognition device 30 compares the obtained difference with a predetermined amount.

[0159] If the comparison result shows that the difference is less than the predetermined amount, the image recognition device 30 determines that the image recognition has been successful. In this case (Yes in ACT23), the image recognition device 30 stores the image recognition result acquired in ACT22 in the RAM 303. Then, the operation proceeds to ACT25.

[0160] On the other hand, if the comparison result shows that the difference is equal to or greater than the predetermined amount, the image recognition device 30 determines that the image recognition has failed. In this case (No in ACT23), the image recognition device 30 proceeds to the operation of ACT24.

[0161] In ACT24, the image recognition device 30 transmits a product image request to the scanner 20 to request acquisition and transmission of another product image, and performs the operations of ACT21 to ACT23 again. In addition, the image recognition device 30 determines the product image for which image recognition has failed as a product image to be used for additional learning, and stores the determined product image in the RAM 303.

[0162] In ACT25, the image recognition device 30 transmits the image recognition result stored in the RAM 303 to the scanner 20 via the LAN. Thus, the CPU 301 executes information processing based on the commodity image recognition program, and the computer with the CPU 301 as the central part functions as a transmission unit that transmits the obtained commodity code value to a higher-level device. The image recognition device 30 also transmits to the scanner 20 the identification code of the commodity image from which the image recognition result was obtained.

[0163] In ACT13, the scanner 20 detects a product barcode from the product image. In other words, the scanner 20 determines whether or not a product barcode is captured in the product image.

[0164] If the product barcode cannot be detected in ACT13 (No in ACT13), the process returns to ACT11 and the next product image is acquired. In this case, the operation of ACT12 may be skipped and the operation of detecting the product barcode in ACT13 may be performed again. If the operation of ACT12 is not skipped, additional image recognition may be performed using the product image in the image recognition device 30, or if it has already been determined that recognition was successful in ACT23, the product image may be ignored. Examples of situations in which the product barcode cannot be detected include when the customer does not hold the product barcode over the imaging unit 220 of the scanner 20 correctly, or when the product barcode is unnaturally bent or folded.

[0165] If the product barcode is detected in ACT13 (Yes in ACT13), the scanner 20 extracts and decodes the detected product barcode in ACT14. For example, if the product is provided with a product barcode having the above-mentioned JAN-13 code value "4912345678904", the scanner 20 decodes the product barcode to obtain a code value of "4912345678904", which is a 13-digit number. The scanner 20 determines that the code type is "JAN-13" based on the number of digits of this code value, and obtains a code type mark, for example "F", that is preset for this code type, from the mark type setting table 2121. The format of the decoded result of the scanner 20 is defined as "code type mark" + "code value". Therefore, the decoded result obtained by the scanner 20 is "F4912345678904".

[0166] In ACT 15, the scanner 20 stores the obtained decoded result in the RAM 203.

[0167] Next, in ACT16, the scanner 20 transmits the decoded result (the code type mark+the code value) to the image recognition device 30.

[0168] In ACT17, the scanner 20 checks whether it has received the image recognition result of the product image ("flag information indicating that this is an image recognition result" + "code type mark" + "code value") from the image recognition device 30. If the image recognition result has not been received (No in ACT17), the scanner 20 repeats the operation of ACT17. In other words, the scanner 20 waits to receive the image recognition result.

[0169] If an image recognition result has been received (Yes in ACT17), in ACT18, the scanner 20 stores the received image recognition result in the RAM 203. Thus, the CPU 201 executes information processing based on the control program, and the computer with the CPU 201 as its central part functions as a code value receiving unit that receives the code value output from the image recognition device 30. The scanner 20 may identify a product image corresponding to the image recognition result based on the identification code received together with the image recognition result.

[0170] In ACT19, the scanner 20 executes an output code creation process to create an output code including the decoded result ("code type mark" + "code value") and the image recognition result ("flag information indicating that it is an image recognition result" + "code type mark" + "code value"). The details of this output code creation process will be described later.

[0171] In ACT110, the scanner 20 transmits the generated output code to the POS terminal 10. Thus, the CPU 201 executes information processing based on the control program, and the computer with the CPU 201 as the core functions as a code value transmitting section that transmits the code value acquired by the decoding section and the code value received by the code value receiving section to the registration device. Then, the scanner 20 finishes the operation for registering this one product.

[0172] When the scanner 20 transmits the decoded result ("code type mark" + "code value") in ACT 16, the image recognition device 30 receives the decoded result in ACT 26. The image recognition device 30 stores the received decoded result in the RAM 303.

[0173] In ACT 27, the image recognition device 30 determines whether there is a product image to be used for additional learning. For example, if there is a product image for which image recognition has failed in ACT 23 and a decoded result has been received from the scanner 20 in ACT 26, the image recognition device 30 determines that there is a product image to be used for additional learning.

[0174] If it is determined in ACT27 that there is a product image to be used for additional learning (Yes in ACT27), in ACT28, the image recognition device 30 transmits the product image that has failed image recognition and is stored in the RAM 303, and the decoded result received from the scanner 20, to the server 40. Then, the image recognition device 30 ends the operation for registering this one product.

[0175] If it is determined in ACT27 that there are no product images to use for additional learning (No in ACT27), the image recognition device 30 skips the operation in ACT28 and ends the operation for registering this one product.

[0176] FIG. 17 is a flowchart showing details of the output code generation process in FIG. In ACT191, the scanner 20 creates an output code including the decoded result ("code type mark" + "code value") and the image recognition result ("flag information" + "code type mark" + "code value").

[0177] In ACT192, the scanner 20 checks the transmission mode registered in the transmission mode setting table 2122. In the transmission mode setting table 2122, a transmission mode is set which indicates whether the POS terminal 10, which is the higher-level device, distinguishes between the decoded result and the image recognition result or does not distinguish between them. For example, in the case where the POS terminal 10 presents to the customer whether the product has been recognized from the product barcode or from the product package, i.e., the product's appearance, a transmission mode which distinguishes the results is set.

[0178] In ACT193, the scanner 20 determines whether the transmission mode is a mode in which the decoded result and the image recognition result are distinguished in the output code.

[0179] If it is determined in ACT193 that the transmission mode is not one that distinguishes between the two results (No in ACT193), the scanner 20 deletes the flag information in the image recognition result from the created output code in ACT194. For example, if the decoded result is "F4912345678904" and the image recognition result is "@F4912345678904" as in the above example, the "@" is deleted from the image recognition result "@F4912345678904". As a result, the output code includes two "F4912345678904". Then, the scanner 20 ends this output code creation process and moves to ACT110 in the upper routine. Therefore, the scanner 20 transmits the output code including the two "F4912345678904" to the POS terminal 10.

[0180] Also, if it is determined in ACT193 that the transmission mode is one in which both results are distinguished (if Yes in ACT193), the scanner 20 ends this output code creation process and moves to ACT110 in the upper routine. Therefore, the "@" in "@F4912345678904", which is the image recognition result in the output code, is maintained. Therefore, the scanner 20 transmits an output code including both the decoded result "F4912345678904" and the image recognition result "@F4912345678904" to the POS terminal 10.

[0181] In the product master 112 of the POS terminal 10, both the decoded result "F4912345678904" and the image recognition result "@F4912345678904" are registered as an output code corresponding to a certain product code. Therefore, the POS terminal 10 can identify the record of the corresponding product code from the product master 112 based on the output code transmitted from the scanner 20, and register the purchased product based on the product name and price registered in that record.

[0182] As described above, the product information registration system RS according to the first embodiment includes a decoding means 2002 for decoding a product barcode from an image of the product barcode included in a product image, which is an image of a product to be registered, to obtain a code value represented by the product barcode, an image recognition means 3002 for recognizing the appearance of the product from an appearance image of the product included in the product image of the product to be registered, and a registration means 1002 for registering information about a product identified by a product code corresponding to the code value based on the code value obtained by the decoding means 2002 and the image recognition means 3002. In this way, the same information as the barcode is defined as the correct answer data for the image recognition, and the output code of the code recognition and the image recognition is formed according to the purpose, thereby allowing the code recognition and the image recognition to coexist. That is, the code value is included in the output result of the image recognition, and the output result is converted into a code in the same format as the result of the code recognition. Therefore, even when the code recognition and the image recognition are used together, it is possible to prevent the same product from being registered twice.

[0183] Note that the product information registration system RS according to the first embodiment further includes a matching dictionary 313 that associates appearance features and code values ​​for each of a plurality of products to be registered, and the image recognition means 3002 acquires the code value by matching the appearance features of the product with the matching dictionary 313. In this way, by defining the same information as the barcode as the correct answer data for image recognition stored in the matching dictionary 313, it is possible to prevent duplicate registration of the same product even when code recognition and image recognition are used in combination.

[0184] Moreover, according to the product information registration system RS of the first embodiment, the image recognition means 3002 adds flag information to the acquired code value to distinguish it from the code value acquired by the decoding means 2002. Therefore, it becomes possible to easily determine whether the code value is a code value acquired by the decoding means 2002 or a code value acquired by the image recognition means 3002, depending on whether the flag information is added or not.

[0185] Moreover, the product information registration system RS according to the first embodiment further includes an output code creation means 2004 that creates an output code to be output to the registration means 1002 from the code value acquired by the decoding means 2002 and the code value and flag information acquired by the image recognition means 3002, and the registration means 1002 registers information on the product identified by the product code corresponding to the code value, based on the output code created by the output code creation means 2004. This enables the registration means 1002 to easily acquire the code value acquired by the decoding means 2002 and the code value and flag information acquired by the image recognition means 3002.

[0186] The output code generating means 2004 generates an output code based on the setting of whether or not to include flag information in the output code, which is stored in the transmission mode setting table 2122 from the registration means 1002. Therefore, by setting in advance, it becomes possible to switch between an output code including flag information and an output code not including it.

[0187] Moreover, according to the product information registration system RS of the first embodiment, the decoding means 2002 and the image recognition means further acquire the code type based on the acquired code value. Therefore, not only the code value but also the code type can be acquired and included in the output code.

[0188] The product information registration system RS includes a scanner 20 including an acquisition means 2001 and a decoding means 2002 for photographing a product to be registered and acquiring an image, an image recognition device 30 including an image recognition means 3002, and a POS terminal 10 including a registration means 1002, and the scanner 20 photographs a product to be registered and outputs the photographed image to the image recognition device 30 in addition to the decoding means 2002, and further includes a transmission / reception means 2003 for receiving a code value output from the image recognition device 30, and a transmission means 2005 for transmitting at least one of the code value acquired by the decoding means 2002 and the code value received by the transmission / reception means 2003 to the POS terminal 10. Thus, the product information registration system RS can be configured by the POS terminal 10, the scanner 20, and the image recognition device 30.

[0189] As described above, the information processing device according to the first embodiment includes an acquisition unit 2001 for photographing a product to be registered and acquiring a product image, a scanner 20 for decoding a product barcode from an image of the product barcode included in the product image acquired by the acquisition unit 2001 to acquire a code value represented by the product barcode, and an image recognition device 30 for acquiring a product image of a product to be registered output by the scanner 20, recognizing the appearance of the product from an appearance image of the product included in the product image, and outputting the acquired code value to the scanner 20, and the scanner 20 transmits the acquired code value and the code value output by the image recognition device 30 to the POS terminal 10, which is a higher-level device. This makes it possible to prevent the code value of the code recognition result from being different from the code value of the image recognition result, and therefore makes it possible to prevent the same product from being registered twice even when both code recognition and image recognition are used in the POS terminal 10.

[0190] As described above, the image recognition device 30 according to the first embodiment includes a matching dictionary 313 that associates the appearance of each of a plurality of products to be registered with a code value represented by a product barcode placed on the appearance of each product, a receiving means 3001 that receives a product image of a product to be registered transmitted from the scanner 20, which is a higher-level device, an image recognition means 3002 that acquires a code value of the photographed product by matching an appearance image of the product included in the product image with the matching dictionary 313, and a transmitting means 3005 that transmits the code value to the scanner 20. In this way, by defining the same information as the barcode as the correct answer data for image recognition stored in the matching dictionary 313, it is possible to prevent the same product from being registered twice even when code recognition and image recognition are used in combination.

[0191] [Second embodiment] In the first embodiment, the product information registration system RS is composed of three devices: a POS terminal 10, a scanner 20, and an image recognition device 30. That is, the functions of the product information registration system RS are distributed among the three devices. In contrast, the product information registration system according to the second embodiment is an example in which the functions of the product information registration system RS are integrated into one device.

[0192] 18 is a block diagram showing a hardware configuration of a product information registration system RS according to the second embodiment. The product information registration system RS is composed of a POS terminal 10.

[0193] In this embodiment, the storage unit 110 of the POS terminal 10 stores, in addition to the configuration described in the first embodiment, a setting table 212 and a processing result 213 which are the hardware configuration of the scanner 20, and a trained model 312, a matching dictionary 313, and a setting table 314 which are the hardware configuration of the image recognition device 30. The control program 111 stored in the storage unit includes a program equivalent to the control program 211 of the scanner 20 and the control program 311 of the image recognition device 30. The POS terminal 10 also includes an imaging unit 220 which is the hardware configuration of the scanner 20.

[0194] 19 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10. In addition to the functional means described in the first embodiment, the control unit 100 has an acquisition means 2001, a decoding means 2002, and an output code creation means 2004, which are functional means of the control unit 200 of the scanner 20, and a reception means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, and a transmission means 3005, which are functional means of the control unit 300 of the image recognition device 30.

[0195] In this way, even if the functions of the product information registration system RS are integrated into the POS terminal 10, the same operations as in the first embodiment are possible, and therefore the same effects can be obtained.

[0196] (others) The above-described embodiment can be modified in various ways as follows.

[0197] For example, the flow of the information processing performed by the control unit 200 of the scanner 20 and the control unit 300 of the image recognition device 30 is an example, and is not limited to this order. For example, the order of ACT25 and ACT26 in the flowchart of Fig. 16 may be reversed, or they may be performed in parallel. In this way, the order of the processing may be changed or multiple processing may be performed in parallel, as long as there is no discrepancy with the preceding or subsequent processing.

[0198] 15 and 16, an output code is created when both the decoded result and the image recognition result are obtained. However, depending on the appearance of the product in the product image to be recognized, it may be possible that both results are not obtained. In such a case, the scanner 20 may create an output code from only one of the results obtained, for example, when a certain amount of time has passed since the product was first photographed. This allows the POS terminal 10 to register information about a product identified by a product code that corresponds to a code value contained in at least one of the decoded result and the recognition result.

[0199] In addition, although the scanner 20 is always configured to transmit the decoded results to the image recognition device 30, if the image recognition device 30 determines that additional learning is necessary, it may request the scanner 20 to transmit the decoded results, and the scanner 20 may transmit the decoded results in response to this request.

[0200] In the above embodiment, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 may be configured to be provided by recording them on a computer-readable recording medium such as a CD-ROM. Also, the control programs executed by the POS terminal 10, the scanner 20, the image recognition device 30, and the server 40 in the above embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading them via the network, or may be configured to be provided via a network such as the Internet.

[0201] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0202] 1... Self-service POS system, 10... POS terminal, 100, 200, 300, 400... Control unit, 1001... Input means, 1002... Registration means, 1003... Accounting means, 1004... Output means, 101, 201, 301, 401... CPU, 102, 202, 302, 402... ROM, 103, 203, 303, 403... RAM, 1031... Product information unit, 110, 210, 310, 410... Storage unit, 111, 211, 311, 411... Control program, 112, 415... Product master, 1 20... audio output unit, 121, 221... connection interface, 122, 222, 321... LAN interface, 123, 223, 322, 422... bus, 50... main body, 51... display, 52... touch panel, 53... printer, 531... issue port, 54... card reader, 541... card insertion port, 55... change machine, 551... coin insertion port, 552... coin discharge port, 553... coin receiving tray, 554... banknote insertion port, 555... banknote discharge port, 57... pole, 571... light emitting unit, 70... first basket mounting stand, 71...first weighing section, 80...second basket stand, 81...second weighing section, 82...support rod, 83...temporary stand, 20...scanner, 2001...acquisition means, 2002...decoding means, 2003...transmission / reception means, 2004...output code creation means, 2005...transmission means, 212, 314...setting table, 2121, 3142...mark type setting table, 2122...transmission mode setting table, 213...processing result, 220...imaging section, 230...reading window, 30...image recognition device, 3001...reception means, 300 2...image recognition means, 3003...determination means, 3004...update means, 3005...transmission means, 312, 412...trained model, 313, 413...matching dictionary, 3141...recognition flag setting table, 40...server, 4002...additional learning means, 4003...master management means, 4004...transmission means, 4005...transaction management means, 414...transaction database, 421...communication unit, LH...LAN hub, MS...product information management system, NW...network, RS...product information registration system, RT...router.

Claims

1. a decoding unit that obtains a code value represented by a code symbol by decoding the code symbol from an image of the code symbol included in an image of a product to be registered; a recognition unit that acquires the code value by recognizing an appearance of the product from an appearance image of the product included in the image obtained by photographing the product to be registered; a registration unit that registers information about the product identified by a product code corresponding to the code value acquired by at least one of the decoding unit and the recognition unit, based on the code value; A product information registration system comprising:

2. The method further includes a comparison dictionary in which the appearance features of each of a plurality of products to be registered are associated with a code value, the recognition unit obtains the code value by comparing the appearance feature of the product with the comparison dictionary.

2. The product information registration system according to claim 1.

3. The code value acquired by the recognition unit may further include an adding unit that adds flag information to the code value acquired by the recognition unit to distinguish the code value from the code value acquired by the decoding unit. The product information registration system according to claim 2.

4. a creation unit that creates an output code to be output to the registration unit from the code value acquired by the decoding unit, the code value acquired by the recognition unit, and the flag information added by the addition unit; the registration unit registers information about the product identified by a product code corresponding to the code value based on the output code created by the creation unit; The product information registration system according to claim 3.

5. the creation unit creates the output code based on a setting as to whether or not the flag information is to be included in the output code. The product information registration system according to claim 4.

6. The product information registration system includes: a scanner including an image acquisition unit for acquiring an image of the product to be registered and the decoding unit; A commodity image recognition device including the recognition unit; A registration device including the registration unit; having The scanner includes: A photographing unit that photographs the product to be registered and outputs the photographed image to the decoding unit and also to the product image recognition device; A code value receiving unit that receives the code value output from the commodity image recognition device; a code value transmitting unit configured to transmit at least one of the code value acquired by the decoding unit and the code value received by the code value receiving unit to the registration device; Further comprising:

2. The product information registration system according to claim 1.

7. a scanner including an image acquisition unit for photographing a product to be registered and acquiring an image thereof, the scanner acquiring a code value represented by a code symbol by decoding the code symbol from an image of the code symbol included in the image acquired by the image acquisition unit, and outputting the image acquired by the image acquisition unit; an image recognition device that acquires the image of the product to be registered output by the scanner, recognizes the appearance of the product from an appearance image of the product included in the image, acquires the code value, and outputs the acquired code value to the scanner; Equipped with The scanner transmits at least one of the acquired code value and the code value output by the image recognition device to a higher-level device. Information processing device.

8. a comparison dictionary that associates the appearance of each of a plurality of products to be registered with a code value represented by a code symbol placed on the appearance of each product; A receiving unit that receives an image of the product to be registered, the image being transmitted from a higher-level device; an acquisition unit that acquires the code value of the photographed product by comparing an appearance image of the product included in the image with the comparison dictionary; a transmitter for transmitting the code value to the higher-level device; A product image recognition device comprising:

9. A processor; A memory that stores a comparison dictionary that associates the appearance of each of a plurality of products to be registered with a code value represented by a code symbol placed on the appearance of each product; A computer comprising: A receiving unit that receives an image of the product to be registered, the image being transmitted from a higher-level device; an acquisition unit that acquires the code value of the photographed product by comparing an appearance image of the product included in the image with the comparison dictionary; a transmission unit that transmits the code value to the higher-level device; A product image recognition program to function as a.

Images