Product information registration system, information processing device, and program

Abstract

To prevent the same commodity from being registered redundantly even when code recognition and image recognition are used simultaneously.SOLUTION: A commodity information registration system includes a decoding unit, a recognition unit, a selection unit, and a registration unit. The decoding unit decodes a code symbol from an image of the code symbol included in an image of a commodity to be registered, to obtain a code value represented by the code symbol. The recognition unit recognizes the appearances of the commodity from an appearance image of the commodity included in the image of the commodity to be registered, to obtain a code value. The selection unit selects, based on a rule specified in advance, one of the code value acquired by the decoding unit and the code value acquired by the recognition unit. The registration unit registers information on a commodity identified by a commodity code corresponding to the code value, based on the code value selected by the selection unit.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] Embodiments of the present invention relate to a product information registration system, an information processing apparatus, and a program.

Background Art

[0002] In stores that sell products and services (hereinafter also collectively referred to as "products"), a POS (Point Of Sales) system is used. This type of POS system can register information on the product that a customer intends to purchase by reading a code symbol attached to the product (hereinafter also referred to as a "product barcode") with a scanner. Product barcodes include one-dimensional codes and two-dimensional codes. In one example, the product barcode is printed on a sticker and affixed to the product. In another example, the product barcode is printed on the package that houses the product.

[0003] An operator who operates the POS system needs to search for the product barcode from the product and hold it in front of the scanner, so the operation of reading the product barcode is troublesome. As a result, the work of registering product information (hereinafter also referred to as "product registration") by the operator requires time and effort. In particular, in a so-called self-POS system, since a customer who is not used to product registration needs to scan the product barcode, it is desirable to improve the operability of reading the product code, that is, the operability of the operator for identifying the product.

[0004] Therefore, it has been considered to use a method of registering a product by recognizing an image of the product using an image captured by a camera.

[0005] When realizing both the code recognition for reading a product barcode and such image recognition functions in one device (system), if the output codes by the respective functions are different, the same product may be registered repeatedly.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2014-021915 [Overview of the project] [Problems that the invention aims to solve]

[0007] The problem that this invention aims to solve is to provide a product information registration system, information processing device, and program that can prevent duplicate registration of the same product even when code recognition and image recognition are used in combination. [Means for solving the problem]

[0008] According to the embodiment, the product information registration system includes a decoding unit and A dictionary for comparison, The system comprises a recognition unit, a selection unit, and a registration unit. The decoding unit obtains the code value represented by the code symbol by decoding the code symbol from the image of the code symbol contained in the image of the product to be registered. The matching dictionary associates the visual features and code values ​​for each of the multiple products to be registered. The recognition unit recognizes the appearance of the product from the image of the product's appearance included in the photograph of the product to be registered. By comparing the visual features of the product with a matching dictionary... , The features of the product's appearance are associated with The code value is obtained. The selection part is predetermined. choice Based on the rules, select either the code value obtained by the decoding unit or the code value obtained by the recognition unit. The selection rule includes a rule for selecting the code value obtained from among the code values ​​obtained by the decoding unit and the code values ​​obtained by the recognition unit that yields a more complete result. The registration unit registers information about the product identified by the product code corresponding to the code value selected by the selection unit. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows the overall configuration of a product information management system, including a product information registration system, according to the first embodiment. [Figure 2] Figure 2 is a perspective view showing the appearance of a self-service POS system as an example of a product information registration system according to the first embodiment. [Figure 3]Figure 3 is a block diagram showing the hardware configuration of a POS terminal in a product information registration system according to the first embodiment. [Figure 4] Figure 4 is a block diagram showing the hardware configuration of a scanner as an example of an information processing device according to the first embodiment in the product information registration system according to the first embodiment. [Figure 5] Figure 5 is a block diagram showing the hardware configuration of the image recognition device in the product information registration system according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing the hardware configuration of the server included in the product information management system. [Figure 7] Figure 7 shows the data structure of the product master stored in the memory unit of the POS terminal shown in Figure 3. [Figure 8] Figure 8 shows the data structure of the mark setting table and selection rule setting table stored in the memory unit of the scanner shown in Figure 4. [Figure 9] Figure 9 shows the data structure of the matching dictionary stored in the memory unit of the image recognition device shown in Figure 5. [Figure 10] Figure 10 shows the data structure of the mark setting table stored in the memory unit of the image recognition device shown in Figure 5. [Figure 11] Figure 11 is a block diagram showing the functional configuration of the control unit of the POS terminal shown in Figure 3. [Figure 12] Figure 12 is a block diagram showing the functional configuration of the control unit of the scanner shown in Figure 4. [Figure 13] Figure 13 is a block diagram showing the functional configuration of the control unit of the image recognition device shown in Figure 5. [Figure 14] Figure 14 is a block diagram showing the functional configuration of the control unit of the server in Figure 6. [Figure 15] Figure 15 is a sequence diagram showing an overview of the operation of the product information management system. [Figure 16] Figure 16 is a flowchart showing the operation flow when registering one product using the scanner and image recognition device of the embodiment. [Figure 17]FIG. 17 is a block diagram showing the hardware configuration of the product information registration system according to the second embodiment. [Figure 18] FIG. 18 is a block diagram showing the functional configuration of the control unit of the POS terminal in FIG. 17.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described with reference to the drawings. Note that the present invention is not limited by the embodiments described below.

[0011] [First Embodiment] (Product Information Management System) First, referring to FIG. 1, the product information management system MS in the first embodiment will be described. FIG. 1 is a diagram showing the overall configuration of the product information management system MS in the first embodiment. The product information management system MS includes a product information registration system RS according to the first embodiment and a server 40.

[0012] The product information registration system RS includes a POS terminal 10, a scanner 20 as an example of an information processing device according to the first embodiment, and an image recognition device 30. The POS terminal 10 and the scanner 20 are communicably connected via a LAN (Local Area Network) or a USB (Universal Serial Bus) cable. The scanner 20 and the image recognition device 30 are communicably connected via a LAN. In the embodiment, Ethernet (registered trademark) and TCP (Transmission Control Protocol) are used together for LAN communication, but the communication protocol is not limited. Also, the LAN is not limited to a wired LAN and may be a wireless LAN, or a combination of wired and wireless may be used.

[0013] The POS terminal 10 and the image recognition device 30 are connected to a LAN hub LH via LAN so as to be able to communicate. The LAN hub LH is connected to a router RT via LAN so as to be able to communicate. The router RT can communicate with the server 40 via the network NW. The router RT may be connected to the network NW via a wired connection and communicate via wired communication, or it may be connected to the network NW via wireless communication and communicate via wireless communication.

[0014] Although Figure 1 shows only one product information registration system RS per LAN hub LH, there may be multiple product information registration systems RS. Similarly, although only one LAN hub LH per router RT is shown, there may be multiple LAN hub LHs. Furthermore, there may be multiple router RTs. For example, one or more LAN hub LHs under one router RT and one or more product information registration systems RS under each LAN hub LH can be deployed in one store. Server 40 can be, for example, a management server located in the headquarters that manages and oversees the operations of multiple stores in a company that operates multiple stores.

[0015] The scanner 20 captures an image of the product, obtaining an image of the product's appearance (hereinafter referred to as the product image). The scanner 20 transmits the acquired product image to the image recognition device 30 via LAN. The scanner 20 also detects and decodes the product barcode (1D code and 2D code), which is a code symbol contained in the acquired product image. In other words, the scanner 20 reads the information held by the product barcode. The decoding result of the product barcode includes a code value, which is a barcoded numerical value, and a code mark that indicates the code type of the barcode. The code type is the type of barcode, such as an EAN code or a JAN code. The code value is a numerical value that uniquely identifies the product to which the product barcode is attached. For example, in the case of a JAN code, the code value consists of a 13-digit number including a company code (9 or 7 digits) indicating the manufacturer of the product, a product item code (3 or 5 digits) that identifies the product, and a check digit (1 digit). The stores handle products such as prepared foods and fresh produce that do not have product barcodes provided by the manufacturer. For such products, the stores assign a store-specific in-store code to the product item code, create a store-specific product barcode by combining it with the store's business code, and attach it to these products. In addition, some stores introduce private-label products, and these products also use store-specific product barcodes in the same way.

[0016] The image recognition device 30 receives product images from the scanner 20 via LAN. The image recognition device 30 has an image recognition function, for example, using a pre-trained model created by machine learning, and performs image recognition on the product images using this function. Specifically, the image recognition device 30 acquires feature quantities contained in the product image using its image recognition function, compares the acquired feature quantities with a matching dictionary that associates multiple feature quantities with the code values ​​of the product barcodes attached to the appearance of the product contained in the product image, and acquires the code value associated with the feature quantity that is closest to the acquired feature quantity. The matching dictionary contains the code values ​​of the product barcodes for each product as the correct image recognition data based on the product's feature quantities. In other words, the matching dictionary defines the same information as the product barcode of the product as the correct image recognition data. Therefore, the image recognition device 30 uses such a matching dictionary to acquire code values ​​corresponding to the appearance of the product, such as the packaging. The image recognition device 30 determines the code type of the acquired code value and converts it into a code mark indicating that code type. The image recognition device 30 then uses these code marks and code values ​​as the final image recognition result. The image recognition device 30 may also perform image recognition on product images using methods such as deep learning. The image recognition device 30 transmits the image recognition results to the scanner 20 via LAN.

[0017] The scanner 20 receives image recognition results from the image recognition device 30 via the LAN. Based on pre-configured rules, the scanner 20 selects either the decoded result or the image recognition result and creates an output code containing the selected result. The rules can be set, for example, from the server 40 to the scanner 20 via the POS terminal 10. The scanner 20 sends the created output code to the POS terminal 10 via the LAN or a USB cable.

[0018] The POS terminal 10 is equipped with a product master that registers output codes, product names, and prices corresponding to product codes. A product code is an example of product identification information that uniquely identifies each product. The output code in the product master includes information corresponding to the decoding result and information corresponding to the recognition result. Therefore, the POS terminal 10 identifies the record of the corresponding product code from the product master based on the output code received from the scanner 20, and registers the purchased product based on the product name and price registered in that record.

[0019] Furthermore, the image recognition device 30 determines whether to use the product images received from the scanner 20 for additional training. For example, if the difference between an acquired feature and the feature closest to the acquired feature is greater than or equal to a predetermined amount, the image recognition device 30 determines that the product images received from the scanner 20 are product images to be used for additional training. The image recognition device 30 transmits the product images determined to be used for additional training, along with the decoded code values ​​and code marks received from the scanner 20, to the server 40 via the LAN.

[0020] Server 40 receives product images and decoding results from the image recognition device 30 via LAN. Server 40 stores the received product images and decoding results in association with each other. Server 40 uses the stored product images and decoding results to perform additional training on the pre-trained model for product image recognition. Server 40 updates the pre-trained model through this additional training. In addition, Server 40 updates the matching dictionary in conjunction with the additional training. Updating the matching dictionary includes modifying existing records and adding new records. Server 40 transmits the updated pre-trained model and the updated matching dictionary to the image recognition device 30 via LAN.

[0021] The image recognition device 30 receives the updated trained model and the updated matching dictionary from the server 40 via LAN. The image recognition device 30 updates its currently held trained model with the received updated trained model. The image recognition device 30 also updates its currently held matching dictionary with the received updated matching dictionary.

[0022] In this product information management system (MS), when a product's packaging changes, the effort required to re-collect product images for additional training of the pre-trained model used for product image recognition is eliminated.

[0023] Furthermore, Server 40 manages the transaction database and product master. POS terminals 10 transmit accounting information (transaction information related to transactions that have been accounted for) to router RT via LAN hub LH. Router RT transmits the received accounting information to Server 40 via network NW. Server 40 updates the transaction database based on the received accounting information. For example, Server 40 receives accounting information from multiple POS terminals 10 and records transaction information related to each transaction in the transaction database, such as when, which product, for how much, and how many units were sold for each of the multiple POS terminals 10. Server 40 also manages the product master held by each POS terminal 10. Product master management includes modifying existing records and adding new records. Server 40 also transmits the product master appropriate for each POS terminal 10 via network NW, router RT, and LAN hub LH. Note that this management of the transaction database and product master may be performed by a store server (not shown) located in each store, rather than by Server 40.

[0024] The image recognition device 30 may also transmit the product images received from the scanner 20 to the server 40, regardless of whether or not they are to be used for additional training. In this case, the product images stored in the server 40 may be used, for example, to analyze the state (events) of image recognition by the image recognition device 30.

[0025] (Product Information Registration System) The following describes a self-service POS system, operated by the customer purchasing the product, as an example of a product information registration system. However, product information registration systems are not limited to self-service POS systems. A product information registration system may also be a POS system operated by the store staff selling the product.

[0026] Figure 2 is a perspective view showing the external appearance of the self-service POS system 1. The self-service POS system 1 includes a POS terminal 10, a scanner 20 built into the POS terminal 10, and an image recognition device 30 built into the POS terminal 10. Here, the self-service POS system 1 is an example of a product information registration system, the POS terminal 10 is an example of a registration device, and the scanner 20 is an example of an information processing device according to the first embodiment.

[0027] The scanner 20 decodes the barcode attached to the product purchased by the customer, obtains the coded code value, determines the barcode type from this code value, and obtains a decoded result that includes the code value and a code mark indicating the barcode type. The scanner can send this decoded result as an output code to the POS terminal 10. Based on the output code input from the scanner 20, the POS terminal 10 identifies the product code of the purchased product from the product master, performs product registration, and also performs accounting processing related to the registered product. Accounting processing is the process that the POS terminal 10 performs in order for the customer to pay for the purchased product.

[0028] As shown in Figure 2, the POS terminal 10 comprises a main unit 50, a first basket stand 70, and a second basket stand 80. The main unit 50 has a display 51 and a touch panel 52 on its top, and houses a printer 53, a card reader 54, and a change dispenser 55 (see Figure 3 for each) inside.

[0029] Display 51 is a display device such as an LCD (Liquid Crystal Display) that displays various information. For example, display 51 displays product information such as the product name and price of products registered by the customer during the customer registration operation. Display 51 also displays accounting information such as the total transaction amount and change amount in conjunction with the customer's accounting operation.

[0030] The touch panel 52 is located on the surface of the display 51 and inputs information corresponding to the touch location to the control unit 100 (see Figure 3) of the POS terminal 10. Customers using the POS terminal 10 can operate the touch panel 52 by touching various buttons and operation keys displayed on the display 51.

[0031] As shown in Figure 2, below the display 51, facing the direction of the customer, are the reading window 230 for the scanner 20, the receipt printing slot 531 for the printer 53, and the card insertion slot 541 for the card reader 54.

[0032] The reading window 230 is a window through which the scanner 20 captures an image of the product's appearance. The customer, who is the operator of the self-service POS system 1, performs the product registration operation by holding the product they wish to purchase over the reading window 230. The customer holds the product over the reading window 230, changing its orientation, until the self-service POS system 1 completes the product registration and a registration completion sound is emitted.

[0033] The issuing slot 531 is an opening for ejecting receipts printed by the printer 53. The receipts have information (accounting information) related to the transactions processed by the POS terminal 10 printed on them. The POS terminal 10 may also be configured to issue electronic receipts instead of paper receipts printed by the printer 53.

[0034] The card slot 541 is an opening for inserting credit cards or other cards to be read by the card reader 54. When a customer makes a payment with a credit card, or when a customer has a store membership card or similar card read, a card is inserted through the card slot 541.

[0035] Furthermore, the main body 50 has a coin slot 551, a coin outlet 552, a coin tray 553, a banknote slot 554, and a banknote outlet 555, which correspond to the position of the change machine 55 housed inside.

[0036] The coin slot 551 is an opening into which customers insert coins to pay. Coins inserted into the coin slot 551 are stored in the change machine 55. The coin outlet 552 is an opening that discharges coins dispensed from the change machine 55. The coin outlet 552 discharges change or coins that were inserted into the coin slot 551 but not accepted by the change machine 55. The coin tray 553 is a tray that receives coins discharged from the coin outlet 552. The banknote slot 554 is an opening into which customers insert banknotes to pay. Banknotes inserted into the banknote slot 554 are stored in the change machine 55. The banknote outlet 555 is an opening that discharges banknotes, which are the change dispensed from the change machine 55.

[0037] The first basket rest 70 is a rest on which a shopping basket (not shown) containing the goods that a customer intends to purchase is placed. The first basket rest 70 is equipped with a first weighing unit 71. The first weighing unit 71 weighs the total weight of the shopping basket placed on the first basket rest 70 and the goods contained in the shopping basket.

[0038] The second basket placement platform 80 is a platform on which a shopping basket (not shown) containing products that have been removed from a shopping basket placed on the first basket placement platform 70 and whose product barcodes have been read, i.e., registered products, is placed. The second basket placement platform 80 comprises a second weighing unit 81, a pair of support rods 82, and a temporary placement platform 83.

[0039] The second weighing unit 81 weighs the total weight of a shopping basket placed on the second basket placement platform 80 and the goods contained in the shopping basket, or the total weight of a shopping bag (not shown) supported by a pair of support rods 82 and the goods contained in the shopping bag. The support rods 82 are rods for hooking and supporting a shopping bag that contains registered goods. The support rods 82 make it easier to put goods into the shopping bag by supporting the shopping bag in an open state. The temporary placement platform 83 is a platform for temporarily placing easily damaged or deformable goods among the registered goods, such as tofu and eggs. The customer can then place the goods placed on the temporary placement platform 83 into the shopping basket or shopping bag at the end.

[0040] The customer registers the items by holding the items they have taken out of the shopping basket placed on the first basket stand 70 up to the reading window 230. The customer then places the registered items into a shopping basket placed on the second basket stand 80 or into a shopping bag supported by the support rod 82.

[0041] The first weighing unit 71 measures the weight before and after removing items from a shopping basket placed on the first basket stand 70, allowing the control unit 100 (see Figure 3) to recognize the weight change. The second weighing unit 81 measures the weight before and after placing items in a shopping basket placed on the second basket stand 80 or in a shopping bag supported by a support rod 82, allowing the control unit 100 to recognize the weight change. By determining whether these two weight changes are equal, the control unit 100 can determine whether all the items the customer intends to purchase have been registered.

[0042] Furthermore, the main body 50 is equipped with a cylindrical pole 57 that extends upward. The upper tip of the pole 57 is provided with a light-emitting part 571 that illuminates when calling a store employee.

[0043] Next, the hardware configurations of the POS terminal 10, scanner 20, image recognition device 30, and server 40 will be described with reference to Figures 3 to 6. Figure 3 is a block diagram showing the hardware configuration of the POS terminal 10, Figure 4 is a block diagram showing the hardware configuration of the scanner 20, Figure 5 is a block diagram showing the hardware configuration of the image recognition device 30, and Figure 6 is a block diagram showing the hardware configuration of the server 40.

[0044] (POS terminal hardware configuration) As shown in Figure 3, the POS terminal 10 includes a control unit 100, a storage unit 110, a display 51, a touch panel 52, a printer 53, a card reader 54, a change dispenser 55, a first weighing unit 71, a second weighing unit 81, a light-emitting unit 571, an audio output unit 120, a connection interface 121, and a LAN interface 122. These components—the control unit 100, the storage unit 110, the display 51, the touch panel 52, the printer 53, the card reader 54, the change dispenser 55, the first weighing unit 71, the second weighing unit 81, the light-emitting unit 571, the audio output unit 120, the connection interface 121, and the LAN interface 122—are connected to each other via a bus 123 or the like.

[0045] The control unit 100 consists of a computer equipped with a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103. The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 123.

[0046] The CPU 101 controls the entire POS terminal 10. The CPU 101 is an example of a processor, and an MPU (Micro Processing Unit) may be used instead. A multi-core / multi-threaded processor can execute multiple processes in parallel. Furthermore, the processor may be implemented in various other forms, including integrated circuits such as ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), DSP (Digital Signal Processor), SoC (System on a Chip), and PLD (Programmable Logic Device), or a combination of several of these.

[0047] ROM 102 stores various programs and data, such as programs used to drive the CPU 101. RAM 103 is used as the work area for the CPU 101 and loads the various programs and data stored in ROM 102 and the memory unit 110. The control unit 100 executes various control processes of the POS terminal 10, for example, by having the CPU 101 operate according to the control program 111 stored in the memory unit 110 and loaded into RAM 103.

[0048] The RAM 103 has a product information unit 1031. The product information unit 1031 stores product information read from the product master 112 (described later) based on the output code obtained from the scanner 20. Product registration is performed when product information is stored (registered) in the product information unit 1031. The product information stored in the product information unit 1031 includes the product code, product name, and price. The product information unit 1031 stores product information related to one transaction and is cleared when the accounting processing for that transaction is completed.

[0049] The storage unit 110 is composed of a storage medium such as an HDD (Hard Disk Drive) or flash memory, and retains its contents even when the power is cut off. The storage unit 110 stores the control program 111 and the product master 112.

[0050] The control program 111 is a program that implements functions such as receiving output codes from the scanner 20, performing product registration based on the received output codes, performing accounting processing for registered products, and changing the settings of the scanner 20 and the image recognition device 30. The control program 111 may be stored in the ROM 102 instead of being stored in the storage unit 110.

[0051] The product master 112 is a master file that stores product information for the products handled by the store. Since the products handled by the store change daily, the product master 112 is updated as needed by the server 40 or store server (not shown) connected via the LAN interface 122. Figure 7 shows the data structure of the product master 112. The product master 112 includes the product code, the output code associated with the product code, the product category, the product name, and the price.

[0052] The product code is information that identifies a product. Product codes can be set independently by each store, for example. The product category is information that classifies the products sold in the store. The product name is information that indicates the name of the product. The price is information that indicates the price (unit price) of the product. In addition, the product master 112 may also include other information associated with the product code.

[0053] Returning to Figure 3, we will continue the explanation of the hardware configuration of the POS terminal 10. The configuration and functions of the display 51 and touch panel 52 are as described above.

[0054] The printer 53 is configured, for example, as a thermal printer. The printer 53 prints transaction information on roll-shaped receipt paper and issues it as a receipt from the issuing slot 531.

[0055] The card reader 54 reads the credit information necessary for payment from the credit card inserted through the card slot 541. The card reader 54 also reads member information that identifies the member from a membership card or point card or other medium inserted through the card slot 541. The card reader 54 may read information magnetically from the card, or it may be an IC card reader that reads information from an IC card with a built-in IC (Integrated Circuit) chip.

[0056] The change dispenser 55 accepts the banknotes and coins paid by the customer when they pay for the goods in cash, and dispenses change as needed. Banknotes are inserted through the banknote slot 554, and coins are inserted through the coin slot 551. The banknotes of the change are discharged through the banknote discharge slot 555, and the coins of the change are discharged through the coin discharge slot 552.

[0057] The configuration and functions of the first weighing unit 71, the second weighing unit 81, and the light-emitting unit 571 are as described above. The voice output unit 120 generates voice as needed to notify the customer of information. For example, the voice output unit 120 outputs a notification sound to indicate that product registration is complete. The voice output unit is an example of a notification unit.

[0058] The connection interface 121 is an interface for communicating with the scanner 20 via LAN or USB cable. The control unit 100 sends and receives information (data) to and from the scanner 20 via the connection interface 121, via LAN or USB cable. For example, the control unit 100 receives output codes from the scanner 20 via the connection interface 121, via LAN or USB cable.

[0059] The LAN interface 122 is an interface for communicating with the LAN hub LH via the LAN. The control unit 100 sends and receives information (data) to and from the LAN hub LH via the LAN interface 122. For example, the control unit 100 sends and receives information between the server 40 and the product master 112 via the LAN through the LAN hub LH and router RT.

[0060] The LAN interface 122 also serves as an interface for communicating with higher-level devices such as a store server (not shown) managed by the store. The control unit 100 sends and receives information with the higher-level devices via the LAN interface 122.

[0061] (Scanner hardware configuration) Scanner 20 captures images of products purchased by customers to acquire product images. Subsequently, scanner 20 detects product barcodes included in the product images, i.e., barcodes attached to the captured products, and decodes the detected product barcodes. The decoded product barcode results include the code value held by the product barcode and a code mark indicating the code type determined from this code value. Scanner 20 also transmits the product images to image recognition devices 30 via LAN. Furthermore, scanner 20 can transmit the decoded results to image recognition devices 30 via LAN.

[0062] As shown in Figure 4, the scanner 20 comprises a control unit 200, a storage unit 210, an imaging unit 220, a connection interface 221, and a LAN interface 222. These control unit 200, storage unit 210, imaging unit 220, connection interface 221, and LAN interface 222 are connected to each other via a bus 223, etc.

[0063] The control unit 200 consists of a computer equipped with a CPU 201, ROM 202, and RAM 203. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 223.

[0064] The CPU 201 controls the entire scanner 20. A processor other than the CPU 201 may also be used. The ROM 202 stores various programs and data, such as the program used to drive the CPU 201. The RAM 203 is used as the work area for the CPU 201 and loads the various programs and data stored in the ROM 202 and the storage unit 210. The control unit 200 executes various control processes for the scanner 20, for example, by having the CPU 201 operate according to the control program 211 stored in the storage unit 210 and loaded into the RAM 203.

[0065] The storage unit 210 is composed of a storage medium such as an HDD or flash memory, and retains its contents even when the power is cut off. The storage unit 210 stores the control program 211, the setting table 212, and the processing results 213.

[0066] The control program 211 is a program that implements functions such as capturing images of products with the imaging unit 220 and saving them to the RAM 203, acquiring frame images (product images) held by the RAM 203, detecting product barcodes included in the product images, decoding product barcodes, transmitting product images to the image recognition device 30, receiving image recognition results from the image recognition device 30, selecting the decoding results and image recognition results to create an output code, and transmitting the output code to the POS terminal 10. The control program 211 may be stored in the ROM 202 instead of the storage unit 210.

[0067] The configuration table 212 includes data tables that manage various settings for the scanner 20, which are configured from the POS terminal 10 via the LAN. For example, data tables included in the configuration table 212 include the mark configuration table 2121 and the selection rule configuration table 2122, as shown in Figure 8.

[0068] The mark setting table 2121 is a data table that manages code marks for each code type expected to be used as a product barcode. For example, if the code type is "JAN-13", then "F" is set as the code mark.

[0069] Furthermore, the selection rule setting table 2122 is a data table that manages selection rules when both the decoded result and the image recognition result are obtained as a result of including them in the output code sent from the scanner 20 to the POS terminal 10. Selection rules can be set based on, for example, type, completeness, number of acquisitions, and confidence level, and at least one of these rules is pre-configured.

[0070] The type-based rule is simply a rule that selects based on the type of result, such as a decoded result or an image recognition result. For example, this type-based rule allows setting priorities for decoded results and image recognition results, with the scanner 20 selecting the one with the higher priority. Since the decoded result, which is the result of reading product barcodes, is more accurate than the image recognition result, it is common to set the priority of the decoded result higher than that of the image recognition result. On the other hand, in stores that handle many products without product barcodes, it is often difficult to obtain decoded results, so the priority of the image recognition result is set higher than that of the decoded result.

[0071] The completeness-based rule is to select the result that provides a more complete result from the decoding result and the image recognition result. For example, in the case of a "JAN-13" barcode, a 2-digit or 5-digit add-on code representing price, etc., may be added. If results with and without this add-on code can be obtained, the scanner 20 will select the result that includes the add-on code, which is the more complete result. Also, in the case of multi-layer barcodes, a complete decoding result cannot be obtained unless all barcodes are decoded. If only a partial decoding result is obtained, the scanner 20 will select the image recognition result.

[0072] The rule based on the number of acquisitions is to select the one with the larger number of acquisitions between the decoded result and the image recognition result. Instead of immediately sending a result to the POS terminal 10 when one result is obtained, the scanner 20 waits for a certain period of time, allowing it to acquire multiple results of each type. The scanner 20 then selects the one with the larger number of acquisitions as the more likely result.

[0073] The confidence-based rule is to select the one with the higher confidence level between the decoding result and the image recognition result. For example, the reading accuracy of the decoding result changes depending on the reading conditions, such as the angle at which the product barcode is held over the reading window 230 and the lighting conditions. Based on these reading conditions, the confidence level of the decoding result can be calculated as 50% or 80%. The image recognition device 30 also obtains a code value associated with the feature closest to the acquired feature by comparing the acquired feature of the product's appearance obtained from the product image with a matching dictionary that stores multiple feature quantities. Therefore, in the image recognition result, a confidence level of 50% or 80% can be calculated based on the difference between this acquired feature and the feature in the matching dictionary that is closest to this acquired feature. Based on the confidence levels of the decoding result and the image recognition result, the scanner 20 can select the result with the higher confidence level.

[0074] The processing result 213 includes the decoding result obtained by decoding the product barcode, the image recognition result by the image recognition device 30, and the output code created from the decoding result and the image recognition result.

[0075] The imaging unit 220 is composed of, for example, a color CCD image sensor or a color CMOS image sensor. The imaging unit 220, in accordance with the control of the control unit 200, captures images of products through the reading window 230 and outputs the acquired product images. For example, the imaging unit 220 captures moving images at a predetermined frame rate (for example, 30fps). The imaging unit 220 sequentially stores the captured frame images in the RAM 203. The frame images (product images) held in the RAM 203 are used to identify the code value of the product shown in the frame image.

[0076] The connection interface 221 is an interface for communicating with the POS terminal 10 via LAN or USB cable. The control unit 200 sends and receives information (data) to and from the POS terminal 10 via the connection interface 221, either via LAN or USB cable.

[0077] The LAN interface 222 is an interface for communicating with the image recognition device 30 via the LAN. The control unit 200 sends and receives information (data) to and from the image recognition device 30 via the LAN interface 222.

[0078] The control unit 200 detects the product barcode contained in the frame image (product image) held by the RAM 203, decodes the product barcode, and determines the code type based on the number of digits in the code value indicated by the product barcode. The control unit 200 identifies the code mark corresponding to the determined code type using the mark setting table 2121, and obtains the code value and the code mark representing the code type as the decoded result. The control unit 200 also transmits the frame image (product image) held by the RAM 203 and the acquired decoded result to the image recognition device 30 via the LAN interface 222, and receives the image recognition result from the image recognition device 30. Furthermore, the control unit 200 creates an output code from the decoded result and the image recognition result. Specifically, the control unit 200 selects either the decoded result or the image recognition result using the selection rule setting table 2122, and creates an output code according to the selected result. The control unit 200 transmits the created output code to the POS terminal 10 via the connection interface 221.

[0079] (Image recognition device hardware configuration) The image recognition device 30 receives product images from the scanner 20 via the LAN. The image recognition device 30 performs image recognition on the received product images. Specifically, based on the features of the product images, the image recognition device 30 obtains the code type and code value, which correspond to the code value held by the product barcode of the product shown in the product image. The image recognition device 30 obtains the code value with a code mark indicating this code type added as the image recognition result. The image recognition device 30 transmits this image recognition result to the scanner 20 via the LAN. The image recognition device 30 also receives the decoded result from the scanner 20 via the LAN and, if necessary, transmits the product image and the decoded result to the server 40 via the LAN. Furthermore, the image recognition device 30 receives the updated trained model and matching dictionary from the server 40 via the LAN and updates the currently held trained model and matching dictionary.

[0080] As shown in Figure 5, the image recognition device 30 comprises a control unit 300, a storage unit 310, and a LAN interface 321. These control unit 300, storage unit 310, and LAN interface 321 are connected to each other via a bus 322, etc.

[0081] The control unit 300 consists of a computer equipped with a CPU 301, ROM 302, and RAM 303. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 322.

[0082] The CPU 301 controls the entire image recognition device 30. A processor other than the CPU 301 may also be used. The ROM 302 stores various programs and data, such as the program used to drive the CPU 301. The RAM 303 is used as the work area for the CPU 301 and loads the various programs and data stored in the ROM 302 and the storage unit 310. The control unit 300 executes various control processes for the image recognition device 30, for example, by having the CPU 301 operate according to the control program 311 stored in the storage unit 310 and loaded into the RAM 303.

[0083] The storage unit 310 is composed of a storage medium such as an HDD or flash memory, and retains its contents even when the power is cut off. The storage unit 310 stores the control program 311, the learned model 312, the comparison dictionary 313, and the setting table 314.

[0084] The control program 311 is a program that implements functions such as receiving product images from the scanner 20, performing image recognition on the product images, determining whether the image recognition was successful, sending the image recognition results to the scanner 20, receiving the decoded results from the scanner 20, determining whether to use the product images for additional training, sending the product images and decoded results to the server 40, receiving the updated trained model and updated matching dictionary from the server 40, updating the currently held trained model with the updated trained model, and updating the currently held matching dictionary with the updated matching dictionary. The control program 311 may be stored in the ROM 302 instead of the storage unit 310.

[0085] The pre-trained model 312 is an image recognition model for performing image recognition on product images. The pre-trained model 312 has been trained using machine learning and can be further trained using machine learning.

[0086] The matching dictionary 313 is a data table that manages the correspondence between products and the characteristic features of their appearance. Figure 9 shows the data structure of the matching dictionary 313. The matching dictionary 313 stores the characteristic features of the appearance of each of the multiple products to be registered, and the code value represented by the product barcode placed on the appearance of each product, in correspondence. The matching dictionary 313 may have multiple characteristic features for a single code value. The code values ​​in the matching dictionary 313 are the same as the code values ​​included in the output code of the product master 112 held by the storage unit 110 of the POS terminal 10. Thus, the storage unit 310 functions as a memory that stores the matching dictionary 313. Note that the matching dictionary 313 may store not just the code value, but the code type and the code value in correspondence with the characteristic. That is, the matching dictionary 313 may include the code value in the product code of each product and the code type of that code value as correct data for image recognition based on the characteristic features of the product.

[0087] Feature quantities are parameterized representations of the product's appearance, such as its standard shape, surface color, pattern, and texture. In addition to appearance, feature quantities may also include parameterized representations of the shape of the sticker label on which the product barcode is printed, and the strings or numbers printed on it. The control unit 300 acquires the feature quantities contained in the product image, compares the acquired feature quantities with a matching dictionary, and obtains the code value associated with the acquired feature quantities. In other words, based on the feature quantities acquired from the product image, the control unit 300 obtains the code value of the product shown in the image from among the products pre-recorded in the matching dictionary 313. Furthermore, if the matching dictionary 313 stores the code type and code value associated with the feature quantities, the control unit 300 can obtain the code type and code value of the product shown in the product image.

[0088] The configuration table 314 includes a data table that manages various settings for the image recognition device 30, which are set from the POS terminal 10 via the LAN through the LAN hub LH or via the LAN through the scanner 20. For example, one of the data tables included in the configuration table 314 is the mark configuration table 3141 shown in Figure 10. The mark configuration table 3141 is similar to the mark configuration table 2121 of the scanner 20. This is a data table that manages code marks for each code type assumed to be a product barcode. When configured via the LAN through the scanner 20, the mark configuration table 3141 may be a copy of the mark configuration table 2121.

[0089] Returning to Figure 5, we will continue the explanation of the hardware configuration of the image recognition device 30. The LAN interface 321 is an interface for communicating with the scanner 20 and the LAN hub LH. The control unit 300 sends and receives information (data) with the scanner 20 and the LAN hub LH via the LAN interface 321.

[0090] For example, the control unit 300 receives a frame image (product image) from the scanner 20 via the LAN interface 321. The control unit 300 also obtains the code value of the product shown in the product image using the trained model 312 and the matching dictionary 313 as described above, and determines the code type in the same way as the control unit 200 of the scanner 20. If the matching dictionary 313 stores the code type and code value associated with features, the control unit 300 can obtain the code type and code value from the product image. The control unit 300 identifies the code mark of the code type using the mark setting table 3141. The control unit 300 then obtains the image recognition result by adding this identified code mark to the acquired code value. The control unit 300 transmits this image recognition result, consisting of the code mark and code value, to the scanner 20 via the LAN interface 321.

[0091] Furthermore, the control unit 300 transmits the product image that has undergone image recognition and the code value of the decoded result received from the scanner 20 to the server 40 via the LAN interface 321 as needed. The control unit 300 may also transmit the code mark of the decoded result. The control unit 300 also receives the updated trained model 412 (see Figure 6) and the updated matching dictionary 413 (see Figure 6) from the server 40 via the LAN interface 321. The control unit 300 updates the currently held trained model 312 with the updated trained model 412. The control unit 300 also updates the currently held matching dictionary 313 with the updated matching dictionary 413.

[0092] (Server hardware configuration) Server 40 receives and stores product images and code values ​​(and code marks) from the image recognition device 30. Using the stored product images and code values ​​(and code marks), Server 40 performs additional training on the trained model for product image recognition and updates the trained model and the matching dictionary. Server 40 sends the updated trained model and the updated matching dictionary to the image recognition device 30. Server 40 also manages the product master. Server 40 receives accounting processing information from the POS terminals 10 and manages the product master based on the received accounting processing information. Server 40 sends the product master appropriate for each POS terminal 10 to each POS terminal 10. Server 40 also sends the selection rules to be set in the selection rule setting table 2122 of each scanner 20 to each POS terminal 10 connected to each scanner 20 via LAN. Each POS terminal 10 that receives these selection rules forwards them to each scanner 20, and each scanner 20 sets them in the selection rule setting table 2122. In addition, such selection rules may be set in the selection rule setting table 2122 of the scanner 20 by a store server (not shown) located in each store, rather than by the server 40.

[0093] As shown in Figure 6, the server 40 comprises a control unit 400, a storage unit 410, a communication unit 421, an input device 422, and an output device 423. These control unit 400, storage unit 410, communication unit 421, input device 422, and output device 423 are connected to each other via a bus 424 or the like.

[0094] The control unit 400 consists of a computer equipped with a CPU 401, ROM 402, and RAM 403. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 424.

[0095] The CPU 401 controls the entire server 40. It may be a processor other than the CPU 401. The ROM 402 stores various programs and data, such as the program used to drive the CPU 401. The RAM 403 is used as the work area for the CPU 401 and loads the various programs and data stored in the ROM 402 and the storage unit 410. The control unit 400 executes various control processes for the server 40, for example, by having the CPU 401 operate according to the control program 411 stored in the storage unit 410 and loaded into the RAM 403.

[0096] The storage unit 410 is composed of a storage medium such as an HDD or flash memory, and retains its contents even when the power is cut off. The storage unit 410 stores the control program 411, the learned model 412, the matching dictionary 413, the transaction database 414, and the product master 415.

[0097] The control program 411 is a program that implements functions such as receiving product images and decoding results from the image recognition device 30, storing product images and decoding results, further learning and updating the trained model 412, updating the matching dictionary 413, sending the updated trained model 412 and the updated matching dictionary 413 to the image recognition device 30, receiving accounting processing information from the POS terminal 10, managing the product master 415, sending the product master 415 to the POS terminal 10, and setting selection rules. The control program 411 may be stored in the ROM 402 instead of the storage unit 410.

[0098] The pre-trained model 412 is a machine learning model trained to perform image recognition on product images. More specifically, the pre-trained model 412 is an image recognition model that obtains the code values ​​of the products shown in the product images by acquiring features contained in the product images and comparing the acquired features with the matching dictionary 413.

[0099] The matching dictionary 413 is a data table that manages the correspondence between the code value of a product and the visual features of that product, similar to the matching dictionary 313 held by the image recognition device 30. The matching dictionary 413 may also be a data table that manages the correspondence between the code value and code mark of a product and the visual features of that product. The matching dictionary 413 includes the matching dictionaries 313 held by multiple image recognition devices 30. The server 40 transmits the matching dictionary 313 suitable for each image recognition device 30 to that device 30.

[0100] The transaction database 414 stores transaction information for each store, such as which POS terminal 10 was used, when, which product was sold, for how much, and how many units were sold. The server 40 manages the product master 415 based on the accounting processing information received from the POS terminal 10.

[0101] The product master 415, like the product master 112 held by the POS terminal 10, includes a product code, an output code associated with the product code, a product category, a product name, and a price. The product master 415 is the basis for the product masters 112 held by multiple POS terminals 10 and contains information about all products. The product master 112 of a POS terminal 10 contains information only about products sold in that store. In other words, the product master 112 of a POS terminal 10 is an excerpt of the product master 415 of the server 40. The server 40 transmits the product master 112 appropriate for each POS terminal 10 to that POS terminal 10.

[0102] The communication unit 421 is an interface for communicating with the router RT. The control unit 400 sends and receives information (data) with the router RT via the communication unit 421.

[0103] The input device 422 is a user interface such as a keyboard or mouse, and the output device 423 is a user interface such as a display. The administrator of the server 40 can use these input devices 422 and output devices 423 to input various instructions to the control unit 400 and view various display results from the control unit 400. The server 40 does not necessarily need to have the input device 422 and output device 423; instead, it can use the input and output devices of the administrator's computer connected via the network NW by the communication unit 421.

[0104] For example, the control unit 400 receives and stores product images and decoding results from the image recognition device 30 via the communication unit 421. The control unit 400 uses the stored product images and decoding results to further train the trained model 412 and updates the trained model 412 and the matching dictionary 413. The control unit 400 transmits the updated trained model and the updated matching dictionary to the image recognition device 30 via the communication unit 421.

[0105] Furthermore, the server 40 receives accounting processing information from the POS terminal 10 via the communication unit 421. Based on the received accounting processing information, the server 40 manages the transaction database 414. In addition, the server 40 transmits a product master appropriate for each POS terminal 10 to each POS terminal 10 via the communication unit 421.

[0106] Furthermore, the server 40 receives selection rules to be set in the selection rule setting table 2122 of each scanner 20 from the input device 422, and transmits the selection rules to each scanner 20 via each POS terminal 10 through the communication unit 421.

[0107] (POS terminal function configuration) Next, the functional configuration of the POS terminal 10 will be described with reference to Figure 11. Figure 11 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10.

[0108] The control unit 100 includes an input means 1001, a registration means 1002, an accounting processing means 1003, and an output means 1004. The control unit 100 functions as the input means 1001, registration means 1002, accounting processing means 1003, and output means 1004 when the CPU 101 operates according to a control program stored in the ROM 102 or the storage unit 110. These functions may also be configured using hardware such as dedicated circuits.

[0109] The input means 1001 receives various types of information from the scanner 20, touch panel 52, card reader 54, first weighing unit 71, second weighing unit 81, and so on. For example, the output code is input to the input means 1001 from the scanner 20.

[0110] Furthermore, information based on the operation of the touch panel 52 is input to the input means 1001. For example, transaction start information declaring the start of a transaction is input to the input means 1001. Also, registration completion information indicating that the product registration for the goods related to a transaction has been completed is input to the input means 1001.

[0111] The registration means 1002 performs product registration of the product that the customer intends to purchase based on the customer's operation. In other words, the registration means 1002 registers product information identified by the product identification information entered into the input means 1001. Specifically, the registration means 1002 reads product information from the product master 112 based on the output code entered into the input means 1001. The registration means 1002 then stores the read product information in the product information unit 1031. Product information includes information such as product code, product name, and price. Thus, the computer with the CPU 101 of the POS terminal 10 as its central component functions as a registration unit that registers information about the product identified by the product code corresponding to the code value, based on the code value selected from the code values ​​acquired by the decoding unit and recognition unit described later.

[0112] The accounting processing device 1003 performs accounting processing related to the payment of goods registered by the registration device 1002. Specifically, the accounting processing device 1003 performs processing such as calculating the total amount (transaction amount) of the registered goods, calculating the amount of change, displaying various information, and issuing a receipt. In addition, if the customer wishes to make a cashless payment such as credit card payment or QR code payment, the accounting processing device 1003 can perform the accounting processing necessary for the payment.

[0113] The output means 1004 outputs various information to the scanner 20, printer 53, change machine 55, light-emitting unit 571, audio output unit 120, etc. For example, when transaction start information is input to the input means 1001, the output means 1004 outputs an imaging start signal to instruct the scanner 20 to start imaging. Also, when registration completion information is input to the input means 1001, the output means 1004 outputs an imaging stop signal to instruct the scanner 20 to stop imaging.

[0114] The output means 1004 outputs a control signal to the audio output unit 120 to output a notification sound when the registration means 1002 completes the registration of a product for that product. The output means 1004 functions as a notification means to notify the notification unit (audio output unit 120) that the registration of product information by the registration means 1002 has been completed. It is also possible to provide the audio output unit 120 on the scanner 20. In this case, the audio output unit 120 may emit a notification sound when the scanner 20 identifies a code value.

[0115] Furthermore, once the accounting processing by the accounting processing means 1003 is completed, the output means 1004 transmits the accounting processing information to the server 40 via the LAN hub LH.

[0116] (Scanner Functional Configuration) Next, the functional configuration of the scanner 20 will be described with reference to Figure 12. Figure 12 is a block diagram showing the functional configuration of the control unit 200 of the scanner 20.

[0117] The control unit 200 includes an acquisition means 2001, a decoding means 2002, a transmitting and receiving means 2003, a selection means 2004, and a transmission means 2005. The control unit 200 functions as the acquisition means 2001, decoding means 2002, transmitting and receiving means 2003, selection means 2004, and transmission means 2005 when the CPU 201 operates according to a control program stored in the ROM 202 or the storage unit 210. Note that each of these functions may be configured with hardware such as dedicated circuits.

[0118] The acquisition means 2001 acquires a single frame image as a product image from the moving image captured by the imaging unit 220. The acquisition means 2001 stores the acquired product image in the RAM 203. For example, the acquisition means 2001 acquires all or some of the multiple frame images sequentially captured by the imaging unit 220 as candidate product images, stores all or some of the acquired frame images in the RAM 203, and selects a product image from all or some of the stored frame images. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as an image acquisition unit that photographs the product to be registered and acquires an image.

[0119] Furthermore, the acquisition means 2001 acquires various other information. For example, the acquisition means 2001 acquires an image capture start signal and an image capture stop signal from the POS terminal 10.

[0120] The decoding means 2002 detects the product barcode contained in the product image acquired by the acquisition means 2001 and extracts the product barcode from the product image. The decoding means 2002 decodes the extracted product barcode. As a result of decoding the product barcode, the decoding means 2002 obtains the code value held by the product barcode and a code mark representing the code type determined from that code value. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as a decoding unit that decodes the code symbol from the image of the code symbol contained in the image of the product to be registered, thereby obtaining the code value represented by the code symbol.

[0121] The transmitting / receiving means 2003 transmits the product image acquired by the acquisition means 2001 to the image recognition device 30. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as an imaging unit that photographs the product to be registered, outputs the captured image to the decoding unit, and also outputs it to the image recognition device 30. The image recognition device 30 performs image recognition on the received product image and transmits the image recognition result (the code value of the product shown in the product image and the code mark attached to it) to the scanner 20. The transmitting / receiving means 2003 receives the image recognition result transmitted by the image recognition device 30. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as a code value receiving unit that receives the code value output from the image recognition device 30. Alternatively, the computer with the CPU 201 of the scanner 20 can also be said to function as a transmitting / receiving unit that transmits the image acquired by the image acquisition unit to the image recognition device 30, which acquires the code value by recognizing the appearance of the product from the appearance image of the product contained in the image, and also receives the code value acquired by the image recognition device 30 output from the image recognition device 30. The transmitting / receiving means 2003 transmits the decoding result (code value and code mark held by the product barcode) from the decoding means 2002 to the image recognition device 30.

[0122] The selection means 2004 selects either the decoded result (the code value and code mark held by the product barcode) or the image recognition result (the code value of the product shown in the product image and the code mark attached to it), and creates an output code that includes the selected result. That is, the selection means 2004 selects either the decoded result or the image recognition result according to the selection rules set in the selection rule setting table 2122. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as a selection unit that selects either the code value acquired by the decoding unit or the code value acquired by the recognition unit described later, based on predetermined rules. The selection means 2004 then creates an output code that includes this selected decoded result or image recognition result. Thus, the computer with the CPU 201 of the scanner 20 as its central component functions as a creation unit that creates an output code to be output to the registration unit from the code value acquired by the decoding unit or the code value acquired by the recognition unit described later.

[0123] The selection means 2004 selects the image recognition result if a decoding result cannot be obtained. Situations in which a decoding result cannot be obtained include when the customer does not hold the product barcode correctly over the imaging unit 220 of the scanner 20, or when the product barcode is bent or folded in an unnatural way.

[0124] The transmission means 2005 transmits the output code created by the selection means 2004 to the POS terminal 10 via the connection interface 221.

[0125] (Functional configuration of the image recognition device) Next, the functional configuration of the image recognition device 30 will be described with reference to Figure 13. Figure 13 is a block diagram showing the functional configuration of the control unit 300 of the image recognition device 30.

[0126] The control unit 300 includes a receiving means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, and a transmission means 3005. The control unit 300 functions as the receiving means 3001, image recognition means 3002, determination means 3003, update means 3004, and transmission means 3005 when the CPU 301 operates according to a control program stored in the ROM 302 or the storage unit 310. Note that each of these functions may be configured with hardware such as dedicated circuits.

[0127] The receiving means 3001 receives product images from the scanner 20 via the LAN interface 321. Thus, the computer, with the CPU 301 of the image recognition device 30 as its central component, functions as a receiving unit that receives images of products to be registered, transmitted from the scanner 20, which is a higher-level device. The receiving means 3001 stores the received product images in the RAM 303.

[0128] The image recognition means 3002 performs image recognition on product images using an image recognition function based on a trained model 312. Specifically, the image recognition means 3002 acquires feature quantities contained in the product image, compares the acquired feature quantities with the matching dictionary 313, and obtains a code value associated with the feature quantity closest to the acquired feature quantity. Thus, the computer, with the CPU 301 of the image recognition device 30 as its central component, functions as an acquisition unit that acquires the code value of the photographed product by comparing the appearance image of the product contained in the image with the matching dictionary 313. The image recognition means 3002 determines the code type from the acquired code value. If the matching dictionary 313 stores the code type and code value associated with the feature quantities, the image recognition means 3002 can acquire the code type and code value from the product image. The image recognition means 3002 determines the code mark of this mark type according to the settings in the mark setting table 3141 and adds the determined code mark to the acquired code value. The image recognition means 3002 stores the code value to which the code mark has been added as the image recognition result in RAM 303. Thus, the computer with the CPU 301 of the image recognition device 30 as its central component functions as a recognition unit that acquires the code value by recognizing the appearance of the product from the image of the product's appearance included in the image of the product to be registered.

[0129] The transmission means 3005 transmits the image recognition result from the image recognition means 3002, stored in the RAM 303, to the transmission / reception means 2003 of the scanner 20 via the LAN interface 321. Thus, the computer with the CPU 301 of the image recognition device 30 as its central component functions as a transmission unit that sends the code value to a higher-level device.

[0130] The determination means 3003 determines whether the image recognition by the image recognition means 3002 was successful. For example, if the difference between the acquired feature and the feature closest to the acquired feature is less than a predetermined amount, the determination means 3003 determines that the image recognition was successful. Conversely, if the difference is greater than or equal to the predetermined amount, the determination means 3003 determines that the image recognition was unsuccessful. Furthermore, if the determination means 3003 determines that the image recognition was unsuccessful, it determines that the product image on which the image recognition was performed will be used for additional training.

[0131] Furthermore, the receiving means 3001 receives the decoded result from the scanner 20 via the LAN interface 321. The receiving means 3001 stores the received decoded result in the RAM 303.

[0132] If the determination means 3003 determines that the product image it has performed image recognition on is a product image to be used for additional training, the transmission means 3005 transmits that product image and its decoding result to the server 40 via the LAN interface 321. In other words, since it has been determined that the image recognition failed, the reliability of the image recognition result by the image recognition means 3002 is low. Therefore, the decoding result of the product barcode is transmitted to the server 40 as data for additional training.

[0133] Server 40 uses the received product image and the decoding result to further train the pre-trained model 412 for product image recognition and updates the pre-trained model 412 and the matching dictionary 413. Server 40 transmits the updated pre-trained model 412 and the updated matching dictionary 413 to the receiving means 3001 of the image recognition device 30 via the LAN interface 321.

[0134] The receiving means 3001 receives the updated trained model 412 and the updated matching dictionary 413 from the server 40 via the LAN interface 321. The receiving means 3001 stores the received updated trained model 412 and the updated matching dictionary 413 in the RAM 303.

[0135] The update means 3004 updates the learned model 312 currently held by the storage unit 310 with the received updated learned model 412. The image recognition device 30 also updates the matching dictionary 313 currently held by the storage unit 310 with the received updated matching dictionary 413.

[0136] (Server Functional Configuration) Next, the functional configuration of the server 40 will be described with reference to Figure 14. Figure 14 is a block diagram showing the functional configuration of the control unit 400 of the server 40.

[0137] The control unit 400 includes a receiving means 4001, an additional learning means 4002, a master management means 4003, a transmitting means 4004, and a transaction management means 4005. The control unit 400 functions as the receiving means 4001, the additional learning means 4002, the master management means 4003, the transmitting means 4004, the transaction management means 4005, and the setting means 4006, as the CPU 401 operates according to a control program stored in the ROM 402 and the storage unit 410. Note that each of these functions may be configured with hardware such as dedicated circuits.

[0138] The receiving means 4001 receives product images for additional learning and the decoding results of those product images from the image recognition device 30 via the communication unit 421. The receiving means 4001 stores the received product images and decoding results in the RAM 403. The receiving means 4001 may also store and accumulate the received product images in the RAM 403 or storage unit 410, for example, as product images for analyzing the state (event) of image recognition by the image recognition device 30.

[0139] The additional learning means 4002 performs additional learning on the trained model 412 held by the memory unit 410 at an appropriate time, for example, when a certain amount of product images and decoding results for additional learning have been accumulated, and updates the trained model 412. In addition, the additional learning means 4002 updates the matching dictionary 413 held by the memory unit 410 in conjunction with the additional learning of the trained model 412.

[0140] The transmission means 4004 transmits the updated trained model 412, updated by the additional learning means 4002, and the matching dictionary 413 to the receiving means 3001 of the image recognition device 30 via the communication unit 421, for example, as an image file or a compressed file.

[0141] Furthermore, the receiving means 4001 receives accounting processing information from the POS terminal 10 via the communication unit 421. The receiving means 4001 stores the received accounting processing information in the RAM 403.

[0142] The transaction management means 4005 manages the transaction database 414 held by the storage unit 410 based on the received accounting processing information.

[0143] The master management means 4003 updates the information in the product master 415 based on update information for the product master 415 that is input from an input means (not shown) or received from a management terminal (not shown) by the receiving means 4001.

[0144] The setting means 4006 accepts setting operations for selection rules for each scanner 20 from the input device 422.

[0145] The transmission means 4004 transmits a product master 112 suitable for each store to the POS terminal 10 of that store via the communication unit 421, based on the product master 415 updated by the master management means 4003. The transmission means 4004 also transmits the selection rules for each scanner 20 received by the setting means 4006 to the POS terminal 10 corresponding to each scanner 20 via the communication unit 421.

[0146] (Operation of scanner and image recognition device) Next, referring to Figures 15 and 16, the operation of the scanner 20 and image recognition device 30 involved in registering purchased products in the product information management system MS will be explained.

[0147] Figure 15 is a sequence diagram showing the operation overview of the product information management system MS. As shown in Figure 15, the process related to the registration of purchased products performed by the product information management system MS includes a product registration process PRO1, which is repeatedly performed each time a product is registered in the product information registration system RS, and a learning update process PRO2, which is performed at any given time. As mentioned above, the learning update process PRO2 is performed, for example, when a certain amount of product images and decoding results for additional learning have been accumulated.

[0148] First, let's explain the product registration process PRO1. Here, we will explain using the example where both the decoding result and the image recognition result are obtained. Detailed operation, including cases where only one of them is obtained, will be described later using the operation flowcharts of the scanner 20 and the image recognition device 30. When the scanner 20 captures an image of the product (step S11), the scanner 20 determines whether it was able to detect the product barcode from the captured product image (step S12). If the product barcode was not detected, it returns to step S11.

[0149] Furthermore, the scanner 20 transmits the product image captured in step S11 to the image recognition device 30.

[0150] When the image recognition device 30 receives a product image, it performs a process to obtain an image recognition result including the code value of the product by recognizing the image of the product shown in the captured image (step S13).

[0151] The image recognition device 30 determines whether image recognition was successful, that is, whether an image recognition result was obtained (step S14). If image recognition was not successful, it sends a product image request to the scanner 20 requesting that a new product image be sent. As a result, a new product image is sent from the scanner 20, and image recognition processing can be performed on this new product image.

[0152] If image recognition is successful, the image recognition device 30 transmits the image recognition result, that is, the code value with the code mark attached, to the scanner 20.

[0153] Furthermore, if the scanner 20 can detect a product barcode from the product image, it decodes the product barcode and obtains the decoded result (step S15). The scanner 20 then transmits the obtained decoded result, i.e., the code value with the code mark attached, to the image recognition device 30.

[0154] Then, the scanner 20 selects one of the acquired decoded result (i.e., the code value with the code mark attached) and the image recognition result (i.e., the code value with the code mark attached) transmitted from the image recognition device 30 according to a pre-set selection rule to create an output code (step S16). If the scanner 20 has only acquired either the decoded result or the image recognition result, it selects that result to create the output code. The scanner 20 then transmits the created output code to the POS terminal 10.

[0155] The POS terminal 10 performs the purchase registration process based on the output code transmitted from the scanner 20 (step S17). When the product registration process is complete, the POS terminal 10 can output a notification sound to indicate this.

[0156] Furthermore, the image recognition device 30 determines whether additional training is necessary, that is, whether there are product images to be used for additional training (step S18). If additional training is necessary, the image recognition device 30 sends the product images and training data including the acquired decoding results to the server 40.

[0157] The product information registration system RS performs the product registration process PRO1 described above for each product. Once the registration of all products is complete, that is, once the product registration for all products in a single transaction is complete, the POS terminal 10 performs the accounting process (step S21).

[0158] Next, we will explain the learning update process PRO2. In Figure 15, this learning update process PRO2 is shown to be performed immediately after the product registration process PRO1, but it can be performed concurrently with the product registration process PRO1.

[0159] Once a predetermined amount of training data has been accumulated, the server 40 uses that training data to perform additional training on the trained model 412 it holds and update it (step S31). The server 40 also updates the matching dictionary 413 it holds in conjunction with this additional training of the trained model 412 (step S32). The server 40 then transmits the updated trained model 412 and matching dictionary 413 to the image recognition device 30, for example, as update data in an image file or compressed file.

[0160] The image recognition device 30 receives this update data and updates the trained model 312 and matching dictionary 313 that it holds (step S33). This update may be performed immediately upon receiving the update data, but since the update process takes a certain amount of time, updating in the middle of the product registration process PRO1 would make customers wait, which is undesirable. Therefore, the update should be performed when the image recognition device 30 starts operating, such as when the store opens, or during a batch process at night. Alternatively, the timing of the update in the image recognition device 30 can be controlled by adjusting the time at which the server 40 sends the update data.

[0161] Figure 16 is a flowchart illustrating the operation flow of the scanner 20 and image recognition device 30 in the embodiment when registering one product. In Figure 16, the operation of the scanner 20 is shown on the left, and the operation of the image recognition device 30 is shown on the right. The operation of the scanner 20 is performed by each part of the scanner 20 under the control of the control unit 200 which executes the control program 211. The operation of the image recognition device 30 is performed by each part of the image recognition device 30 under the control of the control unit 300 which executes the control program 311. Unless otherwise specified, the processing operations of the scanner 20 and image recognition device 30 shown in Figure 16 are assumed to transition from ACTn (where n is a natural number) to ACT(n+1).

[0162] In ACT11, the scanner 20 acquires a single product image from the moving image captured by the imaging unit 220. For example, first, the scanner 20 stores all or some of the multiple frame images sequentially captured by the imaging unit 220 in the RAM 203 as candidate product images. Next, the scanner 20 selects one frame image from the multiple frame images held in the RAM 203 and uses the selected frame image as the product image. The scanner 20 also assigns an identification code to the product image to identify it. For convenience, the product image will be assumed to include the identification code below.

[0163] In ACT12, the scanner 20 transmits the product image to the image recognition device 30 via LAN, for example, using TCP communication. Thus, the CPU 201 performs information processing based on the control program, and the computer with the CPU 201 as its central component functions as an imaging unit that outputs the image of the product to be registered to the image recognition device 30, in addition to the decoding unit.

[0164] In ACT21, the image recognition device 30 receives product images and identification codes from the scanner 20 via the LAN. Thus, the CPU 301 performs information processing based on the product image recognition program, and the computer with the CPU 301 as its central component functions as a receiving unit that receives images of products to be registered transmitted from a higher-level device. The image recognition device 30 stores the received product images in the RAM 303.

[0165] In ACT22, the image recognition device 30 performs image recognition on a product image. For example, the image recognition device 30 performs image recognition based on features such as the outline, color, and text of the product shown in the product image. In this case, first, the image recognition device 30 acquires the features contained in the product image. Next, the image recognition device 30 compares the acquired features with the matching dictionary 313 and searches for the feature that is closest to the acquired features. Subsequently, the image recognition device 30 acquires a code value associated with the search result (the feature that is closest to the acquired features). Thus, by having the CPU 301 execute information processing based on the product image recognition program, the computer with the CPU 301 as its central component functions as an acquisition unit that acquires the code value of the photographed product by comparing the appearance image of the product contained in the photograph of the product to be registered, transmitted from a higher-level device, with the matching dictionary.

[0166] For example, consider registering a product with a barcode bearing the JAN-13 code value "4912345678904". The matching dictionary 313 of the image recognition device 30 stores the code value "4912345678904" associated with the product's features. In this embodiment, the correct image recognition data for each product sold by the store is pre-specified to the image recognition device 30. By doing so, the image recognition device 30 obtains a code value consisting of 13 digits, "4912345678904", from the product image. Furthermore, if the matching dictionary 313 of the image recognition device 30 stores, for example, a code value "491234567890412" including a 2-digit add-on code associated with the product's features, it obtains a code value consisting of 15 digits, "491234567890412", from the product image. The image recognition device 30 identifies the code type as "JAN-13" based on the number of digits in the code value. If the matching dictionary 313 stores the code type and code value in association with features, the image recognition device 30 can obtain the code type "JAN-13" and the code value "4912345678904" or "491234567890412" from the product image. The image recognition device 30 obtains a pre-set code mark for this code type, for example "F", from the mark setting table 3141. If the code type cannot be identified, the device assigns the most likely code type, for example, the one with the closest number of digits. In this embodiment, the format of the image recognition result of the image recognition device 30 is defined as "code mark" + "code value". Therefore, the image recognition result obtained by the image recognition device 30 will be "F4912345678904" or "F491234567890412".

[0167] In ACT23, the image recognition device 30 determines whether image recognition was successful. For example, first, the image recognition device 30 calculates the difference between the acquired feature and the feature that is closest to the acquired feature. Next, the image recognition device 30 compares the calculated difference with a predetermined amount.

[0168] If the difference in the comparison is less than a predetermined amount, the image recognition device 30 determines that the image recognition was successful. In this case (if Yes in ACT23), the image recognition device 30 saves the image recognition result obtained in ACT22 to RAM 303. Then, it proceeds to the operation of ACT25.

[0169] Conversely, if the difference in the comparison is greater than or equal to a predetermined amount, the image recognition device 30 determines that the image recognition has failed. In this case (when the result is No in ACT23), the image recognition device 30 proceeds to the operation of ACT24.

[0170] As ACT24, the image recognition device 30 sends a product image request to the scanner 20, requesting the acquisition and transmission of another product image, and repeats the operations of ACT21 to ACT23. The image recognition device 30 also determines that product images that failed to be recognized are to be used for additional training and saves them in RAM 303.

[0171] In ACT25, the image recognition device 30 transmits the image recognition results held in the RAM 303 to the scanner 20 via the LAN. Thus, the CPU 301 performs information processing based on the product image recognition program, and the computer with the CPU 301 as its central component functions as a transmission unit that sends the acquired product code value to a higher-level device. The image recognition device 30 also transmits the identification code of the product image from which the image recognition results were obtained to the scanner 20.

[0172] In ACT13, the scanner 20 detects the product barcode in the product image. In other words, the scanner 20 determines whether or not a product barcode is captured in the product image.

[0173] If the product barcode cannot be detected in ACT13 (resulting in a "No" in ACT13), the process moves to ACT14. In ACT14, the scanner 20 checks whether it has received the image recognition result ("code mark" + "code value") of the product image from the image recognition device 30. If the image recognition result has been received (resulting in a "Yes" in ACT14), the scanner 20 moves to ACT19. If the image recognition result has not been received (resulting in a "No" in ACT14), the scanner 20 moves to ACT15.

[0174] In ACT15, the scanner 20 determines whether a certain amount of time has elapsed since it began detecting the product barcode of a single product. If a certain amount of time has elapsed (Yes in ACT15), the scanner 20 proceeds to the operation of ACT110. If a certain amount of time has not yet elapsed (No in ACT15), the scanner 20 returns to the operation of ACT11. In this way, the scanner 20 repeatedly detects product barcodes and receives image recognition results for a certain period of time.

[0175] If a product barcode is detected in ACT13 (Yes in ACT13), then in ACT16, scanner 20 extracts and decodes the detected product barcode. For example, if a product has a barcode with a JAN-13 code of the aforementioned code value "4912345678904" or a code value including a two-digit add-on code "491234567890412", scanner 20 decodes the product barcode to obtain a code value consisting of a 13-digit number "4912345678904" or a 15-digit number "491234567890412". Based on the number of digits in this code value, scanner 20 determines that the code type is "JAN-13" and obtains a pre-set code mark for this code type, for example "F", from the mark setting table 2121. The format of the decode result from scanner 20 is defined as "code mark" + "code value". Therefore, the decoded result obtained by scanner 20 will be either "F4912345678904" or "F491234567890412".

[0176] As ACT17, scanner 20 saves the acquired decoding result to RAM 203.

[0177] Next, in ACT18, the scanner 20 transmits the decoded result (code mark + code value) to the image recognition device 30. After that, the scanner 20 proceeds to the operation of ACT14.

[0178] Furthermore, if an image recognition result is received in ACT14 (if Yes in ACT14), the scanner 20 stores the received image recognition result in RAM 203 in ACT19. Thus, by having the CPU 201 perform information processing based on the control program, the computer with the CPU 201 as its central component functions as a code value receiving unit that receives code values ​​output from the image recognition device 30. The scanner 20 may also identify a product image corresponding to the image recognition result based on the identification code received along with the image recognition result. After that, the scanner 20 proceeds to the operation of ACT15.

[0179] Therefore, within a certain period of time, the scanner 20 can obtain one or more decoding results or image recognition results, or one or more of each. Alternatively, before the determination of the elapsed time in ACT 15, the scanner 20 may determine whether both decoding results and image recognition results have been obtained. If both results are obtained, it may skip ACT 15 and proceed to the operation of ACT 110.

[0180] In ACT110, the scanner 20 selects either the decoded result ("code mark" + "code value") or the image recognition result ("code mark" + "code value") according to the selection rules pre-set in the selection rule setting table 2122.

[0181] As ACT111, scanner 20 creates an output code that includes the selected result code mark and code value, for example, "F4912345678904" or "F491234567890412".

[0182] In ACT112, the scanner 20 transmits the generated output code to the POS terminal 10. Thus, the CPU 201 performs information processing based on the control program, and the computer, with the CPU 201 as its central component, functions as a transmitting unit that selects either the code value obtained by the decoding unit or the code value obtained by the image recognition device 30 based on predetermined rules and transmits it to the higher-level device, the POS terminal 10. The scanner 20 then completes its operation for registering this one product.

[0183] In ACT18, when the scanner 20 transmits the decoded result ("code mark" + "code value"), the image recognition device 30 receives the decoded result in ACT20. The image recognition device 30 stores the received decoded result in RAM 303. In Figure 16, the operation of ACT20, in which the image recognition device 30 receives this decoded result, is shown between ACT23 and ACT25, but the timing of when the scanner 20 transmits the decoded result to the image recognition device 30 is undefined. In other words, the operation of ACT20 is executed in response to the transmission of the decoded result by the scanner 20. Therefore, the operation of ACT20 is placed before ACT25 in Figure 16 for convenience only, and will actually occur at some point before ACT26, which will be described later.

[0184] In ACT26, the image recognition device 30 determines whether there are product images to be used for additional training. For example, if there are product images that failed to be recognized in ACT23, and the image recognition device 30 has received a decoded result from the scanner 20 in ACT25, the image recognition device 30 determines that there are product images to be used for additional training.

[0185] In the ACT26 determination, if it is determined that there are product images to be used for additional training (Yes in ACT26), then in ACT27, the image recognition device 30 sends the product images that failed to be recognized, which are held in RAM 303, and the decoded results received from scanner 20 to server 40. The image recognition device 30 then completes the operation for registering this one product.

[0186] If, in the ACT26 determination, it is determined that there are no product images to use for additional training (resulting in a "No" in ACT26), the image recognition device 30 skips the operation of ACT27 and terminates the operation for registering this single product.

[0187] As described above, the product information registration system RS according to the first embodiment includes: a decoding means 2002 that decodes the product barcode from an image of the product barcode contained in a product image which is an image of the product to be registered, thereby obtaining a code value represented by the product barcode; an image recognition means 3002 that recognizes the appearance of the product from an image of the product's appearance contained in a product image which is an image of the product to be registered, thereby obtaining a code value; a selection means 2004 that selects one of the code value obtained by the decoding means 2002 and the code value obtained by the image recognition means 3002 based on predetermined rules; and a registration means 1002 that registers information relating to the product identified by the product code corresponding to the code value selected by the selection means 2004.

[0188] In this way, the same information as barcodes is defined in the correct data for image recognition, and the output codes for code recognition and image recognition are formed according to their intended use, thereby enabling the coexistence of code recognition and image recognition. That is, the code value is included in the output result of image recognition, and the output result is converted into a code in the same format as the result of code recognition. Therefore, even when code recognition and image recognition are used together, it is possible to prevent the same product from being registered twice as different products. In addition, since product registration is possible based on the code value selected according to the rules set in advance for barcode recognition and image recognition, it is also possible to prevent a single product from being registered multiple times.

[0189] Furthermore, according to the product information registration system RS of the first embodiment, a matching dictionary 313 is provided which associates the appearance features and code values ​​of each of the multiple products to be registered. The image recognition means 3002 obtains the code value by matching the appearance features of the product with the matching dictionary 313. In this way, by defining the same information as the barcode in the correct image recognition data stored in the matching dictionary 313, it is possible to prevent duplicate registration of the same product even when code recognition and image recognition are used together.

[0190] Furthermore, according to the product information registration system RS of the first embodiment, the selection means 2004 compares a pre-set priority for the code value acquired by the decoding means 2002 with a pre-set priority for the code value acquired by the image recognition means 3002, and selects the code value with the higher priority. Therefore, in stores that handle many products without product barcodes, etc., it becomes possible to appropriately operate the product information registration system RS according to the user's situation, such as prioritizing the image recognition result over the decoding result.

[0191] Furthermore, according to the product information registration system RS of the first embodiment, the selection means 2004 selects the code value that yields a more complete result from among the code value obtained by the decoding means 2002 and the code value obtained by the image recognition means 3002. Therefore, it becomes possible to register products based on a more complete result, thus ensuring high reliability of the registered content.

[0192] Furthermore, according to the product information registration system RS of the first embodiment, the selection means 2004 compares the number of code values ​​obtained by the decoding means 2002 with the number of code values ​​obtained by the image recognition means 3002, and selects the code value with the larger number of obtained values. In this way, by adopting the result with the larger number of obtained values, it becomes possible to register products based on a more reliable result, thus ensuring high reliability of the registered content.

[0193] Furthermore, according to the product information registration system RS of the first embodiment, the selection means 2004 compares the reliability of the code value obtained by the decoding means 2002 with the reliability of the code value obtained by the image recognition means 3002, and selects the code value with the higher reliability. Therefore, product registration based on a more reliable result becomes possible, and thus high reliability can be obtained for the registered content.

[0194] The product information registration system RS comprises a scanner 20 equipped with acquisition means 2001 for photographing and acquiring images of products to be registered, a decoding means 2002, and a selection means; an image recognition device 30 equipped with image recognition means 3002; and a POS terminal 10 equipped with registration means 1002. The scanner 20 further comprises a transmitting and receiving means 2003 that photographs products to be registered, outputs the captured images to the decoding means 2002 and the image recognition device 30, and receives the code values ​​output from the image recognition device 30. Thus, the product information registration system RS can be configured using the POS terminal 10, the scanner 20, and the image recognition device 30.

[0195] Furthermore, as described above, the scanner 20 as an information processing device according to the first embodiment includes: an acquisition means 2001 that photographs the product to be registered and acquires a product image; a decoding means 2002 that decodes the product barcode from the image of the product barcode included in the product image acquired by the acquisition means 2001 to acquire the code value represented by the product barcode; a transmitting and receiving means 2003 that acquires the code value by recognizing the appearance of the product from the appearance image of the product included in the product image acquired by the acquisition means 2001, transmits the acquired code value to an image recognition device 30 that outputs to the scanner 20, and receives the code value acquired by the image recognition device output from the image recognition device 30; and a selection means 2004 that, based on predetermined rules, selects one of the code value acquired by the decoding means 2002 and the code value acquired by the image recognition device 30 and transmits it to the POS terminal 10, which is a higher-level device.Therefore, it is possible to eliminate the discrepancy between the code value of the code recognition result and the code value of the image recognition result, and the same code value is not transmitted redundantly, so it is possible to prevent the same product from being registered redundantly even when code recognition and image recognition are used in combination at the POS terminal 10.

[0196] [Second Embodiment] In the first embodiment, the product information registration system RS is composed of three devices: a POS terminal 10, a scanner 20, and an image recognition device 30. In other words, the functions of the product information registration system RS are distributed among three devices. In contrast, the product information registration system according to the second embodiment is an example in which the functions of the product information registration system RS are integrated into a single device.

[0197] Figure 17 is a block diagram showing the hardware configuration of the product information registration system RS according to the second embodiment. The product information registration system RS is comprised of a POS terminal 10.

[0198] In this embodiment, the storage unit 110 of the POS terminal 10 stores, in addition to the configuration described in the first embodiment, the setting table 212 and processing result 213, which are hardware configurations of the scanner 20, and the learned model 312, matching dictionary 313, and setting table 314, which are hardware configurations of the image recognition device 30. The control program 111 stored in the storage unit includes programs corresponding to the control program 211 of the scanner 20 and the control program 311 of the image recognition device 30. The POS terminal 10 also includes an imaging unit 220, which is a hardware configuration of the scanner 20.

[0199] Figure 18 is a block diagram showing the functional configuration of the control unit 100 of the POS terminal 10. In addition to the functional means described in the first embodiment, the control unit 100 includes an acquisition means 2001, a decoding means 2002, and a selection means 2004, which are functional means of the control unit 200 of the scanner 20, and a receiving means 3001, an image recognition means 3002, a determination means 3003, an update means 3004, and a transmission means 3005, which are functional means of the control unit 300 of the image recognition device 30.

[0200] Thus, even if the functions of the product information registration system RS are integrated into the POS terminal 10, the same operation as in the first embodiment is possible, and therefore the same effects can be obtained.

[0201] Furthermore, in a configuration where the functions of the product information registration system RS are consolidated into a single POS terminal 10, product barcode decoding and image recognition may be performed one at a time rather than in parallel. In this case, the one with the higher priority, according to a pre-set selection rule based on the type, is performed first, and the two are performed alternately until one of the results is obtained. Then, once either the decoding result or the image recognition result has been obtained, an output code including that result is created.

[0202] (others) The above-described embodiment can be modified in various ways as follows.

[0203] For example, the information processing flow described above, performed by the control unit 200 of the scanner 20 and the control unit 300 of the image recognition device 30, is just one example and is not limited to this order. For example, ACT17 and ACT18 in the flowchart of Figure 16 may be performed in reverse order or in parallel. In this way, the order of processing may be changed or multiple processing may be performed in parallel, as long as there is no discrepancy with preceding or succeeding processing.

[0204] Furthermore, although the scanner 20 is configured to always transmit the decoded results to the image recognition device 30, the image recognition device 30 may request the scanner 20 to transmit the decoded results when it determines that additional learning is necessary, and the scanner 20 may transmit the decoded results in response to this request.

[0205] Alternatively, the scanner 20 may send an output code to the POS terminal 10 that includes both the decoded result and the image recognition result, or, if only one result is obtained, the obtained result, without selecting between them. The POS terminal 10 is provided with a selection rule setting table, and the POS terminal 10 selects one of the two results included in the output code sent from the scanner 20 according to the selection rules pre-set in the selection rule setting table. In this case, the scanner 20 must add flag information, etc., to at least one of the decoded result and the image recognition result included in the output code so that the POS terminal 10 can distinguish between the two results. Thus, the computer with the CPU 101 of the POS terminal 10 as its central component functions as a selection unit that selects one of the code value obtained by the decoding unit and the code value obtained by the recognition unit based on pre-defined rules, and a registration unit that registers information about the product identified by the product code corresponding to the code value selected by the selection unit. Even with this configuration, it is possible to eliminate the possibility of the code value from code recognition being different from the code value from image recognition. Furthermore, even if the same code value is transmitted multiple times, the POS terminal 10 can select one result according to the selection rules and register the product, thus preventing the duplicate registration of the same product even when using both code recognition and image recognition.

[0206] In the above embodiment, the control programs executed by the POS terminal 10, scanner 20, image recognition device 30, and server 40 may be provided by recording them on a computer-readable recording medium such as a CD-ROM. Alternatively, the control programs executed by the POS terminal 10, scanner 20, image recognition device 30, and server 40 in the above embodiment may be stored on a computer connected to a network such as the Internet and provided by allowing downloads via the network, or even provided via a network such as the Internet.

[0207] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The original claims of this application are included below. [C1] A decoding unit obtains the code value represented by a code symbol by decoding the code symbol from an image of a code symbol contained in an image of a product to be registered. A recognition unit that acquires the code value by recognizing the appearance of the product from the image of the product's appearance included in the image of the product to be registered, A selection unit that selects one of the code value obtained by the decoding unit and the code value obtained by the recognition unit based on predetermined rules, A registration unit registers information about the product identified by the product code corresponding to the code value selected by the selection unit, A product information registration system equipped with the following features. [C2] The system further includes a matching dictionary that associates the visual features and code values ​​of each of the multiple products to be registered. The recognition unit obtains the code value by comparing the characteristic features of the product's appearance with the matching dictionary. Product information registration system as described in C1. [C3] The selection unit compares a pre-set priority for the code value obtained by the decoding unit with a pre-set priority for the code value obtained by the recognition unit, and selects the code value with the higher priority. Product information registration system as described in C2. [C4] The selection unit selects the code value obtained from the code value acquired by the decoding unit and the code value acquired by the recognition unit that yields a more complete result. Product information registration system as described in C2. [C5] The selection unit compares the number of code values ​​obtained by the decoding unit with the number of code values ​​obtained by the recognition unit, and selects the code value with the larger number of obtained values. Product information registration system as described in C2. [C6] The selection unit compares the reliability of the code value obtained by the decoding unit with the reliability of the code value obtained by the recognition unit, and selects the code value with the higher reliability. Product information registration system as described in C2. [C7] The aforementioned product information registration system is A scanner comprising an image acquisition unit, a decoding unit, and a selection unit, which photograph the product to be registered and acquire the image; A product image recognition device comprising the aforementioned recognition unit, A registration device comprising the registration unit, It has, The aforementioned scanner, A shooting unit outputs the image acquired by the image acquisition unit to the product image recognition device, in addition to the decoding unit. The system further comprises a code value receiving unit that receives the code value output from the product image recognition device, Product information registration system as described in C1. [C8] An image acquisition unit that acquires images of products to be registered, A decoding unit obtains the code value represented by a code symbol by decoding the code symbol from the image of the code symbol contained in the image acquired by the image acquisition unit, A transmitting / receiving unit transmits the image acquired by the image acquisition unit to an image recognition device that acquires the code value by recognizing the appearance of the product from the appearance image of the product contained in the image, and receives the code value acquired by the image recognition device output from the image recognition device. A transmission unit that, based on predetermined rules, selects one of the code values ​​obtained by the decoding unit and the code value obtained by the image recognition device and transmits it to a higher-level device. An information processing device equipped with the following: [C9] A computer that has a processor and memory, Image acquisition unit that acquires images of products to be registered. A decoding unit obtains the code value represented by the code symbol by decoding the code symbol from the image of the code symbol contained in the image obtained by the image acquisition unit. A transmitting / receiving unit transmits the image acquired by the image acquisition unit to an image recognition device that acquires the code value by recognizing the appearance of the product from the appearance image of the product contained in the image, and receives the code value acquired by the image recognition device output from the image recognition device. A transmitting unit that, based on predetermined rules, selects one of the code values ​​obtained by the decoding unit and the code value obtained by the image recognition device and transmits it to a higher-level device. A program designed to function as such. [Explanation of symbols]

[0208] 1...Self-service POS system, 10...POS terminal, 100,200,300,400...Control unit, 1001...Input means, 1002...Registration means, 1003...Accounting processing means, 1004...Output means, 101,201,301,401...CPU, 102,202,302,402...ROM, 103,203,303,403...RAM, 1031...Product information unit, 110,210,310,410...Storage unit, 111,211,311,411...Control program, 112,415...Product master, 120 ...Audio output unit, 121, 221...Connection interface, 122, 222, 321...LAN interface, 123, 223, 322, 424...Bus, 50...Main unit, 51...Display, 52...Touch panel, 53...Printer, 531...Issuing slot, 54...Card reader, 541...Card insertion slot, 55...Change machine, 551...Coin slot, 552...Coin outlet, 553...Coin tray, 554...Banknote slot, 555...Banknote outlet, 57...Pole, 571...Light-emitting unit, 70...First basket mounting base, 7 1...First weighing unit, 80...Second basket placement platform, 81...Second weighing unit, 82...Support rod, 83...Temporary placement platform, 20...Scanner, 2001...Acquisition means, 2002...Decoding means, 2003...Transmission and reception means, 2004...Selection means, 2005...Transmission means, 212,314...Setting tables, 2121,3141...Mark setting tables, 2122...Selection rule setting table, 213...Processing result, 220...Imaging unit, 230...Reading window, 30...Image recognition device, 3001...Receiving means, 3002...Image recognition means, 3003 ...determination means, 3004...update means, 3005...transmission means, 312, 412...trained models, 313, 413...matching dictionaries, 40...server, 4002...additional learning means, 4003...master management means, 4004...transmission means, 4005...transaction management means, 4006...setting means, 414...transaction database, 421...communication unit, 422...input device, 423...output device, LH...LAN hub, MS...product information management system, NW...network, RS...product information registration system, RT...router.

Claims

1. A decoding unit obtains the code value represented by a code symbol by decoding the code symbol from an image of a code symbol contained in an image of a product to be registered. A matching dictionary that associates the appearance features and code values ​​of each of the multiple products to be registered, A recognition unit that recognizes the appearance of the product from the appearance image of the product included in the image of the product taken of the product to be registered, and obtains a code value associated with the appearance features of the product by comparing the appearance features of the product with the matching dictionary, A selection unit selects one of the code value obtained by the decoding unit and the code value obtained by the recognition unit based on a predetermined selection rule, wherein the selection rule includes a rule for selecting the code value from among the code value obtained by the decoding unit and the code value obtained by the recognition unit that yields a more complete result, A registration unit registers information about the product identified by the product code corresponding to the code value selected by the selection unit, A product information registration system equipped with the following features.

2. The selection rule includes a rule that compares a predetermined priority for the code value obtained by the decoding unit with a predetermined priority for the code value obtained by the recognition unit, and selects the code value with the higher priority. The product information registration system according to claim 1.

3. The decoding unit and the recognition unit repeatedly perform the decoding of the code symbol and the recognition of the appearance of the product for a certain period of time for one product, The selection rule includes a rule that compares the number of code values ​​obtained by the decoding unit during the specified period of time with the number of code values ​​obtained by the recognition unit during the specified period of time, and selects the code value with the larger number of obtained values. The product information registration system according to claim 1.

4. The selection rule includes a rule that compares the confidence level based on the reading status of the code value obtained by the decoding unit with the confidence level of the code value obtained by the recognition unit, which is based on the difference between the feature quantity of the product's appearance and the feature quantity of the matching dictionary that is closest to said feature quantity, and selects the code value with the higher confidence level. The product information registration system according to claim 1.

5. The aforementioned product information registration system is A scanner comprising an image acquisition unit, a decoding unit, and a selection unit, which photograph the product to be registered and acquire the image; A product image recognition device comprising the recognition unit and the matching dictionary, A registration device comprising the registration unit, It has, The aforementioned scanner, A shooting unit outputs the image acquired by the image acquisition unit to the product image recognition device, in addition to the decoding unit. The system further comprises a code value receiving unit that receives the code value output from the product image recognition device, The product information registration system according to claim 1.

6. An image acquisition unit that acquires images of products to be registered, A decoding unit obtains the code value represented by a code symbol by decoding the code symbol from the image of the code symbol contained in the image acquired by the image acquisition unit, A transmitting / receiving unit transmits the image acquired by the image acquisition unit to an image recognition device that acquires a code value by recognizing the appearance of the product from the appearance image of the product contained in the image, and receives the code value acquired by the image recognition device output from the image recognition device. A transmission unit selects one of the code values ​​obtained by the decoding unit and the code value obtained by the image recognition device based on a predetermined selection rule and transmits it to a higher-level device, wherein the selection rule includes a rule for selecting the code value obtained by the decoding unit and the code value obtained by the image recognition device that yields a more complete result, An information processing device equipped with the following:

7. A computer that has a processor and memory, Image acquisition unit that acquires images of products to be registered. A decoding unit obtains the code value represented by the code symbol by decoding the code symbol from the image of the code symbol contained in the image obtained by the image acquisition unit. A transmitting / receiving unit transmits the image acquired by the image acquisition unit to an image recognition device that acquires a code value by recognizing the appearance of the product from the appearance image of the product contained in the image, and receives the code value acquired by the image recognition device output from the image recognition device. A transmission unit selects one of the code value obtained by the decoding unit and the code value obtained by the image recognition device based on a predetermined selection rule and transmits it to a higher-level device, wherein the selection rule includes a rule for selecting the code value obtained by the decoding unit and the code value obtained by the image recognition device that yields a more complete result. A program designed to function as such.

Images