Image recognition device, storage medium, and image recognition method

Abstract

The present invention discloses an image recognition device, storage medium, and image recognition method capable of accurately identifying products based on camera images. The image recognition device takes a camera image of a product as input and identifies the product through deep learning using a shared first learning model. The image recognition device takes a camera image of a product as input and identifies the product through deep learning using a unique second learning model. The image recognition device determines the product based on the recognition results from deep learning using the first learning model and the recognition results from deep learning using the second learning model.

Description

[0001] This application claims priority from Japanese patent application No. JP2020-136189, filed on August 12, 2020, and cites the contents of the aforementioned application, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0002] Embodiments of the present invention relate to an image recognition device, a storage medium storing a control program for causing a computer to function as the device, and an image recognition method. Background Art

[0003] Technology is known for using deep learning techniques such as convolutional neural networks to identify products in images captured by cameras. Furthermore, image recognition devices have been developed that utilize this technology to automatically identify products such as fresh produce and fruit that do not have barcodes. These image recognition devices identify products by inputting images of the products being identified into a learning model. A learning model is a machine learning model. A learning model is created by extracting features from a large amount of image data captured of various products and modeling them using a computer equipped with AI (artificial intelligence) capabilities.

[0004] However, apples, for example, are sold nationwide, and their varieties vary widely. Some varieties are sold nationwide, while others are sold only in specific regions. Furthermore, the differences in feature values ​​obtained from camera images for most varieties of apples are small. Therefore, when performing apple image recognition using a learning model with data related to a wide variety of apple varieties, it can sometimes be difficult to distinguish between varieties of apples sold only in specific regions. Summary of the Invention

[0005] In view of the above problems, the technical problem to be solved by the present invention is to provide an image recognition device, a storage medium, and an image recognition method, which can accurately recognize commodities based on camera images.

[0006] To address the above issues, one embodiment of the present invention provides an image recognition device comprising a first recognition unit, a second recognition unit, and a determination unit. The first recognition unit receives a camera image of a product as input and identifies the product through deep learning using a shared first learning model. The second recognition unit receives a camera image of the product as input and identifies the product through deep learning using a unique second learning model. The determination unit determines the product based on the recognition results of the first recognition unit and the second recognition unit.

[0007] According to the above-described image recognition device, it is possible to provide an image recognition device capable of accurately recognizing a commodity based on a captured image.

[0008] In the above-mentioned image recognition device, the first learning model is a learning model commonly used for recognizing commodities in various stores, and the second learning model is a learning model used for recognizing commodities in a limited store.

[0009] According to the above-mentioned image recognition device, it is also possible to accurately recognize commodities sold only in a specific area.

[0010] In the above-mentioned image recognition device, the second learning model is updated by additional learning based on recognition results of products in a limited store, and the first learning model is updated by integrating the learned second learning models.

[0011] According to the above-mentioned image recognition device, the first learning model can recognize a large number of products by integrating the plurality of second learning models.

[0012] In the above-mentioned image recognition device, the identification unit compares the recognition result of the first recognition unit and the recognition result of the second recognition unit, and identifies the commodity based on either recognition result.

[0013] According to the above-mentioned image recognition device, a recognition result can be selected from any one of the recognition results to identify the product.

[0014] In the above-mentioned image recognition device, the identification unit identifies the commodity by increasing the weight of the recognition result of the second recognition unit relative to the recognition result of the first recognition unit.

[0015] According to the above-described image recognition device, it is possible to prioritize the recognition result of the second recognition unit.

[0016] The above-mentioned image recognition device also includes: a judgment unit, which judges whether the product determined by the determination unit is correct; and a sending unit, which sends training data to the server that manages the second learning model. When the training data is judged as a correct answer by the judgment unit, the determined product is used as a correct label; when the training data is judged as an incorrect answer, the corrected product is used as the correct label.

[0017] According to the above-described image recognition device, the server can provide the training data for additional learning of the second learning model.

[0018] The image recognition device may further include an output unit configured to output information of the commodity identified by the identification unit to a terminal, wherein the determination unit determines whether the commodity is correct based on the information from the terminal.

[0019] According to the above-mentioned image recognition device, the authenticity of the product can be determined based on the information from the terminal.

[0020] In the above-mentioned image recognition device, the transmission unit does not transmit the training data when detecting information prohibiting transmission of the training data to the server.

[0021] According to the above-described image recognition device, since the training data is not transmitted to the server, it is possible to prevent the server from executing additional learning of the second learning model.

[0022] A storage medium according to another aspect of the present invention stores a control program for causing a computer having an image recognition device having an interface that takes a camera image of a product as input to function as the following units: a first recognition unit that takes the camera image of the product as input and recognizes the product through deep learning using a common first learning model; a second recognition unit that takes the camera image of the product as input and recognizes the product through deep learning using a unique second learning model; and a determination unit that determines the product based on the recognition results of the first recognition unit and the second recognition unit.

[0023] According to the above-described storage medium, a function capable of accurately recognizing a commodity from a captured image can be provided to the computer of the image recognition device.

[0024] Another aspect of the present invention relates to an image recognition method in which an image recognition device having an interface for taking a camera image of a product as input performs the following steps: taking the camera image of the product as input and identifying the product through deep learning using a common first learning model; taking the camera image of the product as input and identifying the product through deep learning using a unique second learning model; and determining the product based on the product recognition result of deep learning using the first learning model and the product recognition result of deep learning using the second learning model.

[0025] According to the above-described image recognition method, it is possible to provide an image recognition method for accurately recognizing a commodity from a captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a block diagram schematically showing the configuration of an image recognition system including an image recognition device according to one embodiment.

[0027] Figure 2 This is a block diagram showing the main circuit configuration of the center server.

[0028] Figure 3 This is a block diagram showing the main circuit configuration of an edge server.

[0029] Figure 4This is a block diagram showing the main circuit configuration of an image recognition device.

[0030] Figure 5 This is a block diagram showing the main circuit configuration of a POS terminal.

[0031] Figure 6 This is a timing diagram of the main data signals exchanged between the central server, edge server, and image recognition device.

[0032] Figure 7 This is a flowchart showing the main steps of image recognition processing executed by the processor of the image recognition device.

[0033] Figure 8 This is a flowchart showing the main procedures of the processing executed by the processor of the edge server.

[0034] Figure 9 This is a flowchart showing the main procedures of the processing executed by the processor of the center server.

[0035] Description of Reference Numerals

[0036] 10…central server; 20…edge server; 30…image recognition device; 31…processor; 32…main memory; 33…auxiliary storage device; 34…accelerator; 35…device interface; 36, 37…communication interface; 40…POS terminal; 50…first communication network; 60…second communication network; 70…global model; 80…local model; 90…camera device; 100…image recognition system. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments will be described using the drawings.

[0038] It should be noted that this embodiment relates to an image recognition device used in stores of a nationwide retail chain to identify commodities such as fresh food and fruit that do not have barcodes.

[0039] Figure 1 This is a block diagram schematically illustrating the configuration of an image recognition system 100 including an image recognition device 30 according to one embodiment. The image recognition system 100 includes a center server 10, an edge server 20, the image recognition device 30, a POS (Point of Sales) terminal 40, a first communication network 50, and a second communication network 60.

[0040] The image recognition system 100 connects multiple edge servers 20 to a central server 10 via a first communication network 50. The image recognition system 100 connects multiple image recognition devices 30 to each edge server 20 via a second communication network 60. The image recognition system 100 connects a POS terminal 40 to each image recognition device 30. Each image recognition device 30 is connected to each POS terminal 40 on a one-to-one basis via wired or wireless communication.

[0041] The first communication network 50 is a wide area computer network. The second communication network 60 is a computer network with a narrower range than the first communication network 50. Both the first communication network 50 and the second communication network 60 can utilize a well-known computer network.

[0042] Each store is equipped with a combination of an image recognition device 30 and a POS terminal 40. The number of combinations of image recognition devices 30 and POS terminals 40 that a store can have is not particularly limited. A store may have only one set or multiple sets.

[0043] Each store may have an edge server 20, or one edge server 20 may be provided for multiple stores in the same area (the number of edge servers 20 is not limited to one. As long as the edge server 20 is accessed by multiple stores in the same area, it will be sufficient). Alternatively, one edge server 20 may be provided for multiple adjacent areas (the number of edge servers 20 is not limited to one. As long as the edge server 20 is accessed by stores in multiple adjacent areas, it will be sufficient). It should be noted that an area refers to an administrative division such as a city, or refers to a place that includes adjacent administrative divisions, etc.

[0044] The central server 10 is commonly used by all stores in the nationwide chain and can provide computer resources to the edge servers 20 as cloud computing.

[0045] The central server 10 is a computer equipped with AI capabilities. Using its AI capabilities, the central server 10 creates and updates a global model 70. The global model 70 is a learning model used for image recognition of products such as fresh produce and fruit. The global model 70 is a learning model shared by all stores. The global model 70 is an example of a first learning model. The global model 70 is a learning model used commonly across all stores for product recognition (typically used by all stores, but sometimes some stores do not use the global model 70).

[0046] Each edge server 20 is a computer with AI capabilities. Each edge server 20 uses its AI capabilities to create and update a local model 80. Local model 80 is a learning model used for image recognition of products such as fresh produce and fruit. Local model 80 is a learning model specific to a store or a specific area within that store (not shared). Local model 80 is an example of a second learning model. Local model 80 is a learning model used for product recognition in a limited store (not used in all stores, but in a specific range of stores).

[0047] Each image recognition device 30 is a computer with AI capabilities. Using its AI capabilities, each image recognition device 30 identifies a product in a captured image. Each image recognition device 30 then outputs information about the identified product to the corresponding POS terminal 40.

[0048] Each POS terminal 40 registers sales data of products purchased by consumers based on information about the products recognized by the corresponding image recognition device 30. Each POS terminal 40 also performs settlement processing for commercial transactions with consumers based on the registered sales data of the products.

[0049] Figure 2 This is a block diagram showing the main circuit configuration of the center server 10. The center server 10 includes a processor 11, a main memory 12, an auxiliary storage device 13, an accelerator 14, and a communication interface 15. The center server 10 connects the processor 11, main memory 12, auxiliary storage device 13, accelerator 14, and communication interface 15 via a system bus 16. The system bus 16 includes an address bus, a data bus, and other components. The center server 10 connects the processor 11, the main memory 12, and the auxiliary storage device 13 via the system bus 16, thereby forming a computer.

[0050] The processor 11 corresponds to the core of the computer. The processor 11 controls each component to realize various functions of the center server 10 according to the operating system and application programs. The processor 11 is, for example, a CPU (Central Processing Unit).

[0051] The main memory 12 corresponds to the main storage portion of the aforementioned computer. The main memory 12 includes a nonvolatile storage area and a volatile storage area. The main memory 12 stores an operating system or application programs in the nonvolatile storage area. The main memory 12 uses the volatile storage area to store data required for the processor 11 to execute processes for controlling various components. Furthermore, the main memory 12 uses the volatile storage area as a work area where data can be rewritten as needed by the processor 11. An example of a nonvolatile storage area is a ROM (Read Only Memory). An example of a volatile storage area is a RAM (Random Access Memory).

[0052] The auxiliary storage device 13 corresponds to the auxiliary storage portion of the aforementioned computer. For example, known storage devices such as EEPROM (Electric Erasable Programmable Read-Only Memory), HDD (Hard Disc Drive), or SSD (Solid State Drive) can be used as auxiliary storage device 13, either alone or in combination. The auxiliary storage device 13 stores data used by the processor 11 for various processes, as well as data generated by the processes performed by the processor 11. The auxiliary storage device 13 may also store application programs.

[0053] The accelerator 14 is a processing unit that performs image recognition through AI-based deep learning. Deep learning uses, for example, convolutional neural networks. Examples of accelerators 14 include GPUs (Graphics Processing Units) and FPGAs (Field Programmable Gate Arrays).

[0054] The communication interface 15 is an interface that manages data communication with each edge server 20 connected via the first communication network 50 .

[0055] Figure 3This is a block diagram showing the main circuit configuration of edge server 20. Edge server 20 includes a processor 21, main memory 22, auxiliary storage device 23, accelerator 24, first communication interface 25, and second communication interface 26. Edge server 20 connects processor 21, main memory 22, auxiliary storage device 23, accelerator 24, first communication interface 25, and second communication interface 26 via a system bus 27. System bus 27 includes an address bus, a data bus, and other components. Edge server 20 connects processor 21, main memory 22, and auxiliary storage device 23 via system bus 27, thereby forming a computer.

[0056] The basic functions of the processor 21, main memory 22, auxiliary storage device 23, and accelerator 24 are the same as those of the processor 11, main memory 12, auxiliary storage device 13, and accelerator 14 of the center server 10. Therefore, their description is omitted here.

[0057] The first communication interface 25 is an interface that manages data communication with the center server 10 connected via the first communication network 50 .

[0058] The second communication interface 26 is an interface that manages data communication with each image recognition device 30 connected via the second communication network 60 .

[0059] Figure 4 This is a block diagram showing the main circuit configuration of the image recognition device 30. The image recognition device 30 includes a processor 31, a main memory 32, an auxiliary storage device 33, an accelerator 34, a device interface 35, a first communication interface 36, and a second communication interface 37. The image recognition device 30 connects the processor 31 to the main memory 32, the auxiliary storage device 33, the accelerator 34, the device interface 35, the first communication interface 36, and the second communication interface 37 via a system bus 38. The system bus 38 includes an address bus, a data bus, and other components. The image recognition device 30 connects the processor 31 to the main memory 32 and the auxiliary storage device 33 via the system bus 38, thereby forming a computer.

[0060] The basic functions of the processor 31 , main memory 32 , auxiliary storage device 33 , and accelerator 34 are the same as those of the processor 11 , main memory 12 , auxiliary storage device 13 , and accelerator 14 of the center server 10 , and therefore, their description is omitted here.

[0061] The device interface 35 is an interface for connecting to the imaging device 90. The imaging device 90 is a device for capturing an image of a commodity to be recognized. As the imaging device 90, for example, a CCD camera using a CCD (Charge Coupled Device) as an imaging element is used.

[0062] The first communication interface 36 is an interface that manages data communication with the edge server 20 connected via the second communication network 60 .

[0063] The second communication interface 37 is an interface that manages data communication with the POS terminal 40 connected thereto via wired or wireless communication.

[0064] Figure 5 This is a block diagram showing the main circuit configuration of a POS terminal 40. The POS terminal 40 includes a processor 41, a main memory 42, an auxiliary storage device 43, a communication interface 44, an input device 45, a display device 46, a printer 47, and a change dispenser interface 48. The POS terminal 40 connects the processor 41 to the main memory 42, the auxiliary storage device 43, the communication interface 44, the input device 45, the display device 46, the printer 47, and the change dispenser interface 48 via a system bus 49. The system bus 49 includes an address bus, a data bus, and other components. The POS terminal 40 connects the processor 41 to the main memory 42 and the auxiliary storage device 43 via the system bus 49, thereby forming a computer.

[0065] The basic functions of the processor 41, the main memory 42, and the auxiliary storage device 43 are the same as those of the processor 11, the main memory 12, and the auxiliary storage device 13 of the center server 10. Therefore, their description is omitted here.

[0066] The communication interface 44 is an interface for managing data communication with the image recognition device 30 connected by wire or wirelessly. In addition, the communication interface 44 is an interface for managing data communication with other computer devices such as a store server (not shown).

[0067] The input device 45 is a device for inputting necessary data into the POS terminal 40. The input device 45 is, for example, a keyboard, a touch panel sensor, a card reader, a code scanner, or the like.

[0068] The display device 46 is a device for displaying information presented to the operator or customer of the POS terminal 40. The display device 46 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0069] The printer 47 is a printer for printing receipts.

[0070] The change machine interface 48 is an interface that manages data communication with an automatic change machine (not shown).

[0071] Figure 61 is a timing diagram of main data signals exchanged between the center server 10, the edge server 20, and the image recognition device 30. It should be noted that the center server 10 and each edge server 20 are assumed to have a learned model of the global model 70 or the local model 80, respectively.

[0072] First, when an arbitrary learning model distribution timing comes, the center server 10 distributes the global model 70 to each edge server 20 via the first communication network 50 .

[0073] Upon receiving the global model 70 distributed from the center server 10, each edge server 20 stores the global model 70 in the auxiliary storage device 23. Furthermore, each edge server 20 outputs a query command Ca to each image recognition device 30 connected via the second communication network 60, inquiring whether it is ready to receive the learning model. The query command Ca is transmitted to each image recognition device 30 via the second communication network 60.

[0074] The image recognition device 30 , which is ready to receive the learning model, outputs the permission response command Cb to the edge server 20 . The permission response command Cb is transmitted to the edge server 20 via the second communication network 60 .

[0075] The edge server 20 that has received the permission response command Cb transmits the global model 70 and the local model 80 to the image recognition device 30 that is the source of the command through the second communication network 60 .

[0076] When receiving the global model 70 and the local model 80 transmitted from the edge server 20, the image recognition device 30 stores these global model 70 and local model 80 in the auxiliary storage device 33. By storing the global model 70 and the local model 80, the image recognition device 30 can perform product image recognition.

[0077] The image recognition device 30 capable of recognizing an image performs image recognition processing described below. The image recognition device 30 then outputs training data Da obtained through the image recognition processing to the edge server 20. The training data Da is transmitted to the edge server 20 via the second communication network 60.

[0078] The edge server 20 performs additional learning of the local model 80 based on the training data Da transmitted from each image recognition device 30 connected via the second communication network 60. The edge server 20 then outputs learning result data Db, which is the result of the additional learning of the local model 80, to the center server 10. The learning result data Db is transmitted to the center server 10 via the first communication network 50.

[0079] The center server 10 updates the global model 70 by aggregating the learning result data Db transmitted from each edge server 20 connected via the first communication network 50. Then, at a predetermined learning model distribution timing, the center server 10 distributes the global model 70, updated by aggregating the learning result data Db, to each edge server 20 via the first communication network 50. Thereafter, the center server 10, each edge server 20, and each image recognition device 30 repeat the same operations as described above.

[0080] Therefore, the local model 80 held by each edge server 20 is updated through additional learning based on image recognition results processed by one or more image recognition devices 30 connected to the edge server 20. The local model 80 is different for each specific store (different for each edge server 20), and each of the multiple local models 80 is updated through additional learning based on image recognition results processed for product identification in the specific store.

[0081] Meanwhile, the global model 70 held by the center server 10 is updated by aggregating the learning results of the local models 80 held by the edge servers 20. The global model 70 is updated by aggregating the learned results of the multiple local models 80. This global model 70 can be used to identify products in stores that use the local models 80 (products that cannot be identified by a single local model 80).

[0082] Next, the image recognition process executed by the image recognition device 30 will be described.

[0083] Figure 7 31 is a flowchart showing the main sequence of the image recognition process executed by the processor 31 of the image recognition device 30. The processor 31 executes the image recognition process according to the control program installed in the main memory 32 or the auxiliary storage device 33. Figure 7 The flowchart shows the order of image recognition processing.

[0084] It should be noted that the method for installing the control program into the main memory 32 or the auxiliary storage device 33 is not particularly limited. The control program can be stored in a removable storage medium or distributed via communication over a network and then installed into the main memory 32 or the auxiliary storage device 33. The storage medium is not limited as long as it can store the program, such as a CD-ROM or memory card, and can be read by the device.

[0085] When the operator of the POS terminal 40 receives a request from a consumer to check out the purchased goods, he or she operates the input device 45 to declare the start of registration. After receiving the declaration, the POS terminal 40 outputs a start signal to the image recognition device 30. The processor 31 of the image recognition device 30 starts processing according to the start signal. Figure 7 The flow chart shows the order in which information is processed.

[0086] First, as ACT 1, the processor 31 activates the imaging device 90 to start imaging operation. Then, as ACT 2, the processor 31 waits for input of image data of a product.

[0087] The operator takes the commodities purchased by the consumer in his hand one by one and points them towards the lens of the imaging device 90. In this way, the imaging device 90 can be used to capture images of the commodities.

[0088] The processor 31 performs contour extraction processing on the camera image data input via the device interface 35 to determine whether the product has been photographed. And, if the product has been photographed, it is determined as yes in ACT2 and the process proceeds to ACT3. As ACT3, the processor 31 analyzes the camera image data and confirms whether the barcode attached to the product is reflected in the camera image. If the barcode is reflected in the camera image, the processor 31 determines as yes in ACT3 and the process proceeds to ACT4. As ACT4, the processor 31 executes a well-known barcode recognition process for reading the barcoded code based on the image of the barcode. Then, as ACT5, the processor 31 outputs the code as the recognition result of the barcode recognition process to the POS terminal 40. Thereafter, the processor 31 proceeds to ACT16. The processes after ACT16 will be described later.

[0089] On the other hand, if the captured image does not contain a barcode, processor 31 determines "no" in ACT 3 and proceeds to ACT 6. As ACT 6, processor 31 activates accelerator 34. Processor 31 then instructs accelerator 34 to execute a machine learning algorithm using global model 70. Upon receiving this instruction, accelerator 34 inputs the captured image data into global model 70 stored in auxiliary storage device 33, and performs deep learning using, for example, a convolutional neural network to identify the product captured in the captured image.

[0090] Processor 31, acting as ACT 7, obtains recognition result A from accelerator 34. Recognition result A is a list of products for which the similarity between the feature values ​​modeled by global model 70 and the feature values ​​obtained from the captured image is determined to be at least a predetermined threshold. The product list is a list of product categories. The product recognition results obtained through deep learning using global model 70 are stored in auxiliary storage device 33.

[0091] Next, the processor 31, acting as ACT 8, instructs the accelerator 34 to execute a machine learning algorithm using the local model 80. Upon receiving this instruction, the accelerator 34 inputs the image data of the captured image into the local model 80 stored in the auxiliary storage device 33, and performs deep learning using, for example, a convolutional neural network to recognize the product shown in the captured image.

[0092] Processor 31, acting as ACT 9, obtains recognition result B from accelerator 34. Recognition result B is a list of products for which the similarity between the feature values ​​modeled by local model 80 and the feature values ​​obtained from the captured image is determined to be greater than a predetermined threshold. The product list is a list of product categories. The product recognition results obtained through deep learning using local model 80 are stored in auxiliary storage device 33.

[0093] As ACT 10, the processor 31 makes a final judgment on the recognition results A and B. For example, the processor 31 applies a predetermined weight to the similarity of the product items obtained as the recognition result B to increase the similarity. Then, the processor 31 compares the weighted similarity of the product items obtained as the recognition result B with the similarity of the product items obtained as the recognition result A. Then, the processor 31 determines the product items ranked third in descending order of similarity as candidate products. It should be noted that no weighting is also possible. To determine the product, the recognition results using the global model 70 and the recognition results using the local model 80 stored in the auxiliary storage device 33 are compared. The product is determined based on any one of the recognition results. Any one of the recognition results means that the list of candidate products may contain a mixture of the recognition results using the global model 70 and the recognition results using the local model 80. It should be noted that when the similarities of the candidate products are roughly equal, any recognition result can be used. By comparing the recognition results using the global model 70 with the recognition results using the local model 80, products sold in a specific area can be identified using the local model 80. Products sold in all stores can be identified using the global model 70. It should be noted that, in general, even when products not sold in a specific area are sold on a special basis (e.g., only during a special period), the global model 70 can be used for identification.

[0094] A predetermined weight is applied to the similarities. If the similarities obtained as recognition result A and recognition result B are approximately equal, the weighted process results in a higher similarity as recognition result B. Therefore, recognition result B is prioritized over recognition result A.

[0095] As ACT 12, the processor 31 outputs the final judgment result to the POS terminal 40. As a result, the display device 46 of the POS terminal 40 displays a list of candidate commodities ranked from the first to the third obtained as the final judgment result.

[0096] The operator of the POS terminal 40 checks whether the product being viewed by the camera 90, that is, the product being purchased by the consumer, is included in the list of candidate products. If so, the operator performs an operation to select the product. This operation registers and processes the sales data for the purchased product in the POS terminal 40. If the purchased product is not included in the list of candidate products, the operator operates the input device 45 to register and process the sales data for the purchased product. The POS terminal 40 transmits the data for the product selected by the operator to the image recognition device 30.

[0097] The processor 31 that has output the final determination result checks whether or not there is any correction in the final determination result as ACT 12. This is determined based on the commodity data received from the POS terminal 40.

[0098] If the POS terminal 40 selects fewer than two candidate products, the final determination result must be revised. The processor 31 determines yes in ACT 12 and proceeds to ACT 13. In ACT 13, the processor 31 revises the final determination result. Specifically, the processor 31 revises the final determination result so that the selected candidate product is ranked first in the list. The processor 31 then proceeds to ACT 14.

[0099] On the other hand, if there is no correction in the final determination result, that is, if the first candidate product is selected, the processor 31 makes a negative determination in ACT 12 and proceeds to ACT 14 .

[0100] As ACT 14, the processor 31 generates training data Da. Training data Da is data with correct labels attached to product images input via the device interface 35. The correct label is information identifying the candidate product that will be ranked first in the final judgment result. Specifically, if the final judgment result has not been corrected, this is information about the product that was ranked first by the final judgment. If the final judgment result has been corrected, this is data about the product that was moved to the first position by the correction.

[0101] It should be noted that training data Da need not be generated every time. Depending on the store, it is assumed that special sales are being made for products in areas that are not normally sold. In this case, there is no corresponding product in the local model 80, and the product can be determined based on the recognition results of the global model 70. In the case of products that are not normally sold and are unlikely to be processed in the future, it is assumed that the user does not want to perform additional learning on the local model 80. Therefore, in the case of such products, the transmission of training data Da is prohibited. Specifically, when the POS terminal 40 registers the sales data of the purchased product, it pre-stores training prohibition information indicating that the training data Da is not to be transmitted to the edge server 20 in the referenced product file. When registering the product, the POS terminal 40 transmits the training prohibition information to the image recognition device 30. When the sending unit of the image recognition device 30 receives (detects) the information (training prohibition information) prohibiting the transmission of the training data Da to the edge server 20, it does not generate the training data Da. Therefore, the training data Da is not transmitted.

[0102] As the training data Da generated by the ACT15 processor 31 instruction. Figure 6 As shown, the training data Da is sent from the image recognition device 30 to the edge server 20 through this instruction.

[0103] As ACT 16, the processor 31 that has received the training data Da checks whether the registration has been completed. If the registration has not been completed, the processor 31 returns to ACT 2. The processor 31 executes the processes from ACT 2 onwards in the same manner as described above.

[0104] Therefore, each time the operator positions the consumer's purchased goods toward the camera 90, the same processing as in ACT 2 to ACT 16 is repeated. Then, when the operator has completed registering all the sales data of the consumer's purchased goods, he or she operates the input device 45 to declare the registration complete.

[0105] When the processor 31 detects that the registration completion has been announced in the POS terminal 40, it determines that it is yes in ACT 16 and proceeds to ACT 17. As ACT 17, the processor 31 stops the imaging operation of the imaging device 90. Figure 7 The information processing in the order shown in the flowchart is completed.

[0106] Here, the processor 31 cooperates with the accelerator 34 to execute ACT 6 and ACT 7, thereby forming a first recognition unit. Specifically, the processor 31 recognizes the product by taking a captured image of the product as input and using deep learning of the shared global model 70 managed by the center server 10.

[0107] Furthermore, the processor 31 cooperates with the accelerator 34 to execute ACT 8 and ACT 9, thereby forming a second recognition unit. Specifically, the processor 31 similarly recognizes products by using deep learning of the unique local model 80 managed by the edge server 20 and inputting a captured image of the product.

[0108] Processor 31 then executes the processing of ACT 10, thereby forming a determination unit. Specifically, processor 31 determines the product shown in the captured image based on the recognition result obtained through deep learning using global model 70 and the authentication result obtained through deep learning using local model 80. In this case, processor 31 determines the product by weighting recognition result B obtained through deep learning using local model 80 over recognition result A obtained through deep learning using global model 70.

[0109] Furthermore, the processor 31 executes the process of ACT 12 to constitute a determination unit. Specifically, the processor 31 determines whether the product identified by the identification unit is correct.

[0110] The processor 31 then executes ACTs 13 to 15, thereby forming a sending unit. Specifically, the processor 31 sends the training data Da to the edge server 20. If the judgment unit determines that the answer is correct, the product determined by the determination unit is labeled correctly. If the judgment unit determines that the answer is incorrect, the correct product is labeled correctly.

[0111] Furthermore, the processor 31 cooperates with the second communication interface 37 to execute the process of ACT 11, thereby forming an output unit. Specifically, the processor 31 outputs the product information identified by the identification unit to the POS terminal 40. The determination unit then determines the authenticity of the product using the information obtained from the POS terminal 40 in ACT 12.

[0112] Then, the processor 21 of the edge server 20 that receives the training data Da from each image recognition device 30 connected via the second communication network 60 is configured to execute the program Figure 8 The information processing is performed in the order shown in the flowchart. That is, as ACT 21, the processor 21 waits for receiving the training data Da. Then, if the training data Da is received, the processor 21 determines yes in ACT 21 and proceeds to ACT 22. As ACT 22, the processor 21 saves the training data Da to the auxiliary storage device 23.

[0113] In ACT 23, processor 21 checks whether the amount of training data Da stored in auxiliary storage device 23 has reached a predetermined amount. The predetermined amount is any value greater than "2." For example, the predetermined amount is "100." If the amount of training data Da has not reached the predetermined amount, processor 21 determines "no" in ACT 23 and returns to ACT 21. Processor 21 waits for the next training data Da to be received.

[0114] If the amount of training data Da reaches the specified amount, processor 21 determines yes in ACT 23 and proceeds to ACT 24. In ACT 24, processor 21 activates accelerator 24. Processor 21 then instructs accelerator 24 to perform additional learning of local model 80 using the specified amount of training data Da. Upon receiving this instruction, accelerator 24 extracts features from the image data in training data Da, models these features as features of correctly labeled products, and then adds them to local model 80.

[0115] When the additional learning by the accelerator 24 is completed, the processor 21 outputs learning result data Db, which is the result of the additional learning, to the center server 10 as ACT 25. The learning result data Db is the data of the local model 80 updated by the additional learning. After completing the output of the learning result data Db as ACT 26, the processor 21 deletes the specified amount of training data Da stored in the auxiliary storage device 23. The processor 21 then returns to ACT 21.

[0116] Thereafter, the processor 21 stores the training data Da received from each image recognition device 30 , and repeats the processes of additional learning of the local model 80 , transmission of the learning result data Db, and deletion of the training data Da every time the amount of data reaches a predetermined value.

[0117] On the other hand, the processor 11 of the center server 10 that receives the training data Da from each edge server 20 connected via the first communication network 50 is configured to execute the program Figure 9 The information processing is performed in the order shown in the flowchart. Specifically, in ACT 31, the processor 11 waits for receipt of the learning result data Db. Then, if the learning result data Db is received, the processor 11 determines YES in ACT 31 and proceeds to ACT 32. In ACT 32, the processor 11 stores the learning result data Db in the auxiliary storage device 13.

[0118] In ACT 33, the processor 11 checks whether the amount of learning result data Db stored in the auxiliary storage device 23 has reached a predetermined amount. The predetermined amount is any value greater than "2." For example, the predetermined amount is "5." If the amount of learning result data Db has not reached the predetermined amount, the processor 11 determines "no" in ACT 33 and returns to ACT 31. The processor 11 waits for the next learning result data Db to be received.

[0119] If the amount of learning result data Db reaches a specified amount, the processor 11 determines yes in ACT 23 and proceeds to ACT 34. In ACT 34, the processor 11 activates the accelerator 14. The processor 11 then instructs the accelerator 14 to integrate the specified amount of learning result data Db into the global model 70. Upon receiving this instruction, the accelerator 24 updates the global model 70 so that the data of the local model 80, which represents the learning result data Db, is integrated into the global model 70.

[0120] When the accelerator 14 completes the collection of the learning result data Db, the processor 11 distributes the global model 70 updated by the collection of the learning result data Db to each edge server 20 in ACT 35. Furthermore, in ACT 36, the processor 11 deletes a predetermined amount of learning result data Db stored in the auxiliary storage device 13. The processor 11 then returns to ACT 31.

[0121] In this manner, each edge server 20 performs additional learning of the local model 80 using training data Da obtained from the recognition results of each image recognition device 30 connected via the second communication network 60. Each image recognition device 30 connected via the second communication network 60 is installed in the same store or stores within the same area. Therefore, the local model 80 can be said to be a learned model specific to the region.

[0122] Meanwhile, the center server 10 integrates the local models 80 of the edge servers 20 connected via the first communication network 50, and updates the global model 70. Therefore, the global model 70 can be said to be a nationwide shared learning model.

[0123] The global model 70 managed by the center server 10 and the local model 80 managed by the edge server 20 connected via the second communication network 60 are distributed to each image recognition device 30 .

[0124] The image recognition device 30 uses deep learning using the global model 70 to identify a product from a captured image of the product input via the device interface 35. Furthermore, the image recognition device 30 uses deep learning using the local model 80 to identify a product from the captured image. Each image recognition device 30 then identifies the product shown in the captured image based on the product recognition result A obtained through deep learning using the global model 70 and the product recognition result B obtained through deep learning using the local model 80.

[0125] Therefore, according to this embodiment, it is possible to accurately identify goods based on camera images. In particular, in the past, there was a possibility that the goods could not be accurately identified when the feature values ​​of the goods were small. In this embodiment, the purpose is to provide an image recognition device, a storage medium, and an image recognition method that can accurately identify goods even when the difference in feature values ​​obtained from camera images is small depending on the type. Goods are identified not only by deep learning using a nationally shared learning model, namely the global model 70, but also by deep learning using a regionally specific learning model, namely the local model 80. Therefore, even when the difference in feature values ​​obtained from camera images is small depending on the type, goods can be accurately identified.

[0126] Furthermore, in this embodiment, product recognition result B obtained through deep learning using local model 80 is weighted relative to product recognition result A obtained through deep learning using global model 70, so that recognition result B is prioritized over recognition result A in product identification. Therefore, even if the difference in feature values ​​between products sold only in a specific area and products sold only in other areas is very small, products sold only in the specific area are prioritized for identification, further improving accuracy.

[0127] Although the embodiment of the image recognition device has been described above, the present invention is not limited to this embodiment.

[0128] In the above embodiment, Figure 7 In ACT 6, image recognition is performed using the global model 70, and then in ACT 8, image recognition is performed using the local model 80. In this regard, image recognition may be performed first using the local model 80 and then using the global model 70.

[0129] In the above embodiment, the image recognition device 30 performs image recognition by deep learning using a convolutional neural network. The image recognition algorithm is not limited to the convolutional neural network. The image recognition device 30 can also perform image recognition by deep learning using other image recognition algorithms and using the global model 70 and the local model 80.

[0130] In the above embodiment, the image recognition device 30 weights the similarity of feature quantities obtained as recognition results, thereby giving priority to recognition results using the local model 80 over recognition results using the global model 70. The weighting is not limited to similarity. Weighting may also be performed using an index other than similarity to give priority to recognition results using the local model 80.

[0131] The above embodiment illustrates an image recognition device 30 used to identify commodities such as fresh food and fruit that are not accompanied by bar codes. The use of the image recognition device is not limited to devices for identifying commodities such as fresh food and fruit that are not accompanied by bar codes. It can be applied to all image recognition devices that use images to identify items that are distributed nationwide and regionally specific items. In addition, the image recognition device includes an operating unit, a display unit, etc., and displays the information of the commodity determined by the determination unit on the display unit. It is also possible to input correctness from the operating unit, and determine the correctness of the commodity based on the input. In addition, it is also possible to perform an operation to prohibit the transmission of training data from the operating unit. Information prohibiting the transmission of training data based on this operation is input and detected.

[0132] Furthermore, the first learning model is a global model 70 commonly used for product identification across all stores. The second learning model is a local model 70 used for product identification within a specific store. Alternatively, the first learning model may be a higher-level product category (e.g., apples, pears, etc.), and the second learning model may be the products or varieties included in the product category (e.g., one apple, two apples, one pear, two pears). In this case, the region is irrelevant. The higher-level product category, common to all stores, is used as the first learning model. The subdivided products (or varieties) unique to this category are used as the second learning model. Products are identified (product categories) using deep learning using the first learning model. Products are identified (specific products or varieties included in the product category) using deep learning using the second learning model. Products are determined (product categories are determined, and specific products or varieties included in the product category are determined) based on the respective recognition results. In this case, when selecting a product output as a candidate at the POS terminal 40, the product category and variety (or product) are input. Based on the results of this selection, the second learning model performs additional learning and updates for products and categories. The first learning model performs additional learning and updates for product categories. By combining the first learning model for product categories and the second learning model for specific products or categories, even products with minimal differences in feature values ​​derived from camera images can be accurately identified.

[0133] In addition, although several embodiments of the present invention have been described, these embodiments are provided as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways and can be omitted, replaced, or modified without departing from the scope of the invention. These embodiments and their variations are included within the scope of the invention and are included within the scope of the invention described in the claims and their equivalents.

Claims

1. An image recognition device, characterized in that: have: a first recognition unit that receives a captured image of a product as input, recognizes the product through deep learning using a shared first learning model, and creates an item list, the item list being a list segmented into product categories; a second recognition unit that receives the captured image of the product as input, recognizes the product through deep learning using a unique second learning model, and creates an item list, wherein the item list is a list of product categories; as well as a determination unit that determines the commodity based on the recognition result of the first recognition unit and the recognition result of the second recognition unit, The first learning model is a learning model commonly used in various stores for identifying products. The second learning model is a learning model used for identifying products in a limited store. The second learning model is different for each limited store and is updated through additional learning based on the recognition results of products in the limited store. The first learning model is updated by integrating a plurality of learned second learning models.

2. The image recognition device according to claim 1, wherein: The identification unit compares the recognition result of the first identification unit and the recognition result of the second identification unit, and identifies the product based on either recognition result.

3. The image recognition device according to claim 2, wherein: The identification unit identifies the product by increasing the weight of the recognition result of the second recognition unit relative to the recognition result of the first recognition unit.

4. The image recognition device according to any one of claims 1 to 3, wherein: Also features: a determination unit that determines whether the product determined by the determination unit is correct; and The sending unit sends the training data to the server that manages the second learning model. When the training data is determined by the determination unit to be a correct answer, the determined product is used as the correct label. When the training data is determined to be an incorrect answer, the corrected product is used as the correct label.

5. The image recognition device according to claim 4, wherein: Also features: an output unit for outputting information of the commodity determined by the determination unit to a terminal, The determination unit determines whether the product is correct based on the information from the terminal.

6. The image recognition device according to claim 4, wherein: The transmitting unit does not transmit the training data when detecting information prohibiting transmission of the training data to the server.

7. A storage medium storing a control program, characterized in that: The control program is for causing a computer having an image recognition device having an interface that receives a captured image of a product as input to function as the following units: a first recognition unit that receives the captured image of the product as input, recognizes the product through deep learning using a shared first learning model, and creates an item list, the item list being a list of product categories; a second recognition unit that receives the captured image of the product as input, recognizes the product through deep learning using a unique second learning model, and creates an item list, wherein the item list is a list of product categories; as well as a determination unit that determines the commodity based on the recognition result of the first recognition unit and the recognition result of the second recognition unit, The first learning model is a learning model commonly used in various stores for identifying products. The second learning model is a learning model used for identifying products in a limited store. The second learning model is different for each limited store and is updated through additional learning based on the recognition results of products in the limited store. The first learning model is updated by integrating a plurality of learned second learning models.

8. An image recognition method, characterized in that: The image recognition device having an interface that receives a camera image of a product as input performs the following steps: The camera image of the product is used as input, and the product is identified by deep learning using a common first learning model, and an item list is created, wherein the item list is a list of product types that are subdivided. The camera image of the product is used as input, and the product is identified through deep learning using a unique second learning model, and an item list is created. The item list is a list of product types that are subdivided. The commodity is determined based on a commodity recognition result obtained through deep learning using the first learning model and a commodity recognition result obtained through deep learning using the second learning model, The first learning model is a learning model commonly used in various stores for identifying products. The second learning model is a learning model used for identifying products in a limited store. The second learning model is different for each limited store and is updated through additional learning based on the recognition results of products in the limited store. The first learning model is updated by integrating a plurality of learned second learning models.

Images