Commodity identification device, recording medium, and learning method

Abstract

The present invention provides a product identification device (1), characterized in that it includes: an acquisition unit that acquires a captured image; a label detection unit that uses a label detector to determine whether the captured image is a product with a corresponding label; a product group identification unit that uses a first neural network that has learned products with the label to determine which of a plurality of product groups the captured image belongs to, using a first neural network that has learned products with the label; and a product identification unit that uses a second neural network that has been learned for each of the plurality of product groups to identify a product.

Description

Technical Field

[0001] The present invention relates to a commodity identification device, a recording medium, and a learning method. Background Art

[0002] Technologies exist for performing image recognition on captured images of products to identify the products captured. For example, Patent Document 1 discloses an information processing system that uses a deep learning-based recognition model to identify pre-registered products from captured images and verify their size and color, thereby accurately identifying the products.

[0003] Prior art literature

[0004] Patent Literature

[0005] [Patent Document 1] Japanese Patent No. 6209717 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] However, the invention of Patent Document 1 assumes that the photographed object is one of the pre-registered commodity groups. Therefore, there is a problem that when an unregistered commodity is photographed, the commodity must be forcibly classified as one of the registered commodities.

[0008] An object of the present invention is to provide a commodity identifying device and the like that can appropriately identify a commodity based on a captured image.

[0009] Means for solving problems

[0010] A product identification device of the present invention is characterized by including: an acquisition unit that acquires a captured image; a label detection unit that uses a label detector to determine whether the captured image is a product with a corresponding label; a product group identification unit that uses a first neural network that has learned products with the label to determine which of multiple product groups the captured image belongs to, using a first neural network that has learned products with the label; and a product identification unit that uses a second neural network that has been learned for each of the multiple product groups to identify the product.

[0011] A recording medium of the present invention is characterized in that it stores a program that causes a computer to execute the following processing: obtaining a captured image; determining whether the captured image is a product with a corresponding label by detecting the corresponding label from the captured image; using a first neural network that has learned products with the label to determine which of multiple product groups the captured image belongs to; and using a second neural network that has been learned for each of the multiple product groups to determine the product.

[0012] A learning method of the present invention is characterized in that a computer is caused to perform the following processing: obtaining training data, the training data including training images obtained by photographing a product, and information representing the product, the product group to which the product belongs, and the label possessed by the product; and generating identifiers for respectively identifying the product, the product group, and the label based on the training data.

[0013] Effects of the Invention

[0014] One effect of the present invention is that a product can be appropriately identified based on a captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram showing a configuration example of a product identification system.

[0016] Figure 2 This is a block diagram showing a configuration example of a server.

[0017] Figure 3 It is an explanatory diagram showing the outline of this embodiment.

[0018] Figure 4A This is an explanatory diagram for explaining the mark determination process.

[0019] Figure 4B This is an explanatory diagram for explaining the mark determination process.

[0020] Figure 5 This is an explanatory diagram for explaining the product determination process.

[0021] Figure 6 This is an explanatory diagram showing an example of a product confirmation screen.

[0022] Figure 7 This is a flowchart showing an example of a processing procedure of a learning process of a learning model.

[0023] Figure 8 This is a flowchart showing an example of the processing procedure of the product identification process.

[0024] Figure 9 It is an explanatory diagram showing an outline of a modified example.

[0025] Figure 10 This is a flowchart showing an example of the processing procedure of the product identification process according to the modification.

[0026] Figure 11 This is a functional block diagram showing the operation of the server in the above-mentioned manner. DETAILED DESCRIPTION

[0027] Hereinafter, the present invention will be described in detail based on the accompanying drawings showing embodiments of the present invention.

[0028] (Implementation Method)

[0029] Figure 1 This is a schematic diagram illustrating an example configuration of a product identification system. This embodiment describes a product identification system that identifies a product captured in an image taken by a user based on the image. The product identification system includes a product identification device 1 and a terminal 2. The product identification device 1 and the terminal 2 are communicatively connected via a network N, such as the Internet.

[0030] Product identification device 1 is an information processing device capable of performing various information processing and information transmission and reception, such as a server device or personal computer. In this embodiment, product identification device 1 is a server device, hereinafter referred to as server 1 for simplicity. Server 1 performs processing to identify products within a captured image of a product. Specifically, server 1 preliminarily performs machine learning to learn product features from training product images, thereby constructing a learning model for identifying products from images. Server 1 refers to this learning model to identify products contained in images captured by users.

[0031] Terminal 2 is a terminal device held by each user, such as a smartphone, tablet, or personal computer. In this embodiment, terminal 2 is described as a smartphone with a camera function. Terminal 2 takes a picture of a product in response to user input and transmits the captured image to server 1. Server 1 performs image recognition on the image received from terminal 2 to identify the product.

[0032] Figure 2 1 is a block diagram showing a configuration example of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.

[0033] The control unit 11 includes one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and other processing devices. It performs various information processing and control operations on the server 1 by reading and executing programs P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory, and temporarily stores data required for the control unit 11 to execute processing operations. The communication unit 13 includes processing circuits for performing communication-related processing and transmits and receives information to and from the terminal 2 and other devices.

[0034] The auxiliary storage unit 14 is a large-capacity memory, a hard disk, etc., and stores the program P and other data required for the control unit 11 to execute the processing. In addition, the auxiliary storage unit 14 stores the data of the learning model constructed by machine learning processing. Specifically, the auxiliary storage unit 14 stores a label identifier (label detector) 141, a product group identifier 142 (first neural network) and a product identifier 143 (second neural network). The label identifier 141 is used to detect the label on the product container from the image obtained by photographing the product, the product group identifier 142 is used to determine which of the multiple pre-defined product groups the photographed product belongs to, and the product identifier 143 is used to identify specific individual products. The server 1 uses each identifier to perform processing to determine the product based on the photographed image.

[0035] The auxiliary storage unit 14 may be an external storage device connected to the server 1. In addition, the server 1 may be a multi-server composed of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0036] In this embodiment, the server 1 is not limited to the above-described configuration, and may include, for example, an input unit for accepting operation input, a display unit for displaying information related to the server 1 , a reader for reading information stored in a removable storage medium, and the like.

[0037] Figure 3 It is an explanatory diagram showing the outline of this embodiment. Figure 3 , a case is shown in which the server 1 , which has learned the image feature amount of a predetermined product, identifies the product in the image transmitted by the terminal 2 .

[0038] For example, server 1 performs machine learning to build a learning model. This machine learning involves learning features such as the color and shape of product containers using training image data obtained by capturing pre-registered products (e.g., a company's own products). Server 1 receives product image data captured by a user from terminal 2 and, referring to the learning model, determines which of the pre-registered specific products the product captured in the captured image belongs to. For example, server 1 provides information about the identified product to the user.

[0039] However, there are cases where users capture unregistered products (e.g., products from other companies). In such cases, the server 1 classifies the product in the captured image as any of the registered products. Thus, if an unspecified number of images is input, there is a risk that the image will be forcibly classified as any of the registered products, leading to incorrect product identification.

[0040] To prevent this, server 1 performs pre-processing to exclude images of unregistered products. Specifically, server 1 uses label recognition device 141 to detect labels attached to registered products from captured images. If no labels are detected, server 1 excludes the image from processing.

[0041] Figure 4A and Figure 4B This is an explanatory diagram for explaining the mark determination process. Figure 4A The case of extracting local feature amounts from a captured image is shown. Figure 4B , the situation of classifying the extracted local feature values ​​is shown.

[0042] For example, the server 1 extracts local features representing brightness gradients from the captured image and detects labels on the product container based on the extracted local features. For example, the local features are HOG (Histograms of Oriented Gradients) features. The HOG features are features that represent brightness gradients by forming a histogram of the brightness gradient directions in the local areas (units) obtained by dividing the captured image into predetermined units. Figure 4A As shown, the HOG feature is a unit of 5×5 pixels and a block of 3×3 units. The brightness gradient of a block is converted into binary in 8 directions ( Figure 4A The above-mentioned blocks and cells are merely examples and can be modified arbitrarily. Furthermore, the binary representation of gradients is not limited to eight directions. The HOG feature is characterized by being unaffected by geometric deformation and insensitive to changes in image illumination. In this embodiment, server 1 uses the HOG feature as a feature for identifying the label.

[0043] For example, the server 1 determines the mark using an SVM (Support Vector Machine) algorithm based on the HOG feature value. SVM is a pattern recognition method that uses supervised learning and is a method for classifying input data (captured images). Figure 4B As shown in the figure, in SVM, by dividing each category ( Figure 4B The SVM uses this discriminator to determine in which space the features of the input data lie, bounded by the discriminator.

[0044] In this embodiment, server 1 obtains (inputs) training data. This training data is data containing the correct coordinate values ​​of the label portion in a training image of a registered product with a label on its container. A label is a so-called logo composed of text, graphics, symbols, etc., and is a marker that consumers focus on when identifying a product. Furthermore, as described in a later modification, the "label" in this embodiment is not limited to a logo; the structure of the product container, etc., that serves as a characteristic for consumer identification, i.e., the shape of the product, is also included in the label. Server 1 extracts HOG features at the coordinate positions specified by the training data from the training image, representing the brightness gradient of the label portion as a multidimensional feature vector. Server 1 then maps the HOG features of the label portion into a multidimensional feature space and determines the recognition boundary with the maximum separation, thereby generating a label identifier 141 that identifies the label based on the HOG features. Upon receiving a captured image from terminal 2, server 1 uses label identifier 141 to detect the label attached to the product reflected in the image.

[0045] If the logo of a registered product targeted for machine learning is not detected in the captured image due to reasons such as the logo not being present on the captured product, or the logo being present but not being a registered product, server 1 deems the captured product to be a non-registered product and excludes it from processing. For example, server 1 notifies terminal 2 of the failure in logo detection, thereby terminating the series of processing steps.

[0046] Figure 5 This diagram illustrates the product identification process. If the badge detection is successful, server 1 determines which registered product the product belongs to. Specifically, server 1 pre-generates product group identifier 142 and product identifier 143 using training data, and uses each identifier to identify individual products. Product group identifier 142 and product identifier 143 are neural networks using ResNet (Residual Network).

[0047] ResNet is a neural network with the advantages of very deep layers and high recognition accuracy. Figure 5 The upper side shows the conceptual diagram of ResNet. ResNet has the following network structure: calculate every specified number of neuron layers ( Figure 5 The residual between the input and output values ​​(for every two layers) is calculated and fed into the next layer as the input. ResNet repeats this residual calculation until the final layer (the output layer). This prevents problems such as vanishing gradients and gradient divergence that can arise with many layers, ensuring high recognition accuracy.

[0048] In this embodiment, ResNet-101 is used. Server 1 generates a recognizer with a neural network structure consisting of 101 layers of neurons. More specifically, server 1 constructs a recognizer using a CNN (Convolution Neural Network) model that includes layers of neurons that perform convolution operations. Server 1 uses this generated recognizer to identify individual products.

[0049] Specifically, the server 1 obtains (inputs) training data and generates a product group identifier 142 and a product identifier 143 respectively through the training data. The training data includes training images, information indicating the product group to which the registered product belongs, and the product itself. A product group is a division of products whose feature quantities are similar to a certain extent, such as series of products whose containers of various product brands are similar to each other. The server 1 obtains a training image marked with the correct value of the product group (for example, the name of the product group. hereinafter referred to as the "series name"). The server 1 generates the product group identifier 142 by extracting feature quantities such as the color and shape of the product container, or the text of the series name indicated on the product container from the training image.

[0050] Furthermore, the server 1 generates a product identifier 143 for identifying individual products by using training data in which information identifying each product (e.g., product name) is labeled on training images. In this embodiment, the server 1 generates product identifiers 143a, 143b, 143c, etc. that are specific to each product group. By learning the characteristics of each product for groups of similar-looking products and using a separate product identifier 143 for each product group, image recognition processing is not complicated and accuracy can be improved.

[0051] Server 1 uses the aforementioned product group identifier 142 and product identifier 143 to identify products. First, server 1 inputs a captured image of a logo detected by the logo identifier 141 into product group identifier 142 to identify the product group. For example, server 1 extracts features from the captured image, such as the shape and color of the product container or the text on the container, to identify which product group the container is similar to or which product group series name it carries. This allows server 1 to determine which product group the product in the captured image belongs to.

[0052] After determining the commodity group, the server 1 uses the commodity identifier 143 corresponding to the commodity group to determine which commodity the photographed commodity belongs to. Figure 5As shown schematically below, when server 1 determines that the product series is "Series A" using product group identifier 142, it selects product identifier 143a, which is used to identify products in "Series A," as the identifier for identifying products. Server 1 inputs image data to product identifier 143a, extracts features such as the color and shape of the product container, or text on the container, and identifies the captured product as "Product A1." In this way, server 1 uses different product identifiers 143 to identify individual products, depending on the product series (product group).

[0053] Figure 6 This is an explanatory diagram showing an example of a product confirmation screen. The server 1 outputs the product confirmation result, and the terminal 2 displays Figure 6 The product confirmation screen shown. Figure 6 As shown, the terminal 2 displays the detection result of the mark (logo), the series name and the product name of the product together with the sample image of the determined product. Figure 6 The numbers attached to the series and product names are the reliability calculated during image recognition.

[0054] also, Figure 6 Although both the series name and product name are displayed, server 1 may fail to identify a single product and thus be unable to determine the product name, for example, if the reliability of image recognition falls below a specified threshold. In this case, server 1 displays only the series name on terminal 2. The product group identifier 142 identifies product groups, which are visually similar product containers, collectively, achieving higher accuracy than identifying individual products. By providing the user with a minimal indication of the product group identification result (series name), even if identification of a single product fails, the user can use the series name as a clue to obtain product information.

[0055] Furthermore, if the reliability in commodity group identification is low and the commodity group identification also fails, the server 1 outputs only the mark detection result detected from the captured image to the terminal 2 and ends a series of image recognition processes.

[0056] Figure 7 This is a flowchart showing an example of the learning process sequence of the learning model. Figure 7 The processing contents of the machine learning process executed by the server 1 will be described.

[0057] The control unit 11 of the server 1 obtains training data, which includes a training image obtained by photographing a product container of a product to be learned, and information indicating a label attached to the product container, the product group to which the product belongs, and each product (step S11). A label is a so-called mark composed of, for example, text, graphics, and symbols. A product group is a division of products whose feature quantities are similar to a certain extent, such as a series of product brands. The control unit 11 obtains training data for training in which the correct values ​​of the label, product group, and product itself are marked on the product image. For example, the control unit 11 obtains training data in which information such as the coordinate position of the label attached to the product container, the series name of the product group to which the product belongs, and the product name of each product are marked.

[0058] The control unit 11 performs machine learning to extract features from the training images, and generates identifiers for identifying a label, a product group, and the product itself based on the captured image (step S12). For example, the control unit 11 extracts local features (HOG features) related to brightness gradients at the coordinate positions represented by the training data, maps the extracted local features to a feature space, and finds a recognition boundary. This generates a label identifier 141 capable of detecting (classifying) the label attached to the target product based on the local features. Furthermore, the control unit 11 extracts features such as the color and shape of the product container or the text attached to the product, and associates them with information representing the product group and the product itself (e.g., a series name and a product name). This generates a product group identifier 142 and a product identifier 143 that identify the product group and the product itself based on the features of the image. Specifically, the control unit 11 generates identifiers having a ResNet network structure in which residual calculations between input and output values ​​are repeatedly performed for each predetermined number of neuron layers, and a CNN model including a neuron layer that performs a convolution operation. The control unit 11 generates commodity identifiers 143a, 143b, 143c, ... that are different for each commodity group. The control unit 11 ends the series of processing.

[0059] Figure 8 This is a flowchart showing an example of the processing sequence of the commodity determination process. Figure 8 The content of the product identification process for identifying a product based on the above-described learning model will be described.

[0060] The control unit 11 of the server 1 obtains an image of a product container captured by a user from the terminal 2 (step S31). Using the label recognition unit 141, the control unit 11 detects labels attached to the product container within the captured image (step S32). Specifically, the control unit 11 segments the captured image into predetermined local regions and extracts local features (HOG features) representing the brightness gradient of each local region. The control unit 11 detects pre-registered product labels by determining which space in the feature space, defined by the recognition boundary, the extracted local features fall within.

[0061] The control unit 11 determines whether the logo detection has failed (step S33 ). If the logo detection has failed ( S33 : YES), the control unit 11 outputs a notice of logo detection failure to the terminal 2 (step S34 ) and terminates the series of processing.

[0062] If the logo detection does not fail (S33: NO), the control unit 11 uses the product group identifier 142 (first neural network) to determine to which of a plurality of predefined product groups the imaged product belongs (step S35). Product groups are, for example, a division of products with somewhat similar feature values, such as serialized brand products. The control unit 11 uses the product group identifier 142, which is a neural network using ResNet, to determine to which of the plurality of product groups the imaged product belongs.

[0063] The control unit 11 determines whether product group identification has failed (step S36). For example, the control unit 11 determines whether the reliability calculated during image recognition is below a predetermined threshold. If product group identification has failed (S36: YES), the control unit 11 outputs the label detection result from step S32 to the terminal 2 (step S37), thus concluding the series of processing steps.

[0064] If product group identification has not failed (S36: NO), the control unit 11 uses the product identifier 143 (second neural network) specific to the identified product group to identify individual products (step S38). The product identifier 143 is a ResNet-based neural network that has learned the feature values ​​of individual products for each of the multiple product groups. The control unit 11 inputs the captured image to the neural network specific to the product group identified in step S35 to identify each product.

[0065] The control unit 11 determines whether the determination of a single product in step S35 has failed (step S39). For example, the control unit 11 determines whether the reliability of the image recognition is below a predetermined threshold. If the product determination has not failed (S39: NO), the control unit 11 generates a product determination screen showing the determined product and product group, and outputs it to the terminal 2 (step S40). Specifically, if Figure 6 As shown, the control unit 11 outputs the product name and series name to the product confirmation screen with the reliability indicated. If product confirmation fails (S39: YES), the control unit 11 outputs the product confirmation screen showing only the product group to the terminal 2 (step S41). The control unit 11 ends the series of processing.

[0066] Furthermore, while the above description uses the SVM method to detect the logo, it is also possible to detect the logo using optical methods such as OCR (Optical Character Recognition). In other words, any detection algorithm is not limited to machine learning algorithms as long as the logo can be detected through image recognition.

[0067] Furthermore, although the HOG feature is used as the feature for detecting the mark, features such as SIFT and SURF may also be used.

[0068] In addition, in this embodiment, the processing is performed by the cloud server 1 rather than the user terminal 2, but the data of the learning model can also be installed in the terminal 2 and the terminal 2 can perform a series of determination processes.

[0069] Based on the above, according to this embodiment, server 1 first identifies the labels attached to products based on captured images, and then identifies individual products. Since most products have labels, by filtering processing targets based on these labels, it is possible to accurately determine whether the object identified in the captured image is the intended product. Furthermore, server 1 identifies individual products after identifying product groups. This method, by roughly categorizing products and then identifying them individually, improves accuracy while minimizing the complexity of learning content. In summary, it is possible to accurately identify products based on captured images.

[0070] Furthermore, according to this embodiment, server 1 extracts local features representing brightness gradients (HOG features) from captured images and uses the Support Vector Machine (SVM) method to detect logos in feature space. For example, using OCR technology can hinder proper recognition due to factors such as the container surface material (concave, convex, or mirrored) or the logo's design. However, by extracting local features and performing pattern recognition as in this embodiment, logos can be detected with high accuracy.

[0071] Furthermore, according to the present embodiment, by displaying a commodity identification screen showing commodities and commodity groups identified based on a captured image on the terminal 2 , the user can easily obtain commodity information and can receive appropriate service support.

[0072] Furthermore, according to the present embodiment, even when identification of a specific product fails, a product group (eg, product brand) can be presented to the user, enabling more appropriate service support.

[0073] (Variation)

[0074] In the above embodiment, the logo on the product container is recognized as a mark and detected by the mark recognizer 141. In a modified example, the following method is described: instead of the logo, the characteristic shape of the product is recognized as the mark, and the mark recognizer 141 detects whether it has the learned product shape.

[0075] Figure 9 It is an explanatory diagram showing an outline of a modified example. Figure 9 In FIG, the container shape of the product photographed is shown ( Figure 9 (illustrated with a bold line in the figure) to determine whether the product is a registered product of our company.

[0076] In this modified example, server 1 performs a learning process to pre-learn product container shapes using training images, and then generates a badge recognizer 141 that detects (recognizes) specific product container shapes from captured images. For example, server 1 generates badge recognizers 141a, 141b, ... for each product group or individual product.

[0077] The server 1 inputs the captured image obtained from the terminal 2 to each logo recognizer 141, and determines whether the container of the captured product has the shape learned as a logo. In this way, the server 1 determines whether the product has the company's logo. For example, Figure 9 As shown, when a photographed image of our company's product A is input to the label recognition unit 141, the shape of the container of product A is detected from the image and the product is determined to have a shape that matches the label. As a result, the server 1 inputs the image to the product group recognition unit 142, and identifies each product in the same manner as in the above embodiment. On the other hand, Figure 9 As shown, when a captured image of a product D, which is a product of another company, is input, none of the logo recognizers 141 can detect a product shape that matches a logo, and therefore the image is excluded from processing.

[0078] As mentioned above, a mark is not limited to words, graphics, symbols, or other signs, but can also be the shape of the product itself.

[0079] Furthermore, although not specifically stated, the shape of the product that the mark recognizer 141 detects as a mark can be not only the entire shape of the product, but also a partial shape. For example, in the above example, if the lid of the container has a distinctive shape, the presence of a mark can be determined based on the shape of the lid. Thus, the mark that the mark recognizer 141 detects can be the entire shape of the product, or a partial shape.

[0080] Figure 10This is a flowchart showing an example of the processing sequence of the product identification processing of the modified example. After obtaining the image captured of the product container (step S31), the control unit 11 of the server 1 performs the following processing. The control unit 11 uses the label recognition device 141 that has learned the shape of the product container to detect the product shape that matches the label from the captured image (step S201). The control unit 11 determines whether the detection of the product shape that matches the label has failed (step S202). If the detection of the shape that matches the label has failed (S202: YES), the control unit 11 jumps the processing to step S34. If the detection of the shape that matches the label has not failed (S202: NO), the control unit 11 jumps the processing to step S35.

[0081] Based on the above, when identifying products based on captured images, it is also possible to filter images to be processed based on the shapes of the products.

[0082] Figure 11 This is a functional block diagram illustrating the operation of the server 1 according to the above-described method. When the control unit 11 executes program P, the server 1 operates as follows. The acquisition unit 111 acquires a captured image. The logo detection unit 112 uses a logo detector to determine whether the captured image is a product with a corresponding logo. The product group identification unit 113 uses a first neural network trained on products with the logo to determine which of multiple product groups the captured image, in which the logo has been detected, belongs to. The product identification unit 114 uses a second neural network trained for each of the multiple product groups to identify a product.

[0083] The embodiments disclosed herein are illustrative in all respects and should not be construed as restrictive. The scope of the present invention is defined not by the foregoing description but by the claims, and all modifications within the meaning and scope equivalent to the claims are intended to be encompassed.

[0084] Description of Reference Numerals

[0085] 1 Server (product identification device)

[0086] 11 Control Unit

[0087] 12 Main storage unit

[0088] 13 Ministry of Communications

[0089] 14 Auxiliary storage unit

[0090] P program

[0091] 141 Logo Recognizer

[0092] 142 Product Group Identifier

[0093] 143 Product Identifier

[0094] 2 Terminal

Claims

1. A commodity identification device, characterized in that: include: an acquisition unit that acquires a captured image; a local feature value extraction unit that extracts a local feature value representing a brightness gradient of the captured image; a logo detection unit that uses a logo detector that has learned a recognition boundary for recognizing a target logo in a feature space, and determines whether the product is a product with the logo based on the extracted local feature amount; a product group identification unit for identifying, using a first neural network trained on products having the logo, which is used to identify the captured image in which the logo is detected, one of a plurality of product groups; and a product identifying unit that identifies a product using a second neural network learned for each of the plurality of product groups; The neural network is a ResNet neural network having the following network structure: the residual of the input value and the output value of each specified number of neuron layers is calculated, and the calculated residual is input as the input value to the next layer, and the residual calculation is repeated in the ResNet until the final layer.

2. The commodity identification device according to claim 1, characterized in that: The method includes an output unit for outputting a determination result after determining the product group and the product.

3. The commodity identification device according to claim 2, characterized in that: When the product identification unit fails to identify the product, the output unit outputs the identification result regarding the product group.

4. The commodity identification device according to claim 1, wherein: The emblem is in the shape of the product. The logo detection unit determines whether the product has the logo based on whether the shape of the product is detected from the captured image using the logo detector.

5. A recording medium storing a program for causing a computer to execute the following processing: Obtaining captured images; extracting a local feature quantity representing a brightness gradient of the captured image; using a logo detector that has learned a recognition boundary for recognizing a target logo in a feature space, and determining whether the product is a product having the logo based on the extracted local feature amount; determining, for the captured image in which the logo is detected, which of a plurality of product groups the image belongs to using a first neural network trained on products having the logo; determining a product using a second neural network learned for each of the plurality of product groups; The neural network is a ResNet neural network having the following network structure: the residual of the input value and the output value of each specified number of neuron layers is calculated, and the calculated residual is input as the input value to the next layer, and the residual calculation is repeated in the ResNet until the final layer.

6. A learning method, characterized in that Make the computer execute the following processing: obtaining training data, the training data including training images of products, and information indicating the products, the product group to which the products belong, and labels possessed by the products; generating identifiers for identifying the product, product group, and logo based on the training data; The product group identifier and the product identifier are neural networks involving ResNet, which has the following network structure: the residual between the input value and the output value of each predetermined number of neuron layers is calculated, and the calculated residual is input as the input value to the next layer, and the residual calculation is repeated in the ResNet until the final layer; Among them, the product group identifier is a first neural network that has learned products with the label, the product identifier is a second neural network that has been learned for each of the multiple product groups, and the label identifier is an identifier that has learned to recognize the recognition boundary of the label in the feature space, and the label is detected based on the local feature value representing the brightness gradient of the image.

Images