Item recognition trained model generating method and item recognition trained model generating device

The trained model generation method addresses the challenge of maintaining recognition accuracy for packaged products with minor packaging changes by dividing the deep learning network and using gaze area correction and relearning, ensuring efficient and accurate product recognition.

JP7589515B2Active Publication Date: 2024-11-26FUJI ELECTRIC CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2020190461
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-11-16
Publication Date
2024-11-26
Estimated Expiration
2040-11-16

AI Technical Summary

Technical Problem

Existing image recognition systems struggle to maintain high recognition rates for packaged products due to frequent minor packaging changes, leading to incorrect recognition and increased training data collection loads, and feature value distributions overlapping between classes.

Method used

A trained model generation method that divides the deep learning network into pre- and post-stages, incorporates an attention branch, and uses gaze area correction and relearning based on human knowledge to adapt to minor packaging changes.

Benefits of technology

Generates a trained model capable of robust product recognition despite minor packaging changes, reducing training load and maintaining high recognition accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589515000001
    Figure 0007589515000001
  • Figure 0007589515000002
    Figure 0007589515000002
  • Figure 0007589515000003
    Figure 0007589515000003
Patent Text Reader

Abstract

To make it possible to generate a trained model that can obtain a robust product recognition result for a minute change of a packaged product.SOLUTION: A trained model generation device for product recognition includes: a gaze area correction unit that receives input of an original product image of an original product to generate an original gaze area which is a gaze area of the original product image, corrects the original gaze area to a corrected gaze area that has undergone arbitrary correction based on human knowledge, and makes it re-learned; a gaze area database 15 that stores the corrected original gaze area; a common area calculation unit 23 that receives input of a new product image of a new product which is a product in the same class as the original product and has a similar package to generate a new gaze area which is a gaze area of the new product image, and calculates a common area MP20 of a new gaze area and the corrected gaze area stored in the gaze area database 15 and similar to the new gaze area; and a re-learning unit that re-learns a perception branch 3 and / or an attention branch 5 using the common area MP20.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method for generating a trained model for product recognition and a device for generating a trained model for product recognition, which are capable of generating a trained model that can obtain robust product recognition results against minute changes in packaged products. [Background technology]

[0002] In recent years, due to a decline in the working population, labor shortages have become serious even in the retail industry, including convenience stores and supermarkets, and there are high hopes for the use of image recognition AI to reduce labor and automate payment operations (cash register settlement), which are particularly workload-intensive.

[0003] Retail stores handle a wide variety of products, including packaged products such as snacks and instant noodles, and non-packaged products (daily deliveries) such as lunch boxes and bread. For this reason, a large-scale learning data set is generally created, and an automatic product recognition system is constructed using image recognition AI, using a well-known deep learning technology such as Convolutional Neural Network (CNN). Alternatively, there are also examples of constructing a system using a method in which packaged products are recognized by feature point matching, which is a conventional technology, and non-packaged products are recognized by CNN (see Non-Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-220116 A [Non-patent literature]

[0005] [Non-Patent Document 1] Masaru Kikuchi et al., “Multi-Object Recognition Technology for Recognizing All Retail Products,” NEC Technical Report, Vol. 72, No. 1 [Retrieved November 8, 2020], Internet<URL: https: / / jpn.nec.com / techrep / journal / g19 / n01 / 190118.html> [Non-Patent Document 2] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, “Attention Branch Network: Learning of Attention Mechanism for Visual Explanation”, Chubu University, CVPR_2019 [Non-Patent Document 3] Masahiro Mitsuhara, Hiroshi Fukui (Chubu Univ.), Yusuke Sakashita (Chubu Univ., NIDEK), Takanori Ogata (ABEJA), Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi (Chubu Univ.), “Incorporating human knowledge into deep neural networks via attention maps”, 22nd Symposium on Image Recognition and Understanding, 2019 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, packaged products, which account for approximately 80% of all products, frequently undergo minute changes in their packaging, and so with the general CNN method, products with similar packaging may be mistaken for other packaged products (classes), resulting in a decline in the AI ​​recognition rate. In addition, with a method that combines feature point matching and CNN, even minute changes to the packaging may make it impossible to recognize the product. For this reason, the general method is to acquire a large number of images of new packaged products and retrain the trained model.

[0007] However, when collecting a large number of new training images and performing additional training every time the package changes, there is an issue that the burden of collecting training images increases, and there is also an issue that the distribution of features extracted from the target class, i.e., the data range that the features of the target class can take, becomes larger, so that they may be mixed with the feature distributions of other classes. For example, it is expected that the package of class A will change to be closer to the package of a different class B, and the feature distributions will partially overlap.

[0008] The present invention has been made in consideration of the above, and aims to provide a method for generating a trained model for product recognition and a device for generating a trained model for product recognition that can generate a trained model that can obtain robust product recognition results even with minor changes in packaged products. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems and achieve the object, the present invention provides a method for generating a trained model for commodity recognition that automatically recognizes commodities in a store, by dividing a deep learning network into an upstream deep learning network and a downstream deep learning network, providing an attention branch that generates a gaze area based on a feature extraction image output from the upstream deep learning network, and regressing the gaze area of ​​the attention branch to the feature extraction image and using the feature extraction image weighted by the gaze area, the downstream deep learning network performs deep learning, and generates a trained model for commodity recognition that automatically recognizes commodities in a store by inputting an original commodity image of an original commodity and generating a gaze area of ​​the original commodity image. The method includes a gaze area correction step of generating an original gaze area, correcting the original gaze area to a modified gaze area that has been arbitrarily modified based on human knowledge, and performing re-learning; a storage step of saving the modified gaze area; a common area calculation step of inputting a new product image of a new product that is in the same class as the original product and has similar packaging to generate a new gaze area that is the gaze area of ​​the new product image, and calculating a common area between the new gaze area and the saved modified gaze area that is similar to the new gaze area of ​​a product in the same class as the new product; and a new product re-learning step of re-learning the subsequent deep learning network and / or attention branch using the common area.

[0010] In the present invention, the common area is calculated using a difference between the new gaze area and the corrected gaze area.

[0011] In addition, in the above invention, the present invention is characterized in that, when the degree of commonality between the new attention area and the modified attention area is equal to or greater than a predetermined threshold, the common area is used as the attention area to re-train the attention branch and update the trained model for product recognition.

[0012] In addition, in the above invention, the present invention is characterized in that, when the degree of commonality between the new gaze area and the modified gaze area is equal to or greater than a predetermined threshold, the common area is used as the gaze area to re-train the subsequent deep learning network and update the trained model for product recognition.

[0013] In addition, in the above invention, the present invention is characterized in that if the degree of commonality between the new attention area and the modified attention area is less than a predetermined threshold, the common area is not used as the attention area, and re-learning of the attention branch and / or the downstream deep learning network is not performed.

[0014] In addition, in the above invention, the present invention is characterized in that an interpolation is performed on a difference area other than the common area calculated in the common area calculation step to replace pixel values ​​with 0, and an image including the interpolated common area is used as a gaze area.

[0015] In the present invention, the common area and the different area are displayed on a display unit.

[0016] The present invention also provides a trained model generation device for commodity recognition that generates a trained model for commodity recognition that automatically recognizes commodities in a store by dividing a deep learning network into an upstream deep learning network and a downstream deep learning network, providing an attention branch that generates a gaze area based on a feature extraction image output from the upstream deep learning network, and regressing the gaze area of ​​the attention branch to the feature extraction image and using the feature extraction image weighted by the gaze area, and the downstream deep learning network performs deep learning. The trained model generation device for commodity recognition generates a trained model for commodity recognition that automatically recognizes commodities in a store by inputting an original commodity image of an original commodity and generating an original gaze area that is the gaze area of ​​the original commodity image, and The system is characterized by comprising a gaze area correction unit that corrects the gaze area to a corrected gaze area that has been arbitrarily corrected based on human knowledge and performs re-learning; a gaze area database that stores the corrected gaze area; a common area calculation unit that inputs a new product image of a new product that is in the same class as the original product and has similar packaging to generate a new gaze area that is the gaze area of ​​the new product image and calculates a common area between the new gaze area and the corrected gaze area similar to the new gaze area of ​​a product in the same class as the new product stored in the gaze area database; and a re-learning unit that re-learns the subsequent deep learning network and / or attention branch using the common area. Effect of the Invention

[0017] According to the present invention, it is possible to generate a trained model that can obtain product recognition results that are robust against minute changes in packaged products. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a trained model for commodity recognition applied to a trained model generation device for commodity recognition according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a device for generating a trained model for commodity recognition. [Diagram 3] FIG. 3 is a flowchart showing a procedure of a product learning process performed by the device for generating a trained model for product recognition. [Figure 4] FIG. 4 is a diagram showing a specific example of an original product to be learned that is input in the first learning phase, and a new product to be learned that is input in the second learning phase. [Diagram 5] FIG. 5 is an explanatory diagram illustrating the learning process in the first learning phase. [Figure 6] FIG. 6 is an explanatory diagram illustrating the learning process in the second learning phase. [Figure 7] FIG. 7 is a flowchart showing a procedure of a product learning process performed by the device for generating a trained model for product recognition according to this modified example. [Figure 8] FIG. 8 is an explanatory diagram illustrating the learning process in the second learning phase according to this modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0020] <Configuration of trained model for product recognition> FIG. 1 is a diagram showing a schematic configuration of a trained model M for commodity recognition applied to a trained model generating device for commodity recognition according to an embodiment of the present invention. The trained model M for commodity recognition may be a default model before learning, or may be a trained model. As shown in FIG. 1, the trained model M for commodity recognition uses ABN (Attention Branch Network: see Non-Patent Document 2), which is one of deep learning techniques. The ABN divides a deep learning network 1, which is a CNN, into a front-stage deep learning network 2 and a rear-stage deep learning network (perception branch) 3, provides an attention branch 5 that generates an attention map MP based on a feature extraction image D1 output from the front-stage deep learning network 2, and regresses the attention map MP of the attention branch 5 to the feature extraction image D1, and outputs the feature extraction image weighted by the multiplication unit 4 to the perception branch 3 to perform deep learning.

[0021] The attention branch 5 performs convolution operations and applies activation functions to the feature extraction image D1 (the output of the final convolution layer with K channels), identifies pixels with high response values ​​to the output, i.e., pixels with high influence, and generates a gaze area MP that is combined as one piece of data. By weighting the original feature extraction image D1 by multiplying this gaze area MP, the perception branch 3 generates a feature extraction image that emphasizes the channel that should be noted, and can focus on learning the area that should be noted. This allows for highly accurate product judgment.

[0022] 1, product images of three classes A, B, and C are input as training data, and the decision output unit 6 outputs a decision based on the probability of each class output by the perception branch 3. The decision output unit 6 may also output a decision using the probability output by the attention branch 5.

[0023] <Configuration of a trained model generation device for product recognition> 2 is a block diagram showing the configuration of the commodity recognition trained model generation device 10. As shown in FIG. 2, the commodity recognition trained model generation device 10 includes an image acquisition unit 11, an input unit 12, a display unit 13, a storage unit 14, a gaze area database 15, and a control unit 16.

[0024] The image acquisition unit 11 is a device for acquiring product images to be classified and is, for example, an imaging device such as a camera. The input unit 12 is an input device such as a keyboard or a mouse, and inputs various information. The display unit 13 is a display device such as a liquid crystal panel or a display device.

[0025] The storage unit 14 is a secondary storage medium such as a hard disk drive or an SSD, and stores the commodity recognition trained model M. The gaze area database 15 is a database that stores a corrected gaze area corrected by a gaze area correction unit 22 described later.

[0026] The control unit 16 is a control unit that controls the entire commodity recognition trained model generation device 10, and has a learning unit 21, a gaze area correction unit 22, a common area calculation unit 23, a relearning unit 24, an interpolation unit 25, and a display processing unit 26. In practice, the control unit 16 stores programs corresponding to these functional units in a storage device such as a non-volatile memory, and loads these programs into the memory and executes them on the CPU to execute the corresponding processes.

[0027] The control unit 16 transitions to the second learning phase after the first learning phase. In the first learning phase, learning is performed for the original product, a corrected gaze area is generated by correcting the gaze area (original gaze area) of the original product based on human knowledge, and stored in the gaze area database 15, and re-learning is performed by backpropagation using the corrected gaze area. Meanwhile, in the second learning phase, learning is performed for a new product that is the same as the original product and has a similar package, and at this time, a common area between the new gaze area of ​​the new product and the similar saved corrected gaze area is calculated, and re-learning is performed by backpropagation using this common area.

[0028] In the first learning phase and the second learning phase, the learning unit 21 performs learning on the input product images by forward propagation.

[0029] In the first learning phase, the gaze area correction unit 22 inputs an original product image of an original product to generate an original gaze area that is the gaze area of ​​the original product image, inputs an original product image of the original product (product image of products A, B, and C) to generate an original gaze area that is the gaze area of ​​the original product image, corrects the original gaze area to a corrected gaze area that has been arbitrarily corrected based on human knowledge, and performs re-learning (see Non-Patent Document 3). By correcting the original gaze area, a corrected gaze area is generated in which background and noise are excluded and only characteristic areas used for human judgment remain. This corrected gaze area is stored in the gaze area database 15. Note that the gaze area correction unit 22 is provided with a HITL (Human-in-the-loop) mechanism.

[0030] In the second learning phase, the common area calculation unit 23 inputs a new product image of a new product that is the same product as the original product but has similar packaging, generates a new gaze area that is the gaze area of ​​the new product image, and calculates a common area between the new gaze area and a modified gaze area stored in the gaze area database 15 in the first learning phase that is similar to the new gaze area of ​​a product that is the same product as the new product.

[0031] The re-learning unit 24 performs re-learning by weighting using the corrected attention area in the first learning phase, and performs re-learning by weighting using the common area in the second learning phase.

[0032] In the second learning phase, the interpolation unit 25 performs interpolation to replace pixel values ​​of the difference area other than the calculated common area with 0. The re-learning unit 24 performs re-learning using the attention area having only this interpolated common area.

[0033] The display processing unit 26 performs processing to display the judgment output from the commodity recognition trained model M on the display unit 13. Note that the display processing unit 26 may display the common area and the difference area on the display unit 13, as described later.

[0034] <Learning process procedure> Fig. 3 is a flowchart showing a commodity learning process procedure by the commodity recognition trained model generating device 10. As shown in Fig. 3, in the first learning phase, first, the learning unit 21 learns by inputting an original commodity image of an original commodity to the commodity recognition trained model M (step S110). During this learning, a gaze area MP is generated, the gaze area correction unit 22 corrects the gaze area MP, and the corrected gaze area is multiplied by the feature extraction image D1 to perform re-learning (step S120). Then, this corrected gaze area is stored in the gaze area database 15 (step S130). As a result, a commodity recognition trained model M for the original commodity in the first learning phase is constructed (step S140).

[0035] Then, in the second learning phase, a new product image of a new product similar to the original product, such as when only the package has changed slightly, is input to the trained product recognition model M for learning (step S210). Then, the common area calculation unit 23 extracts a modified gaze area similar to the generated new gaze area from the gaze area database 15 (step S220). Then, a common area common to the new gaze area and the extracted modified gaze area is calculated (step S230). At this time, it is preferable to set the gaze area to a pixel value of a difference area other than the common area, which is 0. Then, the re-learning unit 24 re-learns the new product using the common area (step S240). As a result, the trained product recognition model M for the new product in the second learning phase is constructed (step S140).

[0036] <First and second learning phases> 4 is a diagram showing specific examples of original product A (class A), original product B (class B), and original product C (class C) to be learned and input in the first learning phase, and new product A' (class A), new product B' (class B), and new product C' (class C) to be learned and input in the second learning phase. Class A is a PET bottle drink product, class B is a canned drink product, and class C is a packaged product. New product A' in class A, new product B' in class B, and new product C' in class C each have marks A1 to A3 drawn on them, and their packages are slightly changed from the packages of original products A to C.

[0037] FIG. 5 is an explanatory diagram for explaining the learning process in the first learning phase. As shown in FIG. 5, first, a product image of the original product A of class A is input to the trained model M for product recognition. The feature extraction image D1 output from the final layer of the front-stage deep learning network 2 is input to the attention branch 5 to generate a gaze region MP. The generated gaze region MP is multiplied by the feature extraction image D1 in the multiplication unit 4. The feature extraction image D1 weighted by the gaze region MP is input to the perception branch 3. The perception branch further performs a convolution operation and outputs the classification probability of each product to the judgment output unit 6 via the fully connected layer. This allows learning by forward propagation. The classification probability output by the attention branch 5 may be output to the judgment output unit 6, and the judgment output unit 6 may perform a classification judgment of the product by referring to this classification probability as well.

[0038] The gaze area MP generated during this learning is modified by the gaze area modification unit 22 based on human knowledge to generate a modified gaze area MP10. The generated modified gaze area MP10 is replaced with the original gaze area MP, and the attention branch 5 and / or the perception branch 3 are re-learned by backpropagation using this replaced modified gaze area MP10. Meanwhile, the modified gaze area MP10 is stored in the gaze area database 15. The re-learning for the attention branch 5 and the perception branch is fine-tuned, for example, so that the error Latt(x) between the probability output by the attention branch 5 and the judgment result, the error Lper(x) between the probability output by the perception branch 3 and the judgment result, and the error Lmap(x) between the gaze area MP and the modified gaze area MP10 are small. In particular, since the error Lmap(x) is larger than the error Latt(x) and the error Lper(x), it is preferable to fine-tune the error Lmap(x). The same learning process is also performed on original products B and C of classes B and C.

[0039] In this first learning phase, the ability to arbitrarily modify the gaze area MP makes it possible to learn only the areas that are characteristic of the product (areas used by humans for judgment, areas excluding background and noise), enabling more accurate and efficient learning than normal learning.

[0040] Next, Fig. 6 is an explanatory diagram for explaining the learning process in the second learning phase. As shown in Fig. 6, first, a product image of a new product A' of class A is input to a trained product recognition model M. A feature-extracted image D1 output from the final layer of the front-stage deep learning network 2 is input to an attention branch 5 to generate a new gaze area MP'.

[0041] The common area calculation unit 23 extracts the modified gaze area MP10 that has the same label (same class) as the new product A and is most similar (the extracted feature vector value is closest) from the gaze area database 15. Then, a common area MP20 is calculated by excluding the difference area MP30 using the difference between the new gaze area MP' for the new product image and the extracted modified gaze area MP10. Then, the re-learning unit 24 re-learns the attention branch 5 and / or the perception branch 3 using only the common area MP20. Note that the re-learning unit 24 may perform re-learning using a gaze area in which the difference area MP30 has been interpolated by the interpolation unit 25. Note that a similar learning process is also performed on new products B' and C' of classes B and C.

[0042] In this second learning phase, the modified gaze area MP10 that is most similar to the new gaze area MP' of the new product image is extracted from the gaze area database 15, and a common area MP20 is calculated using the difference between the new gaze area MP' and the modified gaze area MP10. Using only this common area MP20, it is possible to focus on learning common features (color and pattern), enabling robust product recognition based on invariant features even when slight changes occur in the packaging.

[0043] <Modification> In the above embodiment, in the second learning phase, the attention branch 5 and / or the perception branch 3 are re-learned using the common region MP20. Here, if the commonality of the common region MP20 is very small, i.e., below a predetermined threshold, it is assumed that re-learning will not be effective. Therefore, in this modified example, re-learning using the common region MP20 is not performed. This makes it possible to prevent unnecessary learning and makes it easier for learning to converge.

[0044] Fig. 7 is a flowchart showing a procedure of a commodity learning process performed by the commodity recognition trained model generation device 10 according to this modification. As shown in Fig. 7, in this modification, steps S231 and S232 are added after step S230 in the flowchart shown in Fig. 3. Therefore, steps S110 to S230, S240 and S250 are the same as those shown in Fig. 3.

[0045] In step S230, the common area MP20 is calculated and the different area MP30 is calculated concomitantly. In step S231, the display processing unit 26 displays the common area MP20 and the different area MP30 on the display unit 13. At this time, the output result is also displayed.

[0046] In step S232, it is determined whether the commonality of the common region MP20 is equal to or greater than a predetermined threshold. If the commonality is equal to or greater than the predetermined threshold (step S232: Yes), re-learning is performed on the attention branch 5 and / or the perception branch 3 for the new product using the common region MP20 (step S240).

[0047] On the other hand, if the commonality is not equal to or less than the predetermined threshold (step S232: No), the attention branch 5 and / or the perception branch 3 are not re-learned for the new product using the common region MP20, and the process proceeds to step S250.

[0048] Fig. 8 is an explanatory diagram for explaining the learning process in the second learning phase according to this modified example. As shown in Fig. 8, the common area calculation unit 23 calculates the common area MP20 and the difference area MP30, but performs re-learning by replacing the new gaze area MP' with the common area MP20 only when the commonality of the common area MP20 is equal to or greater than a predetermined threshold, and does not perform re-learning by replacing the new gaze area MP' with the common area MP20 when the commonality is less than the predetermined threshold.

[0049] In addition to the output result, the display unit 13 displays the common area MP20 and the difference area MP30. This allows the user to visually confirm what the trained model M for commodity recognition mainly learned and what part of the commodity has changed.

[0050] In addition, the trained model M for commodity recognition generated in the above embodiment and modified example is downloaded as appropriate by stores and the like that use the trained model M for commodity recognition.

[0051] Furthermore, the programs executed by the devices of the above-mentioned embodiments and modifications are provided by being pre-installed in a storage medium (ROM or storage unit) included in the device, but this is not limited, and the programs may be provided by being recorded in an installable or executable format on a computer-readable storage medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). Furthermore, the storage medium is not limited to a medium independent of the computer or embedded system, but also includes a storage medium that stores or temporarily stores a program downloaded via a LAN, the Internet, or the like.

[0052] In addition, the programs executed by the devices of the above embodiments and variations may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network, or may be provided or distributed via a network such as the Internet.

[0053] Note that the configurations illustrated in the above embodiments and modifications are merely functional schematics and do not necessarily have to be physically configured as illustrated. In other words, the distribution and integration of each device and component is not limited to that illustrated, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various usage conditions, etc. [Explanation of symbols]

[0054] 1 Deep Learning Networks 2. Pre-stage deep learning network 3. Perception Branch 4. Multiplication section 5. Attention Branch 6 Judgment output section 10. Product recognition trained model generation device 11 Image acquisition section 12 Input section 13 Display section 14 Storage section 15 Gaze Area Database 16 Control section 21 Learning Department 22 Gaze area correction part 23 Common area calculation section 24 Re-learning section 25 Interpolation section 26 Display processing section A1~A3 marks D1 Feature extraction image Latt error Lmap error Lper error M Trained model for product recognition MP gaze area MP´ New gaze area MP10 modified gaze area MP20 common area MP30 difference area

Claims

1. A method for generating a trained model for commodity recognition, comprising: dividing a deep learning network into an upstream deep learning network and a downstream deep learning network; providing an attention branch that generates a gaze area based on a feature extraction image output from the upstream deep learning network; and causing the downstream deep learning network to perform deep learning using the feature extraction image weighted by the gaze area by regressing the gaze area of ​​the attention branch to the feature extraction image, thereby generating a trained model for commodity recognition that automatically recognizes commodities in a store, A gaze area correction step of inputting an original product image of an original product to generate an original gaze area which is a gaze area of ​​the original product image, and correcting the original gaze area into a corrected gaze area which has been arbitrarily corrected based on human knowledge, and performing re-learning; A storage step of storing the modified gaze area; A common area calculation step of inputting a new product image of a new product that is in the same class as the original product and has a similar package to the new product image, generating a new gaze area that is a gaze area of ​​the new product image, and calculating a common area between the new gaze area and the saved corrected gaze area that is similar to the new gaze area of ​​the product in the same class as the new product; A new product re-learning step of re-learning the subsequent deep learning network and / or attention branch using the common region; A method for generating a trained model for product recognition, comprising:

2. The method for generating a trained model for commodity recognition according to claim 1 , wherein the common area is calculated using a difference between the new gaze area and the corrected gaze area.

3. The method for generating a trained model for product recognition described in claim 1 or 2, characterized in that if the degree of commonality between the new attention area and the modified attention area is equal to or greater than a predetermined threshold, the common area is used as the attention area to re-train the attention branch and update the trained model for product recognition.

4. The method for generating a trained model for commodity recognition described in any one of claims 1 to 3, characterized in that, when the degree of commonality between the new gaze area and the modified gaze area is equal to or greater than a predetermined threshold, the common area is used as the gaze area to re-train the subsequent deep learning network and update the trained model for commodity recognition.

5. A method for generating a trained model for product recognition described in any one of claims 1 to 4, characterized in that if the degree of commonality between the new attention area and the modified attention area is less than a predetermined threshold, the common area is not used as the attention area, and re-learning of the attention branch and / or the subsequent deep learning network is not performed.

6. A method for generating a trained model for commodity recognition according to any one of claims 1 to 4, characterized in that an interpolation is performed on a difference area other than the common area calculated in the common area calculation step, in which pixel values ​​are replaced with 0, and an image including the interpolated common area is used as a fixation area.

7. The method for generating a trained model for product recognition according to claim 6, characterized in that the common area and the difference area are displayed on a display unit.

8. A device for generating a trained model for commodity recognition, which divides a deep learning network into an upstream deep learning network and a downstream deep learning network, provides an attention branch that generates a gaze area based on a feature extraction image output from the upstream deep learning network, and regresses the gaze area of ​​the attention branch to the feature extraction image, and uses the feature extraction image weighted by the gaze area to cause the downstream deep learning network to perform deep learning, thereby generating a trained model for commodity recognition that automatically recognizes commodities in a store, A gaze area correction unit that inputs an original product image of an original product to generate an original gaze area that is a gaze area of ​​the original product image, and corrects the original gaze area into a corrected gaze area that has been arbitrarily corrected based on human knowledge, thereby performing re-learning; a gaze area database for storing the modified gaze area; a common area calculation unit that inputs a new product image of a new product that is in the same class as the original product and has a similar package, generates a new gaze area that is a gaze area of ​​the new product image, and calculates a common area between the new gaze area and the modified gaze area similar to the new gaze area of ​​a product in the same class as the new product stored in the gaze area database; A re-learning unit that re-learns the subsequent deep learning network and / or attention branch using the common region; A product recognition trained model generation device comprising:

Citation Information

Patent Citations

  • Extraction device, extraction method, extraction program, and model

    JP2019049945A

  • Information processing device, determination method, and object determination program

    JP2019220116A