Image recognition device, image recognition method, and image recognition program

Description

Technical Field

[0001] The present disclosure relates to image recognition.

Background Art

[0002] In recent years, the introduction of AI systems has been promoted for business efficiency improvement, work automation, etc. AI means artificial intelligence. To build an AI system, a large number of labeled learning images paired with labels are required. However, since the labels are manually assigned, the labeling cost is high, and it is difficult to collect a large number of labeled learning images. Therefore, technologies have been developed to recognize images with high accuracy even with a small number of labeled learning images.

[0003] One of these technologies is the transfer learning technology for image recognition models. In transfer learning, using learning images of various objects different from the object to be image-recognized (learning images for the image recognition model before transfer learning), image feature amounts of various objects are extracted, and the image recognition model is pre-learned. Then, by further learning the pre-learned image recognition model using a small number of learning images of the object to be image-recognized (learning images of the new task used at the time of transfer learning), the image recognition model is transferred from the pre-learned image recognition model to an image recognition model that recognizes the object to be image-recognized. Thus, by performing transfer learning on the pre-learned image recognition model, it is possible to learn the desired object even with a small number of learning images. However, when there is a large difference between the image feature amounts learned by the image recognition model before transfer learning and the image feature amounts of the object to be image-recognized, transfer learning does not proceed if the number of learning images of the object to be image-recognized is small. Then, it becomes difficult to transfer to an image recognition model that recognizes the object to be image-recognized, and the image recognition accuracy decreases. Here, the difference between the image feature amounts learned by the image recognition model before transfer learning and the image feature amounts of the object to be image-recognized is expressed as the bias of the learning images.

[0004] Patent Document 1 discloses the following machine learning method for transfer learning in order to reduce bias in training images. Record multiple pre-trained models before transfer learning. Evaluate the fit between the transfer learning dataset and the pre-trained models, select the pre-trained model with the highest evaluation result, and perform transfer learning. However, the technology disclosed in Patent Document 1 requires the preparation of multiple pre-trained models. Furthermore, if the bias of the training images is large in all pre-trained models, there is a problem in that the bias of the training images cannot be reduced regardless of which pre-trained model is selected. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-104067 [Non-patent literature]

[0006] [Non-Patent Document 1] Zhongqi Yue,et al.”Interventional few shot learning”,arXiv:2009.13000v2[cs.LG] 4 Dec 2020 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] This disclosure aims to reduce bias in training images used to create image recognition models, thereby enabling higher accuracy in classifying and estimating the location of novel objects. [Means for solving the problem]

[0008] The image recognition device disclosed herein is A probability estimation unit estimates the probability that the target object captured in the target image fits into each of several existing object classes, and An additional feature amount calculation unit that calculates an additional feature amount using the estimated probability of the target object with respect to each of the plurality of existing classes and the matching feature amount calculated as a feature amount of an existing class image in which an existing class object matching each of the plurality of existing classes appears; A feature amount combining unit that combines the additional feature amount with the target feature amount, which is a feature amount of the target image, to calculate a combined feature amount; An object detection unit that estimates a matching probability that the target object matches a new class that is an object class different from the plurality of existing classes and the position of the target object in the target image based on the combined feature amount; Comprising.

Advantages of the Invention

[0009] According to the present disclosure, it is possible to reduce the bias of the learning images used for creating the image recognition model and estimate the classification and position of a new object with higher accuracy.

Brief Description of the Drawings

Modes for Carrying Out the Invention

[0012] Embodiment 1. The image recognition device 100 will be described based on Figures 1 to 7.

[0013] ***Explanation of the structure*** The configuration of the image recognition device 100 will be explained based on Figure 1. The image recognition device 100 is a computer equipped with hardware such as a processor 101, memory 102, auxiliary storage device 103, and input / output interface 104. These hardware components are connected to each other via signal lines.

[0014] The processor 101 is an IC that performs arithmetic operations and controls other hardware. For example, the processor 101 is a CPU, DSP, or GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0015] Memory 102 is a volatile or non-volatile storage device. Memory 102 is also called main memory. For example, memory 102 is RAM. Data stored in memory 102 is saved to auxiliary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0016] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, HDD, flash memory, or a combination thereof. Data stored in the auxiliary storage device 103 is loaded into memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0017] The input / output interface 104 is a port to which input and output devices are connected. For example, the input / output interface 104 is a USB terminal, the input devices are a keyboard and mouse, and the output device is a display. Input and output of the image recognition device 100 are performed using the input / output interface 104. USB is an abbreviation for Universal Serial Bus.

[0018] The image recognition device 100 includes an inference unit 110. The inference unit 110 is implemented in software.

[0019] The auxiliary storage device 103 stores an image recognition program that enables the computer to function as an inference unit 110. The image recognition program is loaded into memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores the operating system. At least a portion of the OS is loaded into memory 102 and executed by the processor 101. Processor 101 runs the image recognition program while simultaneously executing the operating system. OS is an abbreviation for Operating System.

[0020] The image recognition program data (input data, output data, etc.) is stored in the storage unit 190. The auxiliary storage device 103 functions as a storage unit 190. However, storage devices such as memory 102, registers in the processor 101, and cache memory in the processor 101 may function as a storage unit 190 instead of the auxiliary storage device 103, or together with the auxiliary storage device 103.

[0021] Image recognition programs can be recorded (stored) in a computer-readable format on non-volatile recording media such as optical discs or flash memory.

[0022] The configuration of the inference unit 110 will be explained based on Figure 2. The inference unit 110 includes a probability estimation unit 111, an additional feature calculation unit 112, a target feature calculation unit 113, a feature merging unit 114, and an object detection unit 115. The target image 191, the fitted feature database 192, and the inference result 199 are stored in the memory unit 190. DB stands for database.

[0023] The configuration of the image recognition model 200 will be explained based on Figure 3. Image recognition model 200 is a pre-trained model that estimates the probability (classification result) that an object in an input image fits into each of several object classes, and the position of the object in the input image (object position). The object position is represented by a bounding box. The image recognition model 200 is stored in the memory unit 190 and used by the image recognition device 100. For example, image recognition model 200 is composed of a CNN model. CNN is an abbreviation for Convolutional Neural Network.

[0024] The image recognition model 200 comprises an input / hidden layer 210 and an output layer 220. The input / hidden layer 210 calculates the features of the input image. The input / hidden layer corresponds to the backbone and is the basic network part for extracting features from the input image. The output layer 220 estimates the classification result and object position based on the features of the input image. The output layer corresponds to the Head and is the network portion for estimating the type and position of objects in the image.

[0025] The image recognition model 200 before transfer learning is performed for the new object class (new class) is referred to as the pre-transfer learning model 201. The pre-transfer learning model 201 estimates the classification results and object positions for multiple existing object classes (existing classes). The image recognition model 200 after transfer learning for the new class is referred to as the post-transfer-learned model 202. After transfer learning, model 202 estimates class classification results and object locations for new classes and multiple existing classes.

[0026] The formulas shown within the dashed box represent the combination of image features resulting from the intervention process. Image features are represented in vector form. Features represented in vector form are called feature vectors. By calculating this formula, the combined features described later are obtained.

[0027] Figure 4 shows the relationship between the image recognition model 200 and the combined features. A combined feature is created by combining a second feature with a first feature x. The first feature x is a feature of the input image, and is calculated using the input image as input and the hidden layer 210. The second feature is calculated by multiplying the estimated probability by the fitted feature. "m" indicates the number of existing classes. The estimated probability is the probability that an object in the input image fits into each existing class, and it is estimated using the image recognition model 200 with the input image as input. Fitted features are the features of an image containing an object that fits an existing class, and are calculated using the input and hidden layer 210 with the image as input.

[0028] By using the combined features as input to the output layer 220, the bias in the training image set (pre-training dataset) used when creating the pre-transfer learning model 201 is reduced, improving the learning efficiency during transfer learning and the estimation accuracy after transfer learning.

[0029] ***Explanation of operation*** The operating procedure of the image recognition device 100 corresponds to the image recognition method. Furthermore, the operating procedure of the image recognition device 100 corresponds to the processing procedure of the image recognition program.

[0030] The image recognition method will be explained based on Figure 5. In step S101, the probability estimation unit 111 estimates the probability that the object shown in the target image 191 fits into each of several existing classes. The estimated probability is referred to as the estimated probability.

[0031] The object in question is the object shown in image 191.

[0032] The multiple existing classes are the multiple object classes that the pre-transfer learning model learned. These multiple objects are the objects included in the pre-training dataset.

[0033] The estimated probability is calculated as follows: The probability estimation unit 111 takes the target image 191 as input and uses the pre-transfer learning model 201 to estimate the estimated probability of the target object for each of several existing classes.

[0034] In step S102, the additional feature calculation unit 112 calculates additional features using the estimated probability of the target object for each of the multiple existing classes and the fitted features for each of the multiple existing classes.

[0035] The fitted features of each of the multiple existing classes are stored in the fitted feature DB192. The fitted features are obtained by averaging the feature vectors obtained by inputting the pre-trained dataset into the input and hidden layers of the image recognition model in Figure 5. i This is the result. Here, "ai " represents each existing class, and "Ca i " is each existing class a i This represents the features for which the data is obtained.

[0036] The fitted features of the existing class are the features of the existing class image. Existing class images are images that depict existing class objects. Specifically, existing class images are training images used in training to create the pre-transfer learning model 201. An existing class object is an object that fits into an existing class.

[0037] Fitted features are calculated using the input and hidden layer 210 of the pre-transfer learning model 201, with existing class images as input.

[0038] The additional features are calculated as follows: The additional feature calculation unit 112 inputs the target image 191 into the image recognition model 200 in Figure 5 and calculates the estimated probability (a) for each existing class. i Calculate the |x) and estimate the probability (a i |x) contains the fitted feature Ca of the existing class i The class-specific features are calculated by multiplying by . Then, the additional feature calculation unit 112 calculates the average of the multiple class-specific features corresponding to multiple existing classes. The calculated average becomes the additional feature.

[0039] In step S103, the target feature calculation unit 113 calculates the features of the target image 191. The calculated features are referred to as target features.

[0040] The target features are calculated as follows: The target feature calculation unit 113 takes the target image 191 as input and calculates the target features using the input and hidden layer 210 of the transfer-learned model 202.

[0041] In step S104, the feature merging unit 114 merges additional features with the target features. The resulting features are called merged features.

[0042] In step S105, the object detection unit 115 estimates the probability of the target object fitting to the new class based on the combined features. In other words, the object detection unit 115 estimates the probability that the target object fits into a new class.

[0043] The new class is an object class that represents a different type of object from several existing classes.

[0044] Furthermore, the object detection unit 115 estimates the position of the target object in the target image based on the combined features.

[0045] The object detection unit 115 performs estimation as follows: The object detection unit 115 uses the combined features as input and the output layer 220 of the transfer-learned model 202 to estimate the probability of the target object fitting to the new class and the position of the target object in the target image.

[0046] The object detection unit 115 then outputs the inference result 199. Inference result 199 shows the probability of fitting the target object to the new class and the position of the target object in the target image. For example, the object detection unit 115 displays the inference result 199 on the display.

[0047] ***Description of the Example*** The image recognition device 100 may calculate the similarity between the additional features and the target features. The configuration of the inference unit 110 will be explained based on Figure 6. The inference unit 110 further includes a feature similarity calculation unit 180. First, the feature similarity calculation unit 180 calculates the similarity between the additional features and the target features. The calculated similarity is referred to as the feature similarity score 181. Examples of feature similarity metric 181 include cosine similarity, Euclidean distance, and Pearson correlation coefficient. The feature similarity calculation unit 180 then outputs the feature similarity score 181. For example, the feature similarity calculation unit 180 displays the feature similarity score 181 on the display.

[0048] ***Effects of Embodiment 1*** Non-patent document 1 proposes a learning technique that uses causal inference to reduce the bias of the training images used in the image recognition model before transfer learning, thereby improving image recognition accuracy even with a small amount of training images for the new task. Based on Figure 7, the learning technique described in Non-Patent Document 1 will be explained. Figure 7 shows the causal graph during transfer learning in Non-Patent Document 1. "D" represents the pre-trained dataset for the existing task. "I" represents the low-label dataset for the new task. "X" represents the feature vector extracted by inputting the low-label dataset I for the new task into the pre-trained model. "C" represents the intermediate feature vector extracted by inputting the pre-trained dataset D and the low-label dataset I for the new task into the machine learning model. "Y" represents the inference result of the machine learning model. Here, we assume that the pre-trained dataset D is a confounding factor that has a causal relationship not only with the intermediate feature vector C but also with the feature vector X for the new task. This is because when extracting the feature vector X, we use the pre-trained model trained on the pre-trained dataset D. In an ideal low-label data learning technique, it is desirable to calculate the probability P(Y|X) between the feature vector X for a new task and the inference result Y of the machine learning model. However, if the pre-trained dataset D is a confounding factor, it will influence the feature vector X for the new task (see dashed arrow). This makes it difficult to calculate the probability P(Y|X). To prevent the influence of this confounding factor, a pseudo-intervention process is introduced, which involves combining the image features of each class before transfer learning. Then, by calculating the probability P(Y|do(X)) with the feature vector X fixed to an arbitrary value, the influence of the pre-trained dataset D on the feature vector X is reduced. In other words, in the learning technique described in Non-Patent Document 1, the bias of the training images before transfer learning, which acts as a confounding factor, affects the image features extracted from the training images for the new task. As a result, the learning efficiency of the image recognition model decreases. To mitigate bias in the training images before transfer learning, an intervention process is performed using the classification results of the image recognition model before transfer learning. In the intervention process, the probability that the training images for the new task fit the classification results of the image recognition model before transfer learning is calculated, and the image features of the training images before transfer learning are used according to this probability. This reduces the bias of the training images before transfer learning, mitigating the impact on the image features extracted from the training images for the new task, and improving learning efficiency. This type of learning technique makes it possible to accurately classify objects for a new task, even with a small amount of training images for that task. However, while Non-Patent Document 1 discloses a method for intervention processing to be applied to an image recognition model for classification, it does not disclose a method for intervention processing to be applied to an image recognition model for object detection. Furthermore, directly applying the intervention processing methods used for classification to an image recognition model for object detection would significantly reduce the accuracy of the image recognition for object detection.

[0049] Image datasets for object detection include bounding box information (information about the frame surrounding the object to be recognized). On the other hand, many image datasets used for classification lack bounding box information, and the image features for each class before transfer learning are extracted from the entire image area of ​​the image dataset. Therefore, when applying a method for extracting image features for classification to an image dataset for object detection, it ends up including many image features from the background of the image other than the object being recognized, making it impossible to accurately extract image features for each class before transfer learning. Furthermore, while the image recognition model for classification has a model structure where each layer outputs the classification result, the image recognition model for object detection has a model structure that not only outputs the classification result but also outputs the object's position (bounding box). Therefore, if, for an image recognition model used for object detection, the image features of each class before transfer learning are simply combined before the output layer, as is done for an image recognition model used for classification, it will not only affect the classification results but also the bounding box. As a result, the object detection accuracy will decrease. In other words, since the image features of each class before transfer learning do not contain bounding box information, if the image features affect the bounding box, the object detection accuracy will decrease.

[0050] Embodiment 1 relates to an image recognition device for object detection that reduces bias in training images due to causal inference intervention processing.

[0051] By reducing the bias of the training images used to train the image recognition model before transfer learning, the learning efficiency during transfer learning for image recognition of new target objects can be improved. Furthermore, even with a limited number of training images of the new target object, the type (class) and location of the new target object can be detected with high accuracy. The key points of Embodiment 1 are as follows: The probability that an image of a new target object fits into each class before transfer learning is calculated. The weighting of image features in each class before transfer learning is adjusted according to this probability. The adjusted probabilities are combined with image features extracted from the image of the new target object. This reduces the bias of the training images used in the image recognition model before transfer learning, making it possible to detect the type and location of the new target object. Furthermore, the layer that combines with the method for extracting image features in each class before transfer learning is adapted to an image recognition model for object detection.

[0052] Embodiment 1 makes it possible to detect the type and location of a new target object with high accuracy, even if the number of training images of the new target object for the new task is small.

[0053] Embodiment 1 offers the following advantages: By reducing the bias of the training images used to train the image recognition model before transfer learning, the learning efficiency during transfer learning for image recognition of new target objects can be improved. Furthermore, even when there are few training images of the new target object, the type (class) and object position of the new target object can be detected with high accuracy. Embodiment 1 has the following features: It calculates the probability that an image of a new target object fits into each class before transfer learning. It adjusts the weighting of image features in each class before transfer learning according to this probability. It combines the adjusted probabilities with image features extracted from the image of the new target object. This reduces the bias of the training data used in the image recognition model before transfer learning. This method is applied to an image recognition model for object detection to detect the type and location of a new target object.

[0054] In other words, Embodiment 1 is characterized by the following: By using image features that combine additional image features, the bias of the training images used in the image recognition model before transfer learning is reduced, inference processing is performed on the image to be inferred, and the type (class) and location of objects present in the image are detected.

[0055] Embodiment 2. The main differences between the image recognition device 100 and Embodiment 1 will be explained based on Figures 8 to 11.

[0056] ***Explanation of the structure*** The configuration of the image recognition device 100 is the same as that in Embodiment 1. However, the configuration of the inference unit 110 differs from that in Embodiment 1.

[0057] The configuration of the inference unit 110 will be explained based on Figure 8. The inference unit 110 includes a feature separation unit 116, a probability estimation unit 111, an additional feature calculation unit 112, a feature merging unit 114, a new class estimation unit 117, an object position estimation unit 118, and an estimation result integration unit 119.

[0058] Based on Figure 9, the configuration of the output layer 220 of the image recognition model 200 will be explained. The output layer 220 comprises a feature separation layer 221, a feature extraction layer 222, a probability estimation layer 223, and an object position estimation layer 224. The feature separation layer 221 separates the image features into image features for classification and image features for object position estimation. The feature extraction layer 222 performs convolution and other operations on the image features for classification to extract image features for probability estimation. The probability estimation layer 223 estimates the probability of fitting into a new class after transfer learning, based on image features for probability estimation. The object position estimation layer 224 estimates the location (bounding box, etc.) of the target object in the input image based on image features for object position estimation. The feature separation layer 221, the feature extraction layer 222, and the object position estimation layer 224 are composed of multiple layers of convolutional layers and batch normalization layers. The probability estimation layer 223 is composed of convolutional layers.

[0059] ***Explanation of operation*** The image recognition method will be explained based on Figure 10. In step S201, the feature separation unit 116 calculates the feature quantities of the target image 191 in which the target object is depicted. The calculated feature quantities are referred to as target features. The method for calculating the target features is the same as the method in Embodiment 1.

[0060] The feature separation unit 116 then separates the target feature into features for classification and features for location estimation.

[0061] The target features are separated as follows: The feature separation unit 116 takes the target feature as input and uses the feature separation layer 221 to separate the target feature into features for classification and features for location estimation.

[0062] In step S202, the probability estimation unit 111 estimates the probability that the target object fits into each of several existing classes based on the classification features. The estimated probability is referred to as the estimated probability.

[0063] The estimated probability is calculated as follows: The probability estimation unit 111 takes the classification features as input and uses the feature extraction layer 222 and the probability estimation layer 223 to estimate the estimated probability of the target object for each of the multiple existing classes.

[0064] In step S203, the additional feature calculation unit 112 calculates additional features using the estimated probability of the target object for each of the multiple existing classes and the fitted features for each of the multiple existing classes. The method for calculating additional features is the same as the method in Embodiment 1.

[0065] In step S204, the feature merging unit 114 merges additional features with the classification features. The resulting features are called merged features.

[0066] In step S205, the new class estimation unit 117 estimates the probability of the target object fitting to the new class based on the combined features. In other words, the new class estimation unit 117 estimates the probability that the target object fits into a new class.

[0067] The probability of a good fit is estimated as follows: The new class estimation unit 117 uses the combined features as input and the probability estimation layer 223 to estimate the probability of the target object fitting to the new class.

[0068] In step S206, the object position estimation unit 118 estimates the position of the target object in the target image based on the position estimation features.

[0069] The object's position is estimated as follows: The object position estimation unit 118 uses the object position estimation layer 224 to estimate the position of the target object in the target image, taking the position estimation feature quantities as input.

[0070] In step S207, the estimation result integration unit 119 integrates the fit probability of the target object for the new class with the position of the target object in the target image to generate an inference result 199. The estimation result integration unit 119 then outputs the inference result 199.

[0071] ***Description of the Example*** The similarity between the additional features and the target features may also be calculated. The configuration of the inference unit 110 will be explained based on Figure 11. The inference unit 110 further includes a feature similarity calculation unit 180. First, the feature similarity calculation unit 180 calculates the similarity between the additional features and the target features. The calculated similarity is referred to as the feature similarity score 181. Examples of feature similarity metric 181 include cosine similarity, Euclidean distance, and Pearson correlation coefficient. The feature similarity calculation unit 180 then outputs the feature similarity score 181. For example, the feature similarity calculation unit 180 displays the feature similarity score 181 on the display.

[0072] ***Effects of Embodiment 2*** In Embodiment 2, the output layer of the image recognition model for object detection is separated into a class classification layer (a layer that outputs the probability of fitting into each class) and an object position detection layer (a layer that outputs a bounding box). Embodiment 2 combines image features from each class before transfer learning in a class classification layer, thereby reducing the bias of the training images before transfer learning. This makes it possible to apply bias reduction techniques to the image recognition model for object detection. As a result, even when using a small amount of training data during transfer learning, it becomes possible to detect the type (class) and location of the target object with high accuracy.

[0073] Embodiment 3. The methods for calculating fitted features will be explained, primarily based on the differences between Embodiment 1 and Embodiment 2, using Figures 12 to 16.

[0074] ***Explanation of the structure*** The configuration of the image recognition device 100 will be explained based on Figure 12. The image recognition device 100 further includes an element called a fitted feature calculation unit 120. The fitted feature calculation unit 120 is implemented by software. The image recognition program further utilizes a computer as a fitted feature calculation unit 120.

[0075] Based on Figure 13, the configuration of the fitted feature calculation unit 120 will be explained. The fitted feature calculation unit 120 includes an image segmentation unit 121, a duplicate block determination unit 122, and a feature calculation unit 123. The existing class image DB193 is stored in the storage unit 190.

[0076] ***Explanation of operation*** Based on Figure 14, we will explain how to calculate fitted features. Feature extraction is a process included in image recognition methods. The existing class image DB193 stores multiple existing class images corresponding to multiple existing classes. Steps S301 to S303 are performed for each existing class image.

[0077] In step S301, the image splitting unit 121 divides the existing class image into multiple image blocks.

[0078] Figure 15 shows an existing class image 230. Existing class image 230 is an example of an existing class image. The existing class image 230 shows "Camel". Figure 16 shows an existing class image 230 divided into 30 image blocks. The existing class image 230 is divided vertically into 5 sections and horizontally into 6 sections.

[0079] Returning to Figure 14, we will continue the explanation from step S302. In step S302, the duplicate block determination unit 122 determines which of the multiple image blocks contains an existing class object. An image block containing an existing class object is referred to as a duplicate block.

[0080] Duplicate blocks are determined as follows: Existing class images are pre-trained images, and these images are accompanied by bounding boxes that indicate the positions of existing class objects. The duplicate block determination unit 122 selects one or more image blocks from among multiple image blocks that overlap with at least a portion of the bounding box. Each of the one or more selected image blocks is a duplicate block.

[0081] In Figure 15, the existing class image 230 has a bounding box indicating the position of "Camel". In Figure 16, each of the four image blocks enclosed in thick borders overlaps with at least a portion of the bounding box. Therefore, each of the four image blocks enclosed in thick borders is an overlapping block.

[0082] Returning to Figure 14, step S303 will be explained. In step S303, the feature calculation unit 123 calculates the features of the overlapping blocks. The calculated features become the fitted features.

[0083] Fitted features are calculated as follows: First, the feature calculation unit 123 calculates the features of one or more overlapping blocks. Specifically, the feature calculation unit 123 uses each overlapping block as input and the input / hidden layer 210 of the pre-transfer learning model 201 to calculate the features of each overlapping block. The feature extraction unit 123 then calculates the average of one or more features for one or more overlapping blocks. The calculated average becomes the fitted feature.

[0084] The feature calculation unit 123 stores the fitted features in the fitted feature DB 192, associating them with existing classes that correspond to existing class images.

[0085] ***Effects of Embodiment 3*** Embodiment 3 considers the following overlaps for the intervention processing applied to the image recognition model for object detection. Specifically, it considers the overlap between the labeled image regions (BoundingBoxes) of the training images used in the image recognition model before transfer learning and the divided blocks obtained by dividing the training images used in the image recognition model before transfer learning. The image features in each class extracted before transfer learning, taking overlaps into consideration, are combined with image features extracted from images of new target objects. This reduces the bias of the training data before transfer learning and improves the learning efficiency during transfer learning. As a result, even if fewer training images are used during transfer learning, it becomes possible to detect the type (class) and location of the target object with high accuracy.

[0086] ***Embodiment of Embodiment 3*** The duplicate block determination unit 122 may determine an image block that has a high overlap rate with an existing class object as a duplicate block. In step S302, the duplicate block determination unit 122 calculates the overlap rate with existing class objects for each image block, and determines that an image block whose calculated overlap rate is equal to or greater than a threshold is a duplicate block. An example threshold is 50 percent. In this case, image blocks that overlap the bounding box by more than 50 percent are considered overlapping blocks.

[0087] Embodiment 4. The methods for calculating fitted features will be explained, primarily based on the differences between Embodiments 1 to 3, using Figures 17 to 21.

[0088] ***Explanation of the structure*** The configuration of the image recognition device 100 is the same as that in Embodiment 3. However, the configuration of the fitted feature calculation unit 120 differs from that in Embodiment 3.

[0089] The configuration of the fitted feature calculation unit 120 will be explained based on Figure 17. The fitted feature calculation unit 120 includes an image segmentation unit 121, a duplicate block determination unit 122, a feature calculation unit 123, and an average calculation unit 124.

[0090] ***Explanation of operation*** Based on Figure 18, we will explain how to calculate fitted features. The existing class image database (DB193) stores multiple existing class images for each existing class. Steps S401 through S406 are executed for each existing class.

[0091] In step S401, the image splitting unit 121 selects one existing class image that has not been selected.

[0092] In step S402, the image splitting unit 121 divides the selected existing class image into multiple image blocks.

[0093] In step S403, the duplicate block determination unit 122 determines which of the multiple image blocks contains an existing class object. An image block containing an existing class object is referred to as a duplicate block. The method for determining duplicate blocks is the same as the method in Embodiment 3.

[0094] In step S404, the feature calculation unit 123 calculates the features of the overlapping blocks. The calculated features are referred to as image-specific features. The method for calculating the feature quantities of duplicate blocks is the same as the method in Embodiment 3.

[0095] In step S405, the image segmentation unit 121 determines if there are any existing class images that have not been selected. If there are existing class images that have not been selected, the process proceeds to step S401. If there are no existing class images that have not been selected, the process proceeds to step S406.

[0096] In step S406, the average calculation unit 124 calculates the average of multiple image-specific features corresponding to multiple existing class images. The calculated average becomes the fitted feature.

[0097] The average calculation unit 124 then stores the fitted features in the fitted feature DB 192, associating them with existing classes.

[0098] ***Effects of Embodiment 4*** Embodiment 4 provides the same effects as Embodiment 3.

[0099] ***Embodiment of Embodiment 4*** ***Description of Example 1*** Alternatively, the image-specific features can be separated into features for classification and features for location estimation, and the fitting features can be calculated using the features for classification.

[0100] Based on Figure 19, the configuration of the fitted feature calculation unit 120 will be explained. The fitted feature calculation unit 120 further includes a feature separation unit 125.

[0101] Based on Figure 20, we will explain how to calculate fitted features. Steps S411 to S414 are the same as steps S401 to S404.

[0102] In step S415, the feature separation unit 125 separates the image-specific features into features for classification and features for location estimation.

[0103] Image-specific features are separated as follows: The feature separation unit 125 takes the image-specific features as input and uses the feature separation layer 221 of the pre-transfer learning model 201 to separate the image-specific features into features for classification and features for location estimation.

[0104] In step S416, the image segmentation unit 121 determines if there are any existing class images that have not been selected. If there are existing class images that have not been selected, the process proceeds to step S411. If there are no existing class images that have not been selected, the process proceeds to step S417.

[0105] In step S417, the average calculation unit 124 calculates the average of multiple classification features corresponding to multiple existing class images. The calculated average becomes the fitted feature.

[0106] The average calculation unit 124 then stores the fitted features in the fitted feature DB 192, associating them with existing classes.

[0107] ***Description of Example 2*** The existing class image may be divided into multiple block sizes, and fitted features may be calculated for each block size. In this case, the fit probability of the target object for the new class is estimated for each block size using the fitted features corresponding to each of the multiple block sizes. Figure 21 shows an existing class image 230 divided into 120 image blocks. For example, fitted features are calculated for the existing class image 230 divided into 120 image blocks, and fitted features are calculated for the existing class image 230 divided into 30 image blocks (see Figure 16). The calculation of fitted features is performed as follows: Steps S402 through S404 and S406 are executed for each block size. As a result, for each of the multiple existing classes, the existing class image is divided into multiple image blocks by the block size for each pair of the multiple existing class images and the multiple block sizes, and fitted features are calculated for each block size. Then, for each existing class, multiple fitted features corresponding to multiple block sizes are stored in the fitted feature DB192, associated with the existing class. The image recognition process (S101-S105 or S201-Step S207) is executed as follows. Steps S102, S104, and S105 are performed for each block size. Steps S203 through S205 are executed for each block size. As a result, the probability of fitting the target object to the new class is estimated for each block size using fitting features corresponding to each of the multiple block sizes.

[0108] ***Description of Example 3*** The average of two or more image-specific similarity values ​​from among multiple image-specific features corresponding to multiple existing class images may be calculated as the fitted feature. In step S406, the average calculation unit 124 operates as follows: First, the average calculation unit 124 calculates the feature similarity between each image feature included in the multiple image features corresponding to multiple existing class images and image features other than the target image feature. Examples of feature similarity measures include cosine similarity, Euclidean distance, and Pearson correlation coefficient. Next, the average calculation unit 124 excludes image-specific features whose feature similarity is below a threshold from the multiple image-specific features. As a result, two or more image-specific features are selected. In other words, image-specific features with low feature similarity are considered noise and are removed. The average calculation unit 124 then calculates the average of two or more selected image-specific features. The calculated average becomes the fitted feature.

[0109] Embodiment 5. The differences between the transfer learning method and embodiments 1 to 4 will be explained primarily based on Figures 22 to 26.

[0110] ***Explanation of the structure*** The configuration of the image recognition device 100 will be explained based on Figure 22. The image recognition device 100 further includes an element called a transfer learning unit 130. The transfer learning unit 130 is implemented by software. The image recognition program further enables the computer to function as a transfer learning unit 130.

[0111] Based on Figure 23, the configuration of the transfer learning unit 130 will be explained. The transfer learning unit 130 includes a probability estimation unit 131, an additional feature calculation unit 132, a feature merging unit 133, a new class estimation unit 134, an object position estimation unit 135, a loss value calculation unit 136, and a parameter update unit 137. Furthermore, the transfer learning unit 130 includes a feature acquisition unit 140.

[0112] The configuration of the feature acquisition unit 140 will be explained based on Figure 24. The feature acquisition unit 140 includes an image splitting unit 141, a duplicate block determination unit 142, a feature calculation unit 143, and a feature separation unit 144. The learning image DB194 is stored in the memory unit 190.

[0113] ***Explanation of operation*** Transfer learning will be explained based on Figures 25 and 26. Transfer learning is a process included in image recognition methods. The training image database 194 stores one or more training images that contain objects that fit the new class (new class objects). The training images are accompanied by ground truth data. The ground truth data indicates the label that identifies the new class object and the position of the new class object in the training image. Steps S501 to S517 are performed for each training image.

[0114] In step S501, the image splitting unit 141 divides the training image into multiple image blocks.

[0115] In step S502, the duplicate block determination unit 142 determines which of the multiple image blocks contains the new class object. The image block containing the new class object is referred to as a duplicate block. The method for determining duplicate blocks is the same as the method in Embodiment 3.

[0116] In step S503, the feature calculation unit 143 calculates the features of the overlapping blocks. The calculated features are referred to as overlapping block features. The method for calculating overlapping block features is the same as the method for calculating fitted features in Embodiment 3.

[0117] In step S504, the feature separation unit 144 separates the overlapping block features into features for classification and features for location estimation. The method for separating duplicate block features is the same as the method for separating target features in Embodiment 2.

[0118] In step S511, the probability estimation unit 131 estimates the probability that a new class object fits into each of several existing classes based on the classification features. The estimated probability is referred to as the estimated probability. The method for estimating the estimated probability is the same as the method in Embodiment 2.

[0119] In step S512, the additional feature calculation unit 132 calculates additional features using the estimated probability of the new class object for each of the multiple existing classes and the fitted features for each of the multiple existing classes. The method for calculating additional features is the same as the method in Embodiment 2.

[0120] In step S513, the feature merging unit 133 merges additional features with the classification features. The resulting features are called merged features.

[0121] In step S514, the new class estimation unit 134 estimates the probability of fitting a new class object to a new class based on the combined features. In other words, the new class estimation unit 134 estimates the probability that a new class object fits into the new class. The method for estimating the probability of a good fit is the same as the method in Embodiment 2.

[0122] In step S515, the object position estimation unit 135 estimates the position of a new class object in the training image based on the position estimation features. The method for estimating the object's position is the same as the method in Embodiment 2.

[0123] In step S516, the loss value calculation unit 136 calculates the estimated fit probability and the estimated object position loss value for the ground truth data.

[0124] The loss value is calculated, for example, as follows: First, the loss value calculation unit 136 calculates the difference between the correct probability (100%) and the estimated fit probability, and the difference between the position of the new class object shown in the correct data and the estimated position of the new class object. The loss value calculation unit 136 then multiplies the calculated differences. The value calculated by this multiplication becomes the loss value.

[0125] In step S517, the parameter update unit 137 updates the parameters of the pre-transfer learning model 201 (particularly the parameters of the output layer 220) based on the loss value.

[0126] At this time, the parameter update unit 137 updates the parameters of the pre-transfer learning model 201 so that the loss value is minimized.

[0127] ***Effects of Embodiment 5*** Embodiment 5 makes it possible to create an image recognition model that can detect the type (class) and location of an object with high accuracy even when using a small number of training images during transfer learning.

[0128] Embodiment 5 aims to enable image recognition of a novel target object "α" using a small number of training images. In Embodiment 5, the image recognition model is pre-trained with three classes, "β, γ, and δ," before transfer learning. "α" and "β" are classes that have similar image features. To create an image recognition model that recognizes "α" in images, it is appropriate to perform transfer learning using a pre-transfer learning image recognition model that has been trained on many training images of "β" which have image features similar to "α". However, in some cases, the image recognition model before transfer learning may have learned many training images of "γ" and "δ" other than "β". In this case, the bias in the training images becomes a bias, and the learning efficiency during transfer learning of "α" decreases. Furthermore, creating a new image recognition model that has been trained on a large number of "β" training images before transfer learning is costly. Therefore, an intervention process is performed during transfer learning to add and combine image features of "β," thereby increasing the number of image features of "β." This reduces the bias of the training images included in the image recognition model before transfer learning. Furthermore, since the bias of the training images is reduced, the learning efficiency during transfer learning is improved, and it becomes possible to create an image recognition model that can detect "α" anew even with a small number of training images.

[0129] ***Supplementary Information on the Embodiment*** The hardware configuration of the image recognition device 100 will be explained based on Figure 27. The image recognition device 100 includes a processing circuit 109. The processing circuit 109 is hardware that implements the inference unit 110, the fitted feature calculation unit 120, and the transfer learning unit 130. The processing circuit 109 may be dedicated hardware, or it may be a processor 101 that executes a program stored in memory 102.

[0130] If the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0131] The image recognition device 100 may include multiple processing circuits that replace the processing circuit 109.

[0132] In the processing circuit 109, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0133] Thus, the functions of the image recognition device 100 can be realized by hardware, software, firmware, or a combination thereof.

[0134] The image recognition device 100 is not limited to a general-purpose computer. For example, the image recognition device 100 may be an embedded device, an ECU (electronic control unit), an FPGA board, or a GPU board.

[0135] Each embodiment is an example of a preferred form and is not intended to limit the technical scope of this disclosure. Each embodiment may be implemented in part or in combination with other embodiments. Procedures described using flowcharts, etc., may be modified as appropriate.

[0136] The "part" of each element of the image recognition device 100 may be read as "processing," "process," "circuit," or "circuit." [Explanation of symbols]

[0137] 100 Image recognition device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Input / Output interface, 109 Processing circuit, 110 Inference unit, 111 Probability estimation unit, 112 Additional feature calculation unit, 113 Target feature calculation unit, 114 Feature concatenation unit, 115 Object detection unit, 116 Feature separation unit, 117 New class estimation unit, 118 Object position estimation unit, 119 Estimation result integration unit, 120 Fitted feature calculation unit, 121 Image segmentation unit, 122 Duplicate block determination unit, 123 Feature calculation unit, 124 Average calculation unit, 125 Feature separation unit, 130 Transfer learning unit, 131 Probability estimation unit, 132 Additional feature calculation unit, 133 Feature concatenation unit, 134 New class estimation unit, 135 Object position estimation unit, 136 Loss value calculation unit, 137 Parameter update unit, 140 Feature acquisition unit, 141 Image splitting unit, 142 Duplicate block determination unit, 143 Feature calculation unit, 144 Feature separation unit, 180 Feature similarity calculation unit, 181 Feature similarity, 190 Memory unit, 191 Target image, 192 Fitted feature DB, 193 Existing class image DB, 194 Training image DB, 199 Inference result, 200 Image recognition model, 201 Model before transfer learning, 202 Model after transfer learning, 210 Input / hidden layer, 220 Output layer, 221 Feature separation layer, 222 Feature extraction layer, 223 Probability estimation layer, 224 Object position estimation layer, 230 Existing class image.

Claims

1. A probability estimation unit estimates the probability that the target object captured in the target image fits into each of several existing object classes, and An additional feature calculation unit calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images showing existing class objects that fit each of the plurality of classes. A feature concatenation unit calculates a concatenated feature by concatenating the additional feature with the target feature, which is a feature of the target image. An object detection unit estimates, based on the combined features, the probability that the target object fits into a new class which is a different object class from the plurality of existing classes, and the position of the target object in the target image. An image recognition device equipped with the following features.

2. A feature separation unit separates the target feature quantity, which is the feature quantity of the target image in which the target object is captured, into a feature quantity for classification and a feature quantity for position estimation. A probability estimation unit estimates the probability that the target object fits into each of several existing classes, which are multiple object classes, based on the aforementioned classification features, as an estimated probability. An additional feature calculation unit calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images showing existing class objects that fit each of the plurality of classes. A feature concatenation unit that calculates a concatenated feature by combining the aforementioned classification feature with the aforementioned additional feature, A new class estimation unit estimates the probability that the target object fits into a new class that is a different object class from the plurality of existing classes, based on the combined features, An object position estimation unit that estimates the position of the target object in the target image based on the position estimation feature quantities, An image recognition device equipped with the following features.

3. The system includes a fitting feature calculation unit that divides the existing class image into multiple image blocks, determines that the image block containing the existing class object is a duplicate block, and calculates the feature quantities of the duplicate block as the fitting feature quantities. The image recognition device according to claim 1 or claim 2.

4. The system includes a fitting feature calculation unit that, for each existing class image included in multiple existing class images, divides the existing class image into multiple image blocks, determines that the image block containing the existing class object is a duplicate block, calculates the feature quantities of the duplicate blocks as image-specific feature quantities, and calculates the average of the multiple image-specific feature quantities corresponding to the multiple existing class images as the fitting feature quantity. The image recognition device according to claim 1 or claim 2.

5. The aforementioned image recognition device is For each pair of multiple existing class images and each of multiple block sizes, the existing class image is divided into multiple image blocks according to the block size, the image blocks in which the existing class object is depicted are determined to be duplicate blocks, the feature quantities of the duplicate blocks are calculated as image-specific feature quantities, and for each block size, the average of the multiple image-specific feature quantities corresponding to the multiple existing class images is calculated as the fitted feature quantity. Equipped with, The probability of the target object fitting to the new class is estimated for each block size using the fitting features corresponding to each of the plurality of block sizes. The image recognition device according to claim 1 or claim 2.

6. The system includes a fitting feature calculation unit that, for each existing class image included in multiple existing class images, divides the existing class image into multiple image blocks, determines that the image block containing the existing class object is a duplicate block, calculates the feature quantities of the duplicate block as image-specific features, calculates the feature similarity between each image-specific feature included in the multiple image-specific features corresponding to the multiple existing class images and other image-specific features, selects two or more image-specific features by excluding image-specific features whose feature similarity is below a threshold from the multiple image-specific features, and calculates the average of the two or more selected image-specific features as the fitting feature. The image recognition device according to claim 1 or claim 2.

7. The aforementioned fitted feature calculation unit calculates the overlap rate with the existing class object for each image block, and determines that image blocks whose calculated overlap rate is equal to or greater than a threshold are the overlapping blocks. The image recognition device according to claim 3.

8. The matching feature calculation unit calculates the overlap rate with the existing class object for each image block, and determines that the image block whose calculated overlap rate is equal to or greater than a threshold is the overlap block. The image recognition device according to claim 4.

9. The matching feature calculation unit calculates the overlap rate with the existing class object for each image block, and determines that the image block whose calculated overlap rate is equal to or greater than a threshold is the overlap block. The image recognition device according to claim 5.

10. The matching feature calculation unit calculates the overlap rate with the existing class object for each image block, and determines that the image block whose calculated overlap rate is equal to or greater than a threshold is the overlap block. The image recognition device according to claim 6.

11. The system includes a feature similarity calculation unit that calculates the similarity between the additional feature and the target feature. The image recognition device according to claim 1 or claim 2.

12. The probability that the object captured in the target image fits into each of the multiple existing object classes is estimated as an estimated probability. Additional features are calculated using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images in which existing class objects that fit each of the plurality of existing classes are depicted. The additional feature quantities are combined with the target feature quantity, which is a feature quantity of the target image, to calculate the combined feature quantity. Based on the combined features, the probability of fitting the target object to a new class that is a different object class from the plurality of existing classes, and the position of the target object in the target image are estimated. Image recognition methods.

13. The target features, which are the feature quantities of the target image in which the target object is captured, are separated into features for classification and features for location estimation. Based on the aforementioned classification features, the probability that the target object fits into each of the multiple existing classes, which are multiple object classes, is estimated as an estimated probability. Additional features are calculated using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images in which existing class objects that fit each of the plurality of existing classes are depicted. The combined feature is calculated by combining the aforementioned classification feature with the aforementioned additional feature. Based on the combined features, the probability of fitting the target object to a new class that is a different object class from the plurality of existing classes is estimated. Based on the position estimation features, the position of the target object in the target image is estimated. Image recognition methods.

14. A probability estimation process that estimates the probability that an object captured in a target image fits into each of several existing object classes, and An additional feature calculation process that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images showing existing class objects that fit each of the plurality of classes, A feature concatenation process that calculates a concatenated feature by combining the additional feature with the target feature, which is a feature of the target image, An object detection process that estimates, based on the combined features, the probability that the target object fits into a new class which is a different object class from the plurality of existing classes, and the position of the target object in the target image. An image recognition program that causes a computer to execute an image.

15. A feature separation process is performed to separate the target features, which are the feature quantities of the target image in which the target object is captured, into features for classification and features for location estimation. A probability estimation process that estimates the probability that the target object fits into each of several existing classes, which are multiple object classes, based on the aforementioned classification features, An additional feature calculation process that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and the fitted features calculated as features of existing class images showing existing class objects that fit each of the plurality of classes, A feature merging process that combines the aforementioned classification features with the aforementioned additional features to calculate a combined feature, A new class estimation process that estimates the probability of fitting the target object to a new class that is a different object class from the plurality of existing classes, based on the combined features, An object position estimation process that estimates the position of the target object in the target image based on the position estimation features, An image recognition program that causes a computer to execute an image.

Images