Image recognition device, image recognition method, and image recognition program

Abstract

A probability estimation unit (111) estimates an estimated probability that a target object captured in a target image matches each of a plurality of existing classes. An additional feature amount calculation unit (112) uses the estimated probability of the target object for each of the plurality of existing classes and a feature amount of each existing class image of the plurality of existing classes, to calculate an additional feature amount. A feature amount coupling unit (114) couples the additional feature amount to the feature amount of the target image to calculate a coupled feature amount. An object detection unit (115) estimates, on the basis of the coupled feature amount, a matching probability that the target object matches a new class and the position of the target object in the target image.

Description

Image recognition device, image recognition method, and image recognition program

[0001] The present disclosure relates to image recognition.

[0002] In recent years, the introduction of AI systems has progressed to improve business efficiency and automate tasks. AI stands for artificial intelligence. Building an AI system requires a large number of training images paired with labels. However, since labels are assigned manually, labeling costs are high, and it is difficult to collect a large number of labeled training images. For this reason, technologies have been developed that enable high-accuracy image recognition even with a small number of labeled training images.

[0003] One such technology is transfer learning for image recognition models. In transfer learning, image features of various objects different from the target object are extracted using training images (training images for the image recognition model before transfer learning) of various objects to pre-train an image recognition model. The pre-trained image recognition model is then further trained using a small number of training images of the target object (training images for a new task used during transfer learning), thereby transferring the pre-trained image recognition model to an image recognition model that recognizes the target object. In this way, transfer learning of a pre-trained image recognition model makes it possible to learn a desired object even with a small number of training images. However, if there is a significant difference between the image features learned by the image recognition model before transfer learning and the image features possessed by the target object, transfer learning will not progress if there are only a small number of training images of the target object. This makes it difficult to transfer the image features to the image recognition model that recognizes the target object, resulting in reduced image recognition accuracy. Here, the difference between the image features learned by the image recognition model before transfer learning and the image features possessed by the target object is referred to as the bias of the training images.

[0004] Patent Literature 1 discloses the following machine learning method for transfer learning to reduce bias in training images: Multiple pre-training models are recorded before transfer learning; the fitness of the transfer learning dataset and the pre-training models is evaluated, a pre-training model with high evaluation results is selected, and transfer learning is performed. However, the technology disclosed in Patent Literature 1 requires the preparation of multiple pre-training models. Furthermore, if the bias in the training images is large in all pre-training models, there is a problem in that the bias in the training images cannot be reduced regardless of which pre-training model is selected.

[0005] Japanese Patent Application Laid-Open No. 2023-104067

[0006] Zhongqi Yue, et al. “Interventional few shot learning”, arXiv:2009.13000v2 [cs. LG] 4 Dec 2020

[0007] The present disclosure aims to reduce bias in training images used to create an image recognition model, enabling more accurate estimation of the classification and location of novel objects.

[0008] The image recognition device of the present disclosure includes: a probability estimation unit that estimates, as an estimated probability, the probability that a target object shown in a target image matches each of a plurality of existing classes, which are a plurality of object classes; an additional feature calculation unit that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and match features calculated as features of an existing class image in which an existing class object that matches each of the plurality of existing classes is shown; a feature combination unit that calculates combined features by combining the additional features with target features, which are features of the target image; and an object detection unit that estimates, based on the combined features, the match probability that the target object matches a new class, which is an object class different from the plurality of existing classes, and the position of the target object in the target image.

[0009] According to the present disclosure, it is possible to reduce bias in the training images used to create an image recognition model, thereby enabling more accurate estimation of the classification and location of novel objects.

[0010] FIG. 1 is a configuration diagram of an image recognition device 100 according to a first embodiment. FIG. 2 is a configuration diagram of an inference unit 110 according to the first embodiment. FIG. 3 is a configuration diagram of an image recognition model 200 according to the first embodiment. FIG. 4 is a diagram showing the relationship between the image recognition model 200 and combined features according to the first embodiment. FIG. 5 is a flowchart of an image recognition method according to the first embodiment. FIG. 6 is a configuration diagram of the inference unit 110 according to an example of the first embodiment. A causal graph during transfer learning in Non-Patent Document 1. FIG. 7 is a configuration diagram of the inference unit 110 according to a second embodiment. FIG. 8 is a configuration diagram of an output layer 220 of the image recognition model 200 according to the second embodiment. FIG. 9 is a flowchart of an image recognition method according to the second embodiment. FIG. 10 is a configuration diagram of the inference unit 110 according to an example of the second embodiment. FIG. 11 is a configuration diagram of an image recognition device 100 according to a third embodiment. FIG. 12 is a configuration diagram of a matching feature calculation unit 120 according to the third embodiment. FIG. 12 is a flowchart of matching feature calculation according to the third embodiment. FIG. 13 is a diagram showing an existing class image 230 according to the third embodiment. FIG. 14 is a diagram showing a divided existing class image 230 according to the third embodiment. FIG. 14 is a configuration diagram of a matching feature calculation unit 120 according to a fourth embodiment. FIG. 15 is a flowchart of matching feature calculation according to the fourth embodiment. FIG. 1 is a configuration diagram of a suitable feature calculation unit 120 in an example of embodiment 4. FIG. 2 is a flowchart of suitable feature calculation in an example of embodiment 4. FIG. 3 is a diagram showing a divided existing class image 230 in embodiment 4. FIG. 4 is a configuration diagram of an image recognition device 100 in embodiment 5. FIG. 5 is a configuration diagram of a transfer learning unit 130 in embodiment 5. FIG. 6 is a configuration diagram of a feature acquisition unit 140 in embodiment 5. FIG. 7 is a flowchart of transfer learning in embodiment 5. FIG. 8 is a flowchart of transfer learning in embodiment 5. FIG. 9 is a hardware configuration diagram of an image recognition device 100 in an embodiment.

[0011] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0012] First Embodiment An image recognition device 100 will be described with reference to FIGS.

[0013] ***Description of Configuration*** The configuration of the image recognition device 100 will be described with reference to Fig. 1. The image recognition device 100 is a computer equipped with hardware such as a processor 101, a memory 102, an auxiliary storage device 103, and an input / output interface 104. These pieces of hardware are connected to one another via signal lines.

[0014] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU, a DSP, or a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0015] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0016] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0017] The input / output interface 104 is a port to which an input device and an output device are connected. For example, the input / output interface 104 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output of the image recognition device 100 is performed using the input / output interface 104. USB is an abbreviation for Universal Serial Bus.

[0018] The image recognition device 100 includes an inference unit 110. The inference unit 110 is realized by software.

[0019] The auxiliary storage device 103 stores an image recognition program for causing the computer to function as the inference unit 110. The image recognition program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes the image recognition program while executing the OS. OS is an abbreviation for Operating System.

[0020] Data (input data, output data, etc.) of the image recognition program is stored in the storage unit 190. The auxiliary storage device 103 functions as the storage unit 190. However, a storage device such as the memory 102, a register in the processor 101, or a cache memory in the processor 101 may function as the storage unit 190 instead of or together with the auxiliary storage device 103.

[0021] The image recognition program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0022] The configuration of the inference unit 110 will be described with reference to Fig. 2. The inference unit 110 includes a probability estimation unit 111, an additional feature calculation unit 112, a target feature calculation unit 113, a feature combination unit 114, and an object detection unit 115. A target image 191, a matching feature DB 192, and an inference result 199 are stored in a storage unit 190. DB stands for database.

[0023] The configuration of the image recognition model 200 will be described with reference to FIG. 3. The image recognition model 200 is a trained model that estimates the probability that an object shown in an input image matches each of multiple object classes (classification result) and the position of the object in the input image (object position). The object position is indicated by a bounding box. The image recognition model 200 is stored in the storage unit 190 and used by the image recognition device 100. For example, the image recognition model 200 is configured as a CNN model. CNN is an abbreviation for Convolutional Neural Network.

[0024] The image recognition model 200 includes an input / intermediate layer 210 and an output layer 220. The input / intermediate layer 210 calculates the features of an input image. The input / intermediate layer corresponds to the backbone, and is the basic network part for extracting features from the input image. The output layer 220 estimates the class classification result and object position based on the features of the input image. The output layer corresponds to the head, and is the network part for estimating the type and position of an object in an image.

[0025] The image recognition model 200 before transfer learning for a new object class (new class) is referred to as the pre-transfer learning model 201. The pre-transfer learning model 201 estimates class classification results and object positions for multiple existing object classes (existing classes). The image recognition model 200 after transfer learning for the new class is referred to as the post-transfer learning model 202. The post-transfer learning model 202 estimates class classification results and object positions for the new class and multiple existing classes.

[0026] The formula shown in the dashed box indicates the combination of image features by intervention processing. The image features are expressed in vector format. A feature expressed in vector format is called a feature vector. By calculating this formula, the combined feature, which will be described later, is calculated.

[0027] Figure 4 shows the relationship between the image recognition model 200 and the combined feature. The combined feature is created by combining the first feature x with the second feature. The first feature x is a feature of the input image, and is calculated using the input image as input and the input / hidden layer 210. The second feature is calculated by multiplying the matching feature by the estimated probability. "m" indicates the number of existing classes. The estimated probability is the probability that an object shown in the input image matches each existing class, and is estimated using the image recognition model 200 with the input image as input. The matching feature is a feature of an image showing an object that matches an existing class, and is calculated using the input image as input and the input / hidden layer 210.

[0028] The intervention process of inputting the combined features to the output layer 220 reduces the bias of the training image group (pre-training dataset) used when creating the pre-transfer learning model 201, improving the learning efficiency during transfer learning and the estimation accuracy after transfer learning.

[0029] ***Explanation of Operation*** The operational procedure of the image recognition device 100 corresponds to an image recognition method. Also, the operational procedure of the image recognition device 100 corresponds to a processing procedure according to an image recognition program.

[0030] The image recognition method will be described with reference to Fig. 5. In step S101, the probability estimation unit 111 estimates the probability that a target object shown in the target image 191 matches each of a plurality of existing classes. The estimated probability is referred to as an estimated probability.

[0031] The target object is an object that appears in the target image 191 .

[0032] The multiple existing classes are multiple object classes learned by the pre-transfer learning model, and these multiple objects are objects included in the pre-training dataset.

[0033] The estimated probability is estimated as follows: The probability estimation unit 111 receives the target image 191 as input and uses the pre-transfer learning model 201 to estimate the estimated probability of the target object for each of a plurality of existing classes.

[0034] In step S102, the additional feature calculation unit 112 calculates additional features using the estimated probability of the target object for each of the multiple existing classes and the matching features of each of the multiple existing classes.

[0035] The relevant feature of each of the multiple existing classes is stored in the relevant feature DB 192. The relevant feature is a feature Ca obtained by averaging the feature vectors obtained by inputting the pre-training data set to the input / intermediate layer of the image recognition model shown in FIG. ｉ Here, "a ｉ " represents each existing class, and "Ca ｉ " is each existing class a ｉ represents the feature for

[0036] The matching features of the existing class are the features of the existing class image. The existing class image is an image that shows an existing class object. Specifically, the existing class image is a training image used in learning to create the pre-transfer learning model 201. The existing class object is an object that matches the existing class.

[0037] The matching feature is calculated using the input / hidden layer 210 of the model before transfer learning 201 with the existing class image as input.

[0038] The additional feature amount is calculated as follows: The additional feature amount calculation unit 112 calculates an estimated probability (a ｉ |x) and calculate the estimated probability (a ｉ |x) to the existing class matching feature Ca ｉ Then, the additional feature calculation unit 112 calculates the average of the class features corresponding to the multiple existing classes. The calculated average becomes the additional feature.

[0039] In step S103, the object feature amount calculation unit 113 calculates the feature amount of the object image 191. The calculated feature amount is referred to as the object feature amount.

[0040] The target feature amount is calculated as follows: The target feature amount calculation unit 113 calculates the target feature amount using the target image 191 as input and the input / hidden layer 210 of the model after transfer learning 202 .

[0041] In step S104, the feature combining unit 114 combines the additional feature with the target feature. The feature obtained in this way is called a combined feature.

[0042] In step S105, the object detection unit 115 estimates the matching probability of the target object for the new class based on the combined feature amount. That is, the object detection unit 115 estimates the probability that the target object matches the new class.

[0043] A new class is an object class that represents a different kind of object from the multiple existing classes.

[0044] Furthermore, the object detection unit 115 estimates the position of the target object in the target image based on the combined feature amount.

[0045] The object detection unit 115 performs estimation as follows: Using the combined features as input and the output layer 220 of the model after transfer learning 202, the object detection unit 115 estimates the matching probability of the target object for the new class and the position of the target object in the target image.

[0046] The object detection unit 115 then outputs an inference result 199. The inference result 199 indicates the matching probability of the target object for the new class and the position of the target object in the target image. For example, the object detection unit 115 displays the inference result 199 on a display.

[0047] ***Description of the Example*** The image recognition device 100 may calculate the similarity between an additional feature and a target feature. The configuration of the inference unit 110 will be described with reference to FIG. 6 . The inference unit 110 further includes a feature similarity calculation unit 180. First, the feature similarity calculation unit 180 calculates the similarity between the additional feature and the target feature. The calculated similarity is referred to as feature similarity 181. Examples of the feature similarity 181 are cosine similarity, Euclidean distance, and Pearson correlation coefficient. Then, the feature similarity calculation unit 180 outputs the feature similarity 181. For example, the feature similarity calculation unit 180 displays the feature similarity 181 on a display.

[0048] ***Effects of First Embodiment*** Non-Patent Document 1 proposes a learning technique that uses causal inference to reduce bias in training images used in an image recognition model before transfer learning, thereby improving image recognition accuracy even when there are only a small number of training images for a new task. The learning technique of Non-Patent Document 1 will be described with reference to FIG. 7. FIG. 7 shows a causal graph during transfer learning in Non-Patent Document 1. "D" represents a pre-training dataset for an existing task. "I" represents an sparsely labeled dataset for a new task. "X" represents a feature vector extracted by inputting the sparsely labeled dataset for a new task I into a pre-training model. "C" represents an intermediate feature vector extracted by inputting the pre-training dataset D and the sparsely labeled dataset for a new task I into a machine learning model. "Y" represents the inference result of the machine learning model. Here, it is assumed that the pre-training dataset D is a confounding factor that has a causal relationship not only with the intermediate feature vector C but also with the feature vector X for the new task. This is due to the use of a pre-training model trained with the pre-training dataset D when extracting the feature vector X. In an ideal small-label data learning technique, it is desirable to calculate the probability P(Y|X) between the feature vector X for a new task and the inference result Y of a machine learning model. However, if the pre-training dataset D is a confounding factor, the pre-training dataset D will affect the feature vector X for a new task (see the dashed arrow). This makes it difficult to calculate the probability P(Y|X). To prevent the influence of this confounding factor, a pseudo-intervention process is introduced that combines the image features of each class before transfer learning. Then, by calculating the probability P(Y|do(X)) with the feature vector X fixed to an arbitrary value, the influence of the pre-training dataset D on the feature vector X is reduced. In other words, in the learning technique of Non-Patent Document 1, bias in the training images before transfer learning, which is a confounding factor, affects the image features extracted from the training images for a new task. This reduces the learning efficiency of the image recognition model. To reduce bias in the training images before transfer learning, an intervention process is performed using the class classification results of the image recognition model before transfer learning.In the intervention process, the probability that the training images for the new task match the classification results of the image recognition model before transfer learning is calculated, and the image features of the training images before transfer learning are used according to this probability. This reduces the bias of the training images before transfer learning, reducing their influence on the image features extracted from the training images for the new task, thereby improving learning efficiency. This learning technique makes it possible to accurately classify target objects for the new task even with a small number of training images for the new task. However, while Non-Patent Document 1 discloses an intervention process method applied to an image recognition model for class classification, it does not disclose an intervention process method applied to an image recognition model for object detection. Furthermore, if the intervention process method for class classification is directly applied to an image recognition model for object detection, the accuracy of image recognition for object detection will be significantly reduced.

[0049] Image datasets for object detection are supplemented with information about a frame surrounding an object to be recognized (bounding box information). On the other hand, many image datasets for class classification lack bounding box information, and image features for each class before transfer learning are extracted from the entire image region of the image dataset. Therefore, when a method for extracting image features for class classification is applied to an image dataset for object detection, the image dataset contains many image features from the background portion of the image other than the object to be recognized, making it difficult to accurately extract image features for each class before transfer learning. Furthermore, image recognition models for class classification have a model structure in which each layer outputs a class classification result, while image recognition models for object detection have a model structure that not only outputs the class classification result but also outputs the object position (bounding box). Therefore, simply combining image features for each class before transfer learning before the output layer of an image recognition model for object detection, as in the image recognition model for class classification, not only affects the class classification result but also the bounding box. As a result, object detection accuracy decreases. That is, since the image feature values ​​for each class before transfer learning do not include Bounding Box information, if the image feature values ​​affect the Bounding Box, the object detection accuracy will decrease.

[0050] The first embodiment relates to an image recognition device for object detection that reduces bias in training images due to intervention processing of causal inference.

[0051] By reducing the bias of the training images used in training the image recognition model before transfer learning, it is possible to improve the learning efficiency during transfer learning for image recognition of a new target object. Furthermore, even when there are few training images of the new target object, the type (class) and position of the new target object can be detected with high accuracy. The key points of the first embodiment are as follows: The probability that the image of the new target object matches each class before transfer learning is calculated. The weighting of the image features in each class before transfer learning is adjusted according to the probability. The adjusted probability is combined with the image features extracted from the image of the new target object. This reduces the bias of the training images used in the image recognition model before transfer learning, making it possible to detect the type and position of the new target object. Furthermore, the layer combined with the method for extracting image features in each class before transfer learning is adapted to the image recognition model for object detection.

[0052] According to the first embodiment, even if there are only a small number of learning images of a new detection target object that will be a new task, it is possible to detect the type and position of the new target object with high accuracy.

[0053] The first embodiment has the following advantages. By reducing the bias of the training images used in training the image recognition model before transfer learning, it is possible to improve the learning efficiency during transfer learning for image recognition of a new target object. Furthermore, even when there are few training images of the new target object, it is possible to detect the type (class) and object position of the new target object with high accuracy. The first embodiment has the following features. The probability that the image of the new target object matches each class before transfer learning is calculated. The weighting of the image feature amounts for each class before transfer learning is adjusted according to the probability. The adjusted probability is combined with the image feature amounts extracted from the image of the new target object. This reduces the bias of the training data used in the image recognition model before transfer learning. Such a method is applied to an image recognition model for object detection to detect the type and position of a new target object.

[0054] That is, the first embodiment is characterized by the following points: By using image features combined with additional image features, the bias of the training images used in the image recognition model before transfer learning is reduced, inference processing is performed on the inference target image, and the type (class) and position of the object present in the image are detected.

[0055] Second Embodiment The image recognition device 100 will be described below, focusing mainly on the differences from the first embodiment, with reference to FIGS.

[0056] ***Description of Configuration*** The configuration of the image recognition device 100 is the same as that in Embodiment 1. However, the configuration of the inference unit 110 is different from that in Embodiment 1.

[0057] The configuration of the inference unit 110 will be described with reference to Fig. 8. The inference unit 110 includes a feature separation unit 116, a probability estimation unit 111, an additional feature calculation unit 112, a feature combination unit 114, a new class estimation unit 117, an object position estimation unit 118, and an estimation result integration unit 119.

[0058] The configuration of the output layer 220 of the image recognition model 200 will be described with reference to FIG. 9 . The output layer 220 includes a feature separation layer 221, a feature extraction layer 222, a probability estimation layer 223, and an object location estimation layer 224. The feature separation layer 221 separates image features into image features for class classification and image features for object location estimation. The feature extraction layer 222 extracts image features for probability estimation by performing convolution processing or the like on the image features for class classification. The probability estimation layer 223 estimates the probability of matching to a new class after transfer learning based on the image features for probability estimation. The object location estimation layer 224 estimates the position (such as a bounding box) of a target object in an input image based on the image features for object location estimation. The feature separation layer 221, the feature extraction layer 222, and the object location estimation layer 224 are configured with multiple convolutional layers and batch normalization layers. The probability estimation layer 223 is composed of convolutional layers.

[0059] ***Description of Operation*** The image recognition method will be described with reference to Fig. 10. In step S201, the feature separation unit 116 calculates the feature of the target image 191 in which the target object is captured. The calculated feature is referred to as the target feature. The method for calculating the target feature is the same as the method in the first embodiment.

[0060] Then, the feature separating unit 116 separates the target feature into a classification feature and a position estimation feature.

[0061] The target feature is separated as follows: The feature separation unit 116 receives the target feature as input and separates the target feature into a classification feature and a position estimation feature using the feature separation layer 221 .

[0062] In step S202, the probability estimation unit 111 estimates the probability that the target object fits into each of a plurality of existing classes based on the classification features. The estimated probabilities are referred to as estimated probabilities.

[0063] The estimated probability is estimated as follows: The probability estimation unit 111 receives classification features as input and uses the feature extraction layer 222 and the probability estimation layer 223 to estimate the probability of the target object for each of a plurality of existing classes.

[0064] In step S203, the additional feature calculation unit 112 calculates additional features using the estimated probability of the target object for each of the multiple existing classes and the matching features for each of the multiple existing classes. The method for calculating the additional features is the same as the method in the first embodiment.

[0065] In step S204, the feature combining unit 114 combines the additional feature with the classification feature. The feature obtained in this way is called a combined feature.

[0066] In step S205, the new class estimation unit 117 estimates the matching probability of the target object for the new class based on the combined feature amount. That is, the new class estimation unit 117 estimates the probability that the target object matches the new class.

[0067] The matching probability is estimated as follows: The new class estimation unit 117 receives the combined feature as input and uses the probability estimation layer 223 to estimate the matching probability of the target object for the new class.

[0068] In step S206, the object position estimation unit 118 estimates the position of the target object in the target image based on the feature amounts for position estimation.

[0069] The object position is estimated as follows: The object position estimation unit 118 receives the feature for position estimation as input and uses the object position estimation layer 224 to estimate the position of the target object in the target image.

[0070] In step S207, the estimation result integration unit 119 integrates the matching probability of the target object for the new class and the position of the target object in the target image to generate an inference result 199. Then, the estimation result integration unit 119 outputs the inference result 199.

[0071] ***Description of the Example*** The similarity between the additional feature and the target feature may be calculated. The configuration of the inference unit 110 will be described with reference to FIG. 11 . The inference unit 110 further includes a feature similarity calculation unit 180. First, the feature similarity calculation unit 180 calculates the similarity between the additional feature and the target feature. The calculated similarity is referred to as feature similarity 181. Examples of the feature similarity 181 are cosine similarity, Euclidean distance, and Pearson correlation coefficient. Then, the feature similarity calculation unit 180 outputs the feature similarity 181. For example, the feature similarity calculation unit 180 displays the feature similarity 181 on a display.

[0072] ***Effects of Second Embodiment*** In the second embodiment, the output layer of the image recognition model for object detection is separated into a layer for class classification (a layer that outputs the probability of conformance to each class) and a layer for object position detection (a layer that outputs a bounding box). In the second embodiment, the image features of each class before transfer learning are combined in the layer for class classification to reduce bias in the training images before transfer learning. This makes it possible to apply a bias reduction technique to the image recognition model for object detection. And, even if a small amount of training data is used during transfer learning, it becomes possible to detect the type (class) and position of a target object with high accuracy.

[0073] Third Embodiment A mode of calculating the relevant feature amount will be described below, focusing mainly on the differences from the first and second embodiments, with reference to Figs.

[0074] ***Description of Configuration*** The configuration of the image recognition device 100 will be described with reference to Fig. 12. The image recognition device 100 further includes an element called a matching feature calculation unit 120. The matching feature calculation unit 120 is realized by software. The image recognition program further causes a computer to function as the matching feature calculation unit 120.

[0075] The configuration of the matching feature calculation unit 120 will be described with reference to Fig. 13. The matching feature calculation unit 120 includes an image division unit 121, an overlapping block determination unit 122, and a feature calculation unit 123. An existing class image DB 193 is stored in the storage unit 190.

[0076] ***Description of Operation*** The calculation of the matching feature amount will be described with reference to Fig. 14. The calculation of the matching feature amount is a process included in the image recognition method. The existing class image DB 193 stores a plurality of existing class images corresponding to a plurality of existing classes. Steps S301 to S303 are executed for each existing class image.

[0077] In step S301, the image dividing unit 121 divides the existing class image into a plurality of image blocks.

[0078] Fig. 15 shows an existing class image 230. The existing class image 230 is an example of an existing class image. The existing class image 230 shows "Camel." Fig. 16 shows the existing class image 230 divided into 30 image blocks. The existing class image 230 is divided into five vertically and six horizontally.

[0079] 14, the description will continue from step S302. In step S302, the overlapping block determination unit 122 determines, from among a plurality of image blocks, image blocks in which an existing class object appears. An image block in which an existing class object appears is referred to as an overlapping block.

[0080] An overlapping block is determined as follows: The existing class image is a trained image, and a bounding box indicating the position of the existing class object is attached to the existing class image. The overlapping block determination unit 122 selects one or more image blocks from the multiple image blocks that overlap with at least a portion of the bounding box. Each of the selected one or more image blocks is an overlapping block.

[0081] In Fig. 15, a bounding box indicating the position of "Camel" is attached to the existing class image 230. In Fig. 16, each of the four image blocks surrounded by a thick frame overlaps at least a portion of the bounding box. Therefore, each of the four image blocks surrounded by a thick frame is an overlapping block.

[0082] 14, step S303 will be described. In step S303, the feature amount calculation unit 123 calculates the feature amounts of the overlapping blocks. The calculated feature amounts become matching feature amounts.

[0083] The matching feature is calculated as follows. First, the feature calculation unit 123 calculates the feature of each of one or more overlapping blocks. Specifically, the feature calculation unit 123 calculates the feature of each overlapping block using the input / hidden layer 210 of the pre-transfer learning model 201 with each overlapping block as input. Then, the feature calculation unit 123 calculates the average of one or more feature values ​​for one or more overlapping blocks. The calculated average becomes the matching feature.

[0084] The feature calculation unit 123 stores the matching feature in the matching feature DB 192 in association with the existing class corresponding to the existing class image.

[0085] ***Effects of Third Embodiment*** In the third embodiment, the following redundancy is taken into consideration for the intervention process applied to the image recognition model for object detection. That is, the redundancy between the image region (bounding box) labeled in the training image used in the image recognition model before transfer learning and the divided blocks into which the training image used in the image recognition model before transfer learning is divided is taken into consideration. The image features for each class before transfer learning, extracted while taking the redundancy into consideration, are combined with the image features extracted from the image of a new target object. This reduces bias in the training data before transfer learning and improves learning efficiency during transfer learning. Furthermore, even if a small number of training images are used during transfer learning, it becomes possible to detect the type (class) and position of the target object with high accuracy.

[0086] ***Example of Third Embodiment*** The overlapping block determination unit 122 may determine, among image blocks that overlap with existing class objects, image blocks that have a high overlap rate with existing class objects as overlapping blocks. In step S302, the overlapping block determination unit 122 calculates the overlap rate with existing class objects for each image block, and determines, as overlapping blocks, image blocks for which the calculated overlap rate is equal to or greater than a threshold. An example of the threshold is 50 percent. In this case, image blocks that overlap 50 percent or more with the bounding box are overlapping blocks.

[0087] Fourth Embodiment A method for calculating a matching feature amount will be described below, focusing mainly on the differences from the first to third embodiments, with reference to FIGS.

[0088] ***Description of Configuration*** The configuration of the image recognition device 100 is the same as that in the third embodiment. However, the configuration of the relevant feature calculation unit 120 is different from that in the third embodiment.

[0089] The configuration of the matching feature calculation unit 120 will be described with reference to Fig. 17. The matching feature calculation unit 120 includes an image division unit 121, an overlapping block determination unit 122, a feature calculation unit 123, and an average calculation unit .

[0090] ***Explanation of Operation*** The calculation of the matching feature amount will be described with reference to Fig. 18. The existing class image DB 193 stores a plurality of existing class images for each existing class. Steps S401 to S406 are executed for each existing class.

[0091] In step S401, the image dividing unit 121 selects one unselected existing class image.

[0092] In step S402, the image dividing unit 121 divides the selected existing class image into a plurality of image blocks.

[0093] In step S403, the overlapping block determination unit 122 determines which image blocks contain an existing class object from among the multiple image blocks. An image block containing an existing class object is referred to as an overlapping block. The method for determining overlapping blocks is the same as the method in the third embodiment.

[0094] In step S404, the feature amount calculation unit 123 calculates the feature amounts of the overlapping blocks. The calculated feature amounts are referred to as image-specific feature amounts. The method for calculating the feature amounts of the overlapping blocks is the same as the method in the third embodiment.

[0095] In step S405, the image dividing unit 121 determines whether there is an unselected existing class image. If there is an unselected existing class image, the process proceeds to step S401. If there is no unselected existing class image, the process proceeds to step S406.

[0096] In step S406, the average calculation unit 124 calculates the average of the image feature amounts corresponding to the existing class images. The calculated average becomes the matching feature amount.

[0097] Then, the average calculation unit 124 stores the matching feature in the matching feature DB 192 in association with the existing class.

[0098] ***Effects of the Fourth Embodiment*** The fourth embodiment has the same effects as the third embodiment.

[0099] ***Example of Fourth Embodiment*** ***Description of Example 1*** Image features may be separated into class classification features and position estimation features, and the class separation features may be used to calculate matching features.

[0100] The configuration of the matching feature calculation unit 120 will be described with reference to Fig. 19. The matching feature calculation unit 120 further includes a feature separation unit 125.

[0101] The calculation of the matching feature amount will be described with reference to Fig. 20. Steps S411 to S414 are the same as steps S401 to S404.

[0102] In step S415, the feature separating unit 125 separates the image feature into a feature for class classification and a feature for position estimation.

[0103] The image-specific features are separated as follows: The feature separation unit 125 receives the image-specific features as input and separates them into class classification features and position estimation features using the feature separation layer 221 of the model before transfer learning 201.

[0104] In step S416, the image dividing unit 121 determines whether there is an unselected existing class image. If there is an unselected existing class image, the process proceeds to step S411. If there is no unselected existing class image, the process proceeds to step S417.

[0105] In step S417, the average calculation unit 124 calculates the average of the classification features corresponding to the existing class images. The calculated average becomes the matching feature.

[0106] Then, the average calculation unit 124 stores the matching feature in the matching feature DB 192 in association with the existing class.

[0107] ***Description of Example 2*** An existing class image may be divided into multiple block sizes, and matching features may be calculated for each block size. In this case, the matching probability of the target object for the new class is estimated for each block size using matching features corresponding to each of the multiple block sizes. FIG. 21 shows an existing class image 230 divided into 120 image blocks. For example, matching features for the existing class image 230 divided into 120 image blocks and matching features for the existing class image 230 divided into 30 image blocks (see FIG. 16 ) are calculated. The matching feature calculation is performed as follows: Steps S402 to S404 and step S406 are performed for each block size. As a result, for each of the multiple existing classes, the existing class image is divided into multiple image blocks by the block size for each combination of each existing class image of the multiple existing class images and each of the multiple block sizes, and matching features are calculated for each block size. Then, multiple matching features corresponding to the multiple block sizes are associated with each existing class and stored in the matching feature DB 192. The image recognition method (steps S101 to S105 or S201 to S207) is performed as follows: Steps S102, S104, and S105 are performed for each block size. Steps S203 to S205 are performed for each block size. As a result, the matching probability of the target object for the new class is estimated for each block size using matching features corresponding to each of the multiple block sizes.

[0108] ***Description of Example 3*** The average of two or more image-specific similarities among a plurality of image-specific features corresponding to a plurality of existing class images may be calculated as the matching feature. In step S406, the average calculation unit 124 operates as follows. First, for each image-specific feature included in a plurality of image-specific features corresponding to a plurality of existing class images, the average calculation unit 124 calculates the feature similarity with an image-specific feature other than the target image feature. Examples of feature similarity include cosine similarity, Euclidean distance, and Pearson correlation coefficient. Next, the average calculation unit 124 excludes image-specific features whose feature similarity is equal to or less than a threshold from the plurality of image-specific features. This selects two or more image-specific features. In other words, image-specific features with low feature similarity are considered noise and are deleted. Then, the average calculation unit 124 calculates the average of the selected two or more image-specific features. The calculated average becomes the matching feature.

[0109] Fifth Embodiment A fifth embodiment of transfer learning will be described, focusing mainly on the differences from the first to fourth embodiments, with reference to FIGS.

[0110] ***Description of Configuration*** The configuration of the image recognition device 100 will be described with reference to Fig. 22. The image recognition device 100 further includes an element called a transfer learning unit 130. The transfer learning unit 130 is realized by software. The image recognition program further causes a computer to function as the transfer learning unit 130.

[0111] The configuration of the transfer learning unit 130 will be described with reference to Fig. 23. The transfer learning unit 130 includes a probability estimation unit 131, an additional feature calculation unit 132, a feature combination unit 133, a new class estimation unit 134, an object position estimation unit 135, a loss value calculation unit 136, and a parameter update unit 137. The transfer learning unit 130 also includes a feature acquisition unit 140.

[0112] The configuration of the feature acquisition unit 140 will be described with reference to Fig. 24. The feature acquisition unit 140 includes an image division unit 141, an overlapping block determination unit 142, a feature calculation unit 143, and a feature separation unit 144. A training image DB 194 is stored in the storage unit 190.

[0113] ***Explanation of Operation*** Transfer learning will be explained based on Figs. 25 and 26. Transfer learning is a process included in the image recognition method. Training image DB 194 stores one or more training images that show objects that fit into a new class (new class objects). Correct answer data is attached to the training images. The correct answer data indicates a label that indicates the new class object and the position of the new class object in the training image. Steps S501 to S517 are executed for each training image.

[0114] In step S501, the image dividing unit 141 divides the training image into a plurality of image blocks.

[0115] In step S502, the overlapping block determination unit 142 determines which image blocks contain new class objects from among the multiple image blocks. An image block containing a new class object is referred to as an overlapping block. The method for determining overlapping blocks is the same as the method in the third embodiment.

[0116] In step S503, the feature calculation unit 143 calculates the feature of the overlapping block. The calculated feature is referred to as an overlapping block feature. The method for calculating the overlapping block feature is the same as the method for calculating the matching feature in the third embodiment.

[0117] In step S504, the feature separating unit 144 separates the overlapping block features into classification features and position estimation features. The method for separating the overlapping block features is the same as the method for separating the target features in the second embodiment.

[0118] In step S511, the probability estimation unit 131 estimates the probability that the new class object conforms to each of the multiple existing classes based on the classification features. The estimated probability is referred to as the estimated probability. The method for estimating the estimated probability is the same as the method in the second embodiment.

[0119] In step S512, the additional feature calculation unit 132 calculates additional features using the estimated probability of the new class object for each of the multiple existing classes and the matching features for each of the multiple existing classes. The method for calculating the additional features is the same as the method in the second embodiment.

[0120] In step S513, the feature combining unit 133 combines the additional feature with the classification feature. The feature obtained in this way is called a combined feature.

[0121] In step S514, the new class estimation unit 134 estimates the matching probability of the new class object for the new class based on the combined feature. That is, the new class estimation unit 134 estimates the probability that the new class object matches the new class. The method for estimating the matching probability is the same as the method in the second embodiment.

[0122] In step S515, the object position estimation unit 135 estimates the position of the new class object in the training image based on the feature for position estimation. The method for estimating the object position is the same as the method in the second embodiment.

[0123] In step S516, the loss value calculation unit 136 calculates the estimated matching probability and the loss value of the estimated object position for the correct answer data.

[0124] The loss value is calculated, for example, as follows. First, the loss value calculation unit 136 calculates the difference between the accuracy probability (100%) and the estimated matching probability, and the difference between the position of the new class object indicated in the correct data and the position of the estimated new class object. Then, the loss value calculation unit 136 multiplies the calculated differences together. The value calculated by the multiplication becomes the loss value.

[0125] In step S517, the parameter update unit 137 updates the parameters of the pre-transfer learning model 201 (particularly the parameters of the output layer 220) based on the loss value.

[0126] At this time, the parameter update unit 137 updates the parameters of the pre-transfer learning model 201 so as to minimize the loss value.

[0127] ***Effects of Embodiment 5*** According to embodiment 5, it is possible to create an image recognition model that can detect the type (class) and position of a target object with high accuracy, even when a small number of training images are used during transfer learning.

[0128] The fifth embodiment aims to enable image recognition of a new target object "α" using a small number of training images. In the fifth embodiment, three classes, "β, γ, and δ," are pre-trained in an image recognition model before transfer learning. "α" and "β" are classes with similar image features. To create an image recognition model for image recognition of "α," it is appropriate to perform transfer learning using an image recognition model before transfer learning that has been trained with many training images of "β," which have image features similar to "α." However, there are cases where the image recognition model before transfer learning has trained with many training images of "γ and δ" other than "β." In this case, the bias in the training images creates a bias, reducing the learning efficiency during transfer learning of "α." Furthermore, creating a new image recognition model before transfer learning that has been trained with many training images of "β" is costly. Therefore, an intervention process is performed during transfer learning to add and combine image features of "β," thereby increasing the image features of "β." This reduces the bias in the training images included in the image recognition model before transfer learning. Furthermore, since the bias of the training images is reduced, the learning efficiency during transfer learning is improved, and an image recognition model that can newly detect "α" can be created even with a small number of training images.

[0129] *** Supplementary Information of the Embodiment *** The hardware configuration of the image recognition device 100 will be described with reference to Fig. 27. The image recognition device 100 includes a processing circuitry 109. The processing circuitry 109 is hardware that realizes an inference unit 110, a matched feature calculation unit 120, and a transfer learning unit 130. The processing circuitry 109 may be dedicated hardware, or may be a processor 101 that executes a program stored in a memory 102.

[0130] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0131] The image recognition device 100 may include a plurality of processing circuits that replace the processing circuit 109 .

[0132] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0133] In this way, the functions of the image recognition device 100 can be realized by hardware, software, firmware, or a combination of these.

[0134] The image recognition device 100 is not limited to a general computer. For example, the image recognition device 100 may be an embedded device, an ECU (electronic control unit), an FPGA board, a GPU board, or the like.

[0135] Each embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. Each embodiment may be implemented in part or in combination with other embodiments. Procedures described using flowcharts, etc. may be modified as appropriate.

[0136] The "part" of each element of the image recognition device 100 may be read as a "process," a "step," a "circuit," or a "circuitry."

[0137] 100 Image recognition device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Input / output interface, 109 Processing circuit, 110 Inference unit, 111 Probability estimation unit, 112 Additional feature calculation unit, 113 Target feature calculation unit, 114 Feature combination unit, 115 Object detection unit, 116 Feature separation unit, 117 New class estimation unit, 118 Object position estimation unit, 119 Estimation result integration unit, 120 Matched feature calculation unit, 121 Image segmentation unit, 122 Overlapping block determination unit, 123 Feature calculation unit, 124 Average calculation unit, 125 Feature separation unit, 130 Transfer learning unit, 131 Probability estimation unit, 132 Additional feature calculation unit, 133 Feature combination unit, 134 New class estimation unit, 135 Object position estimation unit, 136 Loss value calculation unit, 137 parameter update unit, 140 feature acquisition unit, 141 image division unit, 142 overlapping block determination unit, 143 feature calculation unit, 144 feature separation unit, 180 feature similarity calculation unit, 181 feature similarity, 190 memory unit, 191 target image, 192 adapted feature DB, 193 existing class image DB, 194 training image DB, 199 inference result, 200 image recognition model, 201 model before transfer learning, 202 model after transfer learning, 210 input / hidden layer, 220 output layer, 221 feature separation layer, 222 feature extraction layer, 223 probability estimation layer, 224 object position estimation layer, 230 existing class image.

Claims

1. An image recognition device comprising: a probability estimation unit that estimates, as an estimated probability, the probability that a target object shown in a target image matches each of a plurality of existing classes that are a plurality of object classes; an additional feature calculation unit that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and match features calculated as features of an existing class image in which an existing class object that matches each of the plurality of existing classes is shown; a feature combination unit that calculates combined features by combining the additional features with target features that are features of the target image; and an object detection unit that estimates, based on the combined features, the match probability that the target object matches a new class that is an object class different from the plurality of existing classes, and the position of the target object in the target image.

2. An image recognition device comprising: a feature separation unit that separates object features, which are features of a target image that shows a target object, into class classification features and position estimation features; a probability estimation unit that estimates, based on the class classification features, the probability that the target object matches each of a plurality of existing classes, which are a plurality of object classes, as estimated probabilities; an additional feature calculation unit that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and match features calculated as features of an existing class image that shows an existing class object that matches each of the plurality of existing classes; a feature combination unit that combines the additional features with the class classification features to calculate combined features; a new class estimation unit that estimates, based on the combined features, the match probability that the target object matches a new class that is an object class different from the plurality of existing classes; and an object position estimation unit that estimates the position of the target object in the target image based on the position estimation features.

3. The image recognition device according to claim 1 or claim 2, further comprising a matching feature calculation unit that divides the existing class image into a plurality of image blocks, determines image blocks among the plurality of image blocks in which the existing class object is captured as overlapping blocks, and calculates the features of the overlapping blocks as the matching features.

4. An image recognition device according to claim 1 or claim 2, comprising a matching feature calculation unit that, for each existing class image included in a plurality of existing class images, divides the existing class image into a plurality of image blocks, determines image blocks among the plurality of image blocks in which the existing class object is captured as overlapping blocks, calculates features of the overlapping blocks as image-specific features, and calculates the average of the plurality of image-specific features corresponding to the plurality of existing class images as the matching feature.

5. The image recognition device according to claim 1 or claim 2, further comprising: a matching feature calculation unit that, for each pair of a plurality of existing class images and a plurality of block sizes, divides the existing class image into a plurality of image blocks by the block size, determines image blocks among the plurality of image blocks in which the existing class object appears as overlapping blocks, calculates features of the overlapping blocks as image-specific features, and calculates, for each block size, an average of the plurality of image-specific features corresponding to the plurality of existing class images as the matching feature; and the matching probability of the target object for the new class is estimated for each block size using the matching feature corresponding to each of the plurality of block sizes.

6. An image recognition device according to claim 1 or claim 2, comprising a matching feature calculation unit that, for each existing class image included in a plurality of existing class images, divides the existing class image into a plurality of image blocks, determines image blocks among the plurality of image blocks in which the existing class object is captured as overlapping blocks, calculates features of the overlapping blocks as image-specific features, calculates feature similarity between each image-specific feature included in a plurality of image-specific features corresponding to the plurality of existing class images and image-specific features other than the image-specific features, selects two or more image-specific features by excluding image-specific features whose feature similarity is equal to or less than a threshold from the plurality of image-specific features, and calculates an average of the selected two or more image-specific features as the matching feature.

7. The image recognition device according to any one of claims 3 to 6, wherein the matching feature calculation unit calculates an overlap rate with the existing class object for each image block, and determines an image block for which the calculated overlap rate is equal to or greater than a threshold value as the overlap block.

8. The image recognition device according to any one of claims 1 to 7, further comprising a feature similarity calculation unit that calculates the similarity between the additional feature and the target feature.

9. An image recognition method comprising: estimating, as an estimated probability, the probability that a target object shown in a target image matches each of a plurality of existing classes which are a plurality of object classes; calculating additional features using the estimated probability of the target object for each of the plurality of existing classes and matching features calculated as features of an existing class image in which an existing class object that matches each of the plurality of existing classes is shown; combining the additional features with target features which are features of the target image to calculate combined features; and estimating, based on the combined features, the matching probability that the target object matches a new class which is an object class different from the plurality of existing classes and the position of the target object in the target image.

10. An image recognition method comprising: separating object features, which are features of a target image containing a target object, into classification features and position estimation features; estimating, based on the classification features, the probability that the target object matches each of a plurality of existing classes, which are multiple object classes, as estimated probabilities; calculating additional features using the estimated probability of the target object for each of the plurality of existing classes and matching features calculated as features of an existing class image containing an existing class object that matches each of the plurality of existing classes; combining the additional features with the classification features to calculate combined features; estimating, based on the combined features, the matching probability that the target object matches a new class, which is an object class different from the plurality of existing classes; and estimating the position of the target object in the target image based on the position estimation features.

11. An image recognition program for causing a computer to execute the following steps: a probability estimation process for estimating, as an estimated probability, the probability that a target object shown in a target image matches each of a plurality of existing classes which are a plurality of object classes; an additional feature calculation process for calculating additional features using the estimated probability of the target object for each of the plurality of existing classes and matching features calculated as features of an existing class image showing an existing class object that matches each of the plurality of existing classes; a feature combination process for calculating combined features by combining the additional features with target features which are features of the target image; and an object detection process for estimating, based on the combined features, the matching probability that the target object matches a new class which is an object class different from the plurality of existing classes, and the position of the target object in the target image.

12. An image recognition program that causes a computer to execute the following steps: a feature separation process that separates object features, which are features of a target image that shows a target object, into class classification features and position estimation features; a probability estimation process that estimates, as estimated probabilities, the probability that the target object matches each of a plurality of existing classes, which are multiple object classes, based on the class classification features; an additional feature calculation process that calculates additional features using the estimated probability of the target object for each of the plurality of existing classes and match features calculated as features of an existing class image that shows an existing class object that matches each of the plurality of existing classes; a feature combination process that combines the additional features with the class classification features to calculate combined features; a new class estimation process that estimates, based on the combined features, the match probability that the target object matches a new class, which is an object class different from the plurality of existing classes; and an object position estimation process that estimates the position of the target object in the target image based on the position estimation features.

Images