Image recognition device and image recognition method

Abstract

To determine that an unlearned image in an unknown class belongs to the unknown class.SOLUTION: An image recognition apparatus 1 which performs open-set recognition using a neural network includes an unknown determination unit 10 which determines whether an input image 3 belongs to an unknown class on the basis of a trained model 6 trained with learning data that belongs to a known class so as to output a result of inference for classifying the input image 3, and distances between Gauss distributions of classes calculated from feature quantities of learning data in a multi-dimensional feature space 7 formed in an intermediate layer of the trained model 6 and feature quantities of the input image 3, and corrects the result inferred by the trained model 6 as necessary.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition device and an image recognition method that perform open set recognition using a neural network. [Background technology]

[0002] Open set recognition is an object recognition method that not only recognizes trained objects but also identifies untrained objects as unknown. Most object recognition methods currently under development assume that all input data is known. This type of method is called closed set recognition. Closed set recognition does not consider the input of untrained data. Therefore, when untrained data is input, it is recognized as one of the trained objects. This invention makes it possible to extend closed set recognition to open set recognition with minimal changes to the architecture of closed set recognition. This invention can be applied to fields such as home service robots, which are expected to operate in constantly changing environments.

[0003] One of the most cutting-edge methods in open set recognition is described in Non-Patent Document 1. This method reconstructs inputs while performing classification, using a network architecture called a ladder network as a reference. If the reconstruction error is above a threshold, the input is determined to be unknown. However, since general closed set recognition does not reconstruct inputs, it is difficult to reuse existing closed set recognition models. Furthermore, since the reconstruction network must also be trained when training the recognition model, the training cost is high. Another open set recognition method that does not require reconstruction is also proposed, as described in Non-Patent Document 2, but its recognition accuracy is lower than that of the method described in Non-Patent Document 1. For these reasons, there is currently no method that can easily extend existing closed set recognition to open set recognition, has low training costs, and has high recognition performance. [Prior art documents] [Patent documents]

[0004] [Non-Patent Document 1] X. Sun, Z. Yang, C. Zhang, K.-V. Ling, and G. Peng, “Conditional gaussian distribution learning for open set recognition,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR), pp. 13480-13489, 2020. [Non-patent document 2] A. Bendale and T. E. Boult, “Towards open set deep networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1563-1572, 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] We have developed an open-set recognition method that can reuse existing closed-set recognition models, has low training costs, and has high recognition accuracy. In order to achieve high recognition accuracy with existing open-set recognition methods, it was necessary to create a reconstruction network that is not generally used in classification problems. This made it difficult to reuse existing closed-set recognition models, and the reconstruction network also needed to be trained, which resulted in high training costs.

[0006] The present invention aims to provide a method that can be easily extended to open-set recognition by fine-tuning an existing closed-set recognition model, and that is excellent in terms of learning cost and recognition accuracy. [Means for solving the problem]

[0007] In order to achieve this object, the image recognition device of the present invention performs open set recognition using a neural network, and includes: a trained model trained using training data belonging to known classes so as to output an inference result of classifying an input image; an unknown determination unit that determines whether the input image belongs to an unknown class based on the distance between the feature amounts of the input image and a Gaussian distribution of each class calculated from the feature amounts of the training data in a multidimensional feature space formed in an intermediate layer of the trained model, and corrects the inference result of the trained model as necessary; The unknown determination unit includes a distance calculation unit that calculates the distance between the Gaussian distribution of the class inferred by the trained model and the feature amount of the input image, and a determination unit that determines that the input image belongs to the unknown class when the calculated distance exceeds a predetermined threshold and corrects the inference result of the trained model. It is equipped with:

[0008] The image recognition device of the present invention may also perform distance learning on the feature quantities of the learning data in the multidimensional feature space to expand the open space of the multidimensional feature space.

[0010] Furthermore, the image recognition method of the present invention performs open set recognition using a neural network, and includes a classification step of outputting an inference result of classifying an input image using a trained model trained using training data belonging to known classes so as to output an inference result of classifying the input image; and an unknown determination step of determining whether the input image belongs to an unknown class based on the distance between the feature amount of the input image and a Gaussian distribution of each class calculated from the feature amount of the training data in a multidimensional feature space formed in an intermediate layer of the trained model, and correcting the inference result of the trained model as necessary. The unknown determination step includes a distance calculation step of calculating a distance between a Gaussian distribution of a class inferred by the trained model and a feature amount of the input image, and a determination step of determining that the input image belongs to an unknown class when the calculated distance exceeds a predetermined threshold and correcting the inference result of the trained model. It has the following characteristics.

[0011] The image recognition method of the present invention may also include a distance learning step of performing distance learning on the feature quantities of the training data in the multidimensional feature space, thereby expanding the open space of the multidimensional feature space.

[0013] In other words, distance learning is applied to the multidimensional feature space formed in the intermediate layer of a neural network commonly used in object recognition. As the learning progresses, features of the same class are positioned close to each other in the multidimensional feature space, while features of different classes are positioned farther apart. After the neural network has completed learning, the data used for learning is input into the neural network to obtain the feature quantities of each class in the multidimensional feature space. Next, Gaussian fitting is performed on the obtained feature amounts of each class to obtain a Gaussian distribution of the feature amounts of each class. In actual recognition, the class of input data is first guessed. At this time, the Mahalanobis distance is calculated between the Gaussian distribution of the guessed class and the features of the input data in the multidimensional feature space. If this Mahalanobis distance exceeds a threshold, it is detected as unknown. If it does not exceed the threshold, it is judged to be the initially guessed class. [Effects of the Invention]

[0014] According to the present invention, it is possible to detect unknown objects as unknown with high accuracy while performing normal object recognition, which was not possible with existing object recognition. Furthermore, the present invention has higher recognition performance than existing open-set recognition methods. Furthermore, compared to methods that require input reconstruction, which are considered to have high recognition performance, the present invention does not require reconstruction, so the learning cost is also lower. Furthermore, the present invention facilitates fine-tuning by utilizing existing closed-set recognition models, making it possible to detect unknown objects using already trained recognition models. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram illustrating an example of an image recognition device according to the present invention. [Figure 2] FIG. 2 is a block diagram illustrating the creation of the trained model of FIG. 1. [Figure 3] FIG. 2 is a block diagram of the trained model of FIG. 1. [Figure 4]This is to explain distance learning, where (A) is a conceptual diagram showing a multidimensional feature space before distance learning, and (B) is a conceptual diagram showing a multidimensional feature space after distance learning. [Figure 5] This is to explain Gaussian fitting. (A) is a conceptual diagram showing the multidimensional feature space before Gaussian fitting, and (B) is a conceptual diagram showing the multidimensional feature space after Gaussian fitting. [Figure 6] FIG. 2 is a conceptual diagram showing the flow of image recognition by the image recognition device of the present invention. [Figure 7] 1 is a flowchart showing a processing procedure of an image recognition method according to the present invention. [Figure 8] 1 is a flowchart showing the steps for creating a trained model. [Figure 9] FIG. 1 is a diagram showing a dataset used in validation. [Figure 10] FIG. 10 is an explanatory diagram of random trials performed in the verification. [Figure 11] This is a diagram showing a list of the experimental results of the verification. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an example of an embodiment of an image recognition device according to the present invention will be described with reference to the drawings. 1 to 6 show an image recognition device according to the present invention. The image recognition device 1 performs open set recognition using a neural network 2 and includes a trained model 6 trained using training data 5 belonging to known classes so as to output an inference result 4 for classifying an input image 3, and an unknown class determination unit 10. The unknown class determination unit 10 determines whether the input image 3 belongs to an unknown class based on a distance D between a Gaussian distribution of each class calculated from feature quantities 8 of the training data 5 in a multidimensional feature space 7 formed in an intermediate layer 2a of the trained model 6 and feature quantities 9 of the input image 3, and corrects the inference result 4 of the trained model 6 as necessary.

[0017] As shown in FIG. 2, the trained model 6 is created by supervised learning using a large amount of training data 5. The training data 5 includes images (data) 5a belonging to different classes and correct answer teacher labels 5b indicating the classes to which the images 5a belong. As shown in FIG. 3, the trained model 6 is configured by a neural network 2, and includes an input unit 11 to which an input image 3 is input, a feature extraction unit 12 that extracts features of the input image 3, a classification unit 13 that infers classification based on the extracted feature quantities 8, and an output unit 14 that converts the output of the classification unit 13 into probabilities and outputs the class with the highest probability as the inference result 4. In FIG. 6, the classification unit 13 and the output unit 14 are collectively referred to as a classifier 15. The feature extraction unit 12 and the classification unit 13 are formed in the intermediate layer 2a of the neural network 2.

[0018] The feature extraction unit 12 performs a known feature extraction method. For example, methods such as HOG (Histogram of Oriented Gradients), Harris corner detection, and SIFT (Scale-Invariant Feature Transform) can be used, but are not limited to these. Any method can be used as appropriate depending on the type of input image 3. The feature quantities 8 and 9 extracted by the feature extraction unit 12 are arranged in a multidimensional feature space 7 formed in the intermediate layer 2 a of the neural network 2.

[0019] A known classification algorithm is used for classification by the classification unit 13. For example, k-nearest neighbor method, support vector machine, etc. can be used, but the algorithm is not limited to these. For conversion into probability (certainty) in the output unit 14, for example, a Softmax function can be used, but the present invention is not limited to this.

[0020] Furthermore, in the trained model 6, distance learning is performed on the features 8 of the training data 5 in the multidimensional feature space 7, and the open space 7a of the multidimensional feature space 7 is expanded. That is, distance learning is performed so that feature values ​​8 of the same class are close to each other and feature values ​​8 of different classes are farther apart. In this embodiment, Mahalanobis distance is used as the distance between feature values ​​8, and Mahalanobis distance learning is performed as the distance learning. Furthermore, in the distance learning of this embodiment, a loss is defined that combines center loss and cross-entropy loss used in classification problems (Equation 1).

number

[0021] The advantages of center loss as a distance learning method include fewer constraints during learning and a simple algorithm. Therefore, adopting center loss makes it easy to incorporate distance learning into existing trained models. However, methods other than center loss may also be used. The loss function of center loss is shown in Equation 2.

number

[0022] The concept of distance learning is shown in Figure 4. Figure 4(A) shows the multidimensional feature space 7 before distance learning, and Figure 4(B) shows the multidimensional feature space 7 after distance learning. In the figure, circles represent the features xi (features 8) of the training data 5, and stars represent the center points Cyi of the labels (distributions of training data 5 of the same class) yi. As shown in the figure, distance learning can bring features 8 of the same class closer together and move features 8 of different classes farther apart, thereby expanding the open space 7a in the multidimensional feature space 7.

[0023] Distance learning may be performed together with supervised learning to create a trained model 6 for the purpose of classification, or after creating a trained model 6 for the purpose of classification, distance learning may be performed on the trained model 6.

[0024] Furthermore, in the trained model 6, Gaussian fitting is performed, in which the feature quantities 8 of the training data 5 in the multidimensional feature space 7 after metric learning are fitted to a multivariate Gaussian distribution to obtain a Gaussian distribution 16 for each class. That is, the training data 5 is input again into the trained model 6 created by inputting the training data 5, and the feature quantities 8 of the training data 5 in the multidimensional feature space 7 are acquired again, and Gaussian fitting is performed on this multidimensional feature space 7. The concept of Gaussian fitting is shown in Figure 5. Figure 5(A) shows the multidimensional feature space 7 before Gaussian fitting, and Figure 5(B) shows the multidimensional feature space 7 after Gaussian fitting. As shown in Figure 5, the feature quantities 8 of the training data 5 (circles in Figure 5(A)) are converted into a Gaussian distribution 16 for each class. Gaussian fitting is performed by a Gaussian fitting unit 17.

[0025] The unknown determination unit 10 includes a distance calculation unit 20 that calculates the distance D between the Gaussian distribution 16 of the class inferred by the trained model 6 and the feature 9 of the input image 3, and a determination unit 21 that determines that the input image 3 belongs to an unknown class if the calculated distance D exceeds a predetermined threshold and corrects the inference result 4 of the trained model 6.

[0026] The distance calculation unit 20 calculates the distance D when actual image recognition is performed. That is, the creation of the trained model 6 and Gaussian fitting to the multidimensional feature space 7 are performed as a preliminary step to the actual image recognition. Therefore, when actually performing image recognition of the input image 3, the Gaussian distribution 16 for each class has already been calculated, allowing for rapid processing. When the trained model 6 performs image recognition of the input image 3, the distance calculation unit 20 calculates the distance D (Mahalanobis distance) between the Gaussian distribution 16 of the class inferred by the trained model 6 and the feature 9 of the input image 3.

[0027] The determination unit 21 compares the distance D calculated by the distance calculation unit 20 with a predetermined threshold value that has been set in advance, and if the distance D exceeds the threshold value, determines that the input image 3 is classified into an unknown class and corrects the inference result 4 based on the trained model 6. That is, the unknown class is output as the final inference result 22. On the other hand, if the distance D is equal to or less than the threshold value, the inference result 4 based on the trained model 6 is output as the final inference result 22. As the threshold value, for example, a value based on the 95% confidence interval or the degree of anomaly in Hotelling's theory is adopted.

[0028] The neural network 2 and the unknown determination unit 10 are realized by causing a computer to execute a predetermined program.

[0029] Next, an image recognition method of the present invention will be described. Figures 7 and 8 show the image recognition method. The image recognition method performs open set recognition using a neural network 2, and includes a classification step S51 in which a trained model 6 trained using training data 5 belonging to known classes is used to output an inference result 4 for classifying an input image 3, and an unknown determination step S52 in which a Gaussian distribution for each class calculated from feature quantities 8 of the training data 5 in a multidimensional feature space 7 formed in an intermediate layer 2a of the trained model 6 is used to determine whether the input image 3 belongs to an unknown class based on a distance D between the feature quantities 9 of the input image 3 and a Gaussian distribution for each class calculated from feature quantities 8 of the training data 5 in the multidimensional feature space 7 formed in the intermediate layer 2a of the trained model 6, and the distance D between the feature quantities 9 of the input image 3, and the unknown class determination step S52 in which the inference result 4 of the trained model 6 is corrected as necessary.

[0030] In addition, the image recognition method includes a distance calculation step S53 in which, in the unknown determination step S52, a distance D between the Gaussian distribution 16 of the class inferred by the trained model 6 and the feature 9 of the input image 3 is calculated, and a determination step S54 in which, if the calculated distance D exceeds a predetermined threshold, the input image 3 is determined to belong to an unknown class and the inference result 4 of the trained model 6 is corrected.

[0031] In the image recognition method, first, a trained model 6 is created as preparation for recognizing an input image 3 (step S61 in FIG. 8). The trained model 6 is created by supervised learning using training data 5 (step S62). During this learning, distance learning is performed on the feature quantities 8 of the training data 5 in the multidimensional feature space 7 (step S63), thereby expanding the open space 7a of the multidimensional feature space 7. This makes it possible to more easily and accurately perform Gaussian fitting of the feature quantities 8 of the training data 5 in the multidimensional feature space 7, and also enables more accurate image recognition using the trained model 6.

[0032] Note that the distance learning step S63 may be performed together with the supervised learning step S62 in the step S61 of creating the trained model 6, as shown in FIG. 8, or may be performed after the step of creating the trained model 6.

[0033] Thereafter, the training data 5 is input again into the trained model 6, and the feature quantities 8 of the training data 5 in the multidimensional feature space 7 are acquired (step S64). In this state, distance learning has already been performed, so the acquired feature quantities 8 of the training data 5 are placed in positions after distance learning.

[0034] Next, the Gaussian fitting unit 17 fits the feature quantities 8 of the training data 5 in the multidimensional feature space 7 to a multivariate Gaussian distribution to obtain a Gaussian distribution 16 for each class (step S65). This creates a trained model 6, completing preparations for recognizing the input image 3.

[0035] For example, if image data of the numbers "4," "5," and "6" from the MNIST (Modified National Institute of Standards and Technology) database are used as training data 5, a trained model 6 is created that classifies the image data of the numbers "4," "5," and "6" (step S61). The features 9 of the input image 3 appear in a multidimensional feature space 7, clustered to some extent for each class, i.e., for each type of number (FIG. 4A). Distance learning is then performed (step S63), expanding the open spaces 7a between clusters of feature values ​​8 for each class (FIG. 4B). The training data 5 is then input again into the trained model 6, and the features 8 of the training data 5 are acquired in the multidimensional feature space 7 (step S64). Gaussian fitting is then performed on the features 8 of the training data 5 in the multidimensional feature space 7 (step S65), resulting in a Gaussian distribution 16 for each class of "4," "5," and "6."

[0036] Then, when an input image 3 is input to the trained model 6, image recognition is performed. That is, when the input image 3 is input to the trained model 6, a feature extraction unit 12 extracts features of the input image 3, and a classification unit 13 infers the classification of the input image 3 based on the extracted features. This inference result 4 is output from an output unit 14.

[0037] Meanwhile, the unknown determination unit 10 calculates the distance D between the Gaussian distribution 16 of the class inferred by the trained model 6 and the feature 9 of the input image 3 (step S53). If the calculated distance D exceeds a predetermined threshold, it is determined that the input image 3 is classified into an unknown class, and the inference result 4 by the trained model 6 is corrected. That is, the unknown class is output as the final inference result 22 (step S54). If the distance D is equal to or less than the threshold, the inference result 4 by the trained model 6 is output as the final inference result 22 (step S54).

[0038] In the above example of the numbers "4," "5," and "6," when an image of the number "5" is input as input image 3 to trained model 6, "5" is a known class, and class 5 is output as inference result 4 of trained model 6. Meanwhile, unknown determination unit 10 calculates distance D between Gaussian distribution 16 of class 5 and feature 9 of input image 3. Now, class 5 is a known class, and the calculated distance D is less than or equal to the threshold. Therefore, unknown determination unit 10 outputs class 5 as final inference result 22 (processing of known classes in FIG. 6).

[0039] Next, consider the case where an image of the number "7" is input to the trained model 6 as the input image 3. The image of the number "7" is an image of an unknown class that was not used in training the trained model 6 and should be determined as the unknown class. However, since the trained model 6 is a trained model 6 that performs closed-set recognition and was trained using training data 5 of the numbers "4," "5," and "6," it classifies the input image 3 of the number "7" into one of the classes "4," "5," or "6." Now, consider the case where the image is classified as class 5. In this case, the unknown determination unit 10 calculates the distance D between the Gaussian distribution 16 of class 5 and the feature 9 of the input image 3 of the number "7." However, since the classes are different, the two are far apart, and the calculated distance D exceeds the threshold. Therefore, the unknown determination unit 10 determines that the input image 3 of the number "7" should be classified into the unknown class and corrects the inference result 4 by the trained model 6. That is, the unknown class is output as the final inference result 22 (processing of the unknown class in Figure 6).

[0040] As described above, the image recognition device 1 and image recognition method of the present invention are capable of detecting unknown objects as unknown with high accuracy while performing normal object recognition, something that was not possible with existing object recognition. Furthermore, the recognition performance is higher than that of existing open-set recognition methods. Furthermore, compared to methods that require input reconstruction, which are considered to have high recognition performance, the present invention does not require reconstruction, so the learning cost is also lower. Furthermore, the present invention facilitates fine-tuning by utilizing existing closed-set recognition models, making it possible to detect unknown objects using already trained recognition models.

[0041] The image recognition device 1 and image recognition method of the present invention are particularly suitable for image recognition in industrial robots, domestic robots, etc., but are not limited to these. [Example]

[0042] Verification was carried out to confirm the performance of the image recognition device 1 of the present invention. Only some classes of an existing public dataset were trained, and verification was carried out using all classes during testing. Macro-F1 was used as the evaluation index. When training trained model 6, some classes in the dataset were trained. For example, Figure 9 shows an example when the dataset is MNIST. Six of the 10 classes in the dataset were trained. Specifically, the six classes, class 0, class 1, ..., class 5, were used as training classes. During testing, the four classes that were not trained were used as unknown classes, and inference and evaluation were performed.

[0043] The index of the ratio of known classes to unknown classes is called openness, and in this example it was 13.4%. This index is shown in Equation 3.

number

[0044] For the verification, random trials were performed. This is shown in Figure 10. The known classes and unknown classes were randomly divided and evaluation was performed five times. When training the model, a portion of the classes in the dataset, specifically 6 out of 10 classes, were trained. In the validation, the four classes that were not learned were treated as unknown classes, and inference and evaluation were performed. The ratio of known classes to unknown classes was set to 13.4%.

[0045] The experimental results are shown in Figure 11. The validation conditions were 300 training epochs and 512 dimensions of the feature space for unknown detection. Evaluation was based on the average and variance of macro-F1 after five validation runs. MNIST, SVHN, and CIFAR10 were used as datasets. In all cases, the index for the ratio of known classes to unknown classes was set to 13.4%. For comparison, similar experiments were conducted using existing methods (discriminative model-based: Softomax method, Openmax method; reconstruction model-based: CROSR method, GDFR method, CGDL method). As is clear from the experimental results in Figure 11, the image recognition method of the present invention (the Proposed method in Figure 11) achieved the best results for MNIST, SVHN, and CIFAR10. [Explanation of symbols]

[0046] 1. Image recognition device 2. Neural Networks 2a Hidden layer of neural network 3. Input image 4. Inference results of trained model 5. Training data 6 Pre-trained models 7 Multidimensional feature space 7a Open Space 8 Features of training data 9 Input image features 10 Unknown judgment section 11 Input section 12 Feature Extraction Unit 13 Classification section 14 Output section 15 Classifier 16 Gaussian distribution 17 Gauss fitting section 18 Calculation reference point 19 Reference point calculation section 20 Distance calculation unit 21 Judgment section 22 Final inference result

Claims

1. An image recognition device that performs open set recognition using a neural network, a trained model trained using training data belonging to known classes so as to output an inference result for classifying an input image; an unknown determination unit that determines whether the input image belongs to an unknown class based on a distance between a Gaussian distribution of each class calculated from the feature amounts of the training data in a multidimensional feature space formed in an intermediate layer of the trained model and the feature amount of the input image, and corrects an inference result of the trained model as necessary; The unknown determination unit includes a distance calculation unit that calculates a distance between the Gaussian distribution of the class inferred by the trained model and the feature amount of the input image, and a determination unit that determines that the input image belongs to an unknown class when the calculated distance exceeds a predetermined threshold and corrects the inference result of the trained model. An image recognition device comprising:

2. 2. The image recognition device according to claim 1, wherein distance learning is performed on the feature quantities of the learning data in the multidimensional feature space, thereby expanding the open space of the multidimensional feature space.

3. An image recognition method for performing open set recognition using a neural network, comprising: a classification step of outputting an inference result of classification of the input image using a trained model trained using training data belonging to a known class so as to output an inference result of classification of the input image; an unknown determination step of determining whether the input image belongs to an unknown class based on a distance between a Gaussian distribution of each class calculated from the feature amounts of the training data in a multidimensional feature space formed in an intermediate layer of the trained model and the feature amount of the input image, and correcting an inference result of the trained model as necessary; The unknown determination step includes a distance calculation step of calculating a distance between the Gaussian distribution of the class inferred by the trained model and the feature amount of the input image, and a determination step of determining that the input image belongs to an unknown class if the calculated distance exceeds a predetermined threshold and correcting the inference result of the trained model. An image recognition method comprising:

4. The image recognition method described in Claim 3, characterized in that it includes a distance learning step in which distance learning is performed on the features of the training data in the multidimensional feature space, and the open space of the multidimensional feature space is expanded.

Images