A few-shot infrared image classification method based on deep neural network

Abstract

The present invention discloses a small-sample infrared image classification method based on a deep neural network, belonging to the fields of computer vision and image classification. This method enables a deep network model pre-trained on visible light data (source domain) to be migrated and applied to the recognition of infrared images (target domain). On the premise that the number of target domain pictures is very small, the visual features and attribute features of the images are combined, and the contrastive learning method is used to fine-tune the feature extractor part in the deep network model, so as to maintain high performance in the recognition of infrared images in the target domain. This method can not only save the manpower and time costs of development, but also be easy to implement and has high practicability.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and image classification, and particularly relates to a small-sample infrared image classification method based on a deep neural network. Background Art

[0002] In general image classification problems, a large amount of labeled data is required for each category to be classified to train a neural network. However, in most scenarios such as military, national defense, and surveillance, it is difficult to obtain labeled data. Therefore, classification needs to be performed with a small number of sample images (small samples). Small-sample image classification means that the number of visible images in each category is very small. The neural network needs to quickly learn the features of these categories through one or a few images and be able to correctly classify them. Infrared images reflect the temperature difference or radiation difference of objects. Although visible light images have good color information, they are greatly restricted by the surrounding environment. Infrared images have the advantages of being easy to identify camouflage, being able to work day and night, and being less affected by weather compared with visible light images. Therefore, they have a wide range of applications in various fields, such as detecting targets blocked by smoke or vehicles behind trees.

[0003] In small-sample classification problems, it is often necessary to pre-train a feature extractor with a base class dataset (source domain) with sufficient image quantity. The image categories in this base class dataset do not include the categories that need to be identified in the current small-sample task. Generally, the adopted method is that the base class is consistent with the image type of the predicted class in the small-sample task, and only the categories are different between the two datasets. However, for infrared images, there is no base class with consistent type or a feature extractor pre-trained with a large-scale dataset available. Therefore, only visible light images can be used as the base class. The domain difference between visible light images and infrared images is very large. The test recognition performance of models trained on a large number of visible light datasets on infrared datasets is very poor, and the features learned on visible light cannot be effectively transferred to infrared images.

[0004] To address this problem, the feature extractor pre-trained with visible light as the base class can be fine-tuned through contrastive learning techniques, and intra-domain contrast and inter-domain contrast are realized with out-of-domain images, enabling the feature extractor to better extract the image features in the infrared domain and perform accurate classification. To make full use of the known image information in the small-sample task, the visual features and attribute features of infrared images are further combined, and contrastive learning is performed with the combined features, so that the network can learn as much knowledge as possible and has better generalization ability. Summary of the Invention

[0005] The present invention aims to overcome the deficiencies of the prior art and provides a small-sample infrared image classification method based on a deep neural network. This method enables a deep network model pre-trained on visible light data (source domain) to be migrated and applied to the recognition of infrared images (target domain). On the premise that the number of images in the target domain is very small, by combining the visual features and attribute features of the images and using the method of contrastive learning, the feature extractor part in the deep network model is fine-tuned to maintain high performance in the recognition of infrared images in the target domain.

[0006] The specific technical solution of the present invention is as follows:

[0007] A small-sample infrared image classification method based on a deep neural network, comprising the following steps:

[0008] 1) Preparation of training and test data: Three sets of data are required, namely ① a large visible light image dataset as the source domain data; ② infrared images for small-sample tasks as the target domain data; ③ an image dataset that is not the target domain as a supplement to negative sample data during contrastive learning fine-tuning. For each round of training and testing, the images in ② need to be divided. Randomly select n images from each category as the small-sample training data, and the remaining images are used for testing. The datasets in ① and ③ are both used in the training phase. Only the training data in ② needs to be labeled with both category labels and attribute labels, and the remaining datasets only need to be labeled with category labels.

[0009] 2) Construction and supervised training of the recognition network in the source domain: Use a residual network (Kaiming He, Xiangyu Zhang, and et.al., “Deep residual learning for image recognition,” in Computer Vision and Pattern Recognition, 2016, pp. 770–778.) to construct a deep network model, the structure of which includes two parts: a feature extractor and a classifier. The feature extractor extracts the feature representation of the input visible light image, and the classifier uses the feature representation to generate the recognition result. Use the labeled visible light images in the source domain to train this deep network in a supervised manner.

[0010] 3) Fine-tuning the feature extraction model on the target domain: For the feature extractor of the source domain deep network model obtained in step (2), fine-tuning is performed using the labeled training data (including class labels and attribute labels) of the target domain and the unlabeled data of the non-target domain. The labeled training data of the target domain uses visual features combined with attribute features as the final features, and the unlabeled data of the non-target domain uses only visual features as the final features. A loss function based on the contrast learning mechanism is used for domain adaptation to fine-tune the feature extractor in the deep network model of the source domain, so that the feature extractor can finally be transferred from the source domain to the target domain.

[0011] Construction and supervised training of the recognition network on the source domain, specifically including the following steps:

[0012] 2-1) Use a residual network to construct the recognition network, which includes two parts: a feature extractor and a classifier. The feature extractor is composed of several convolutional blocks to extract the feature representation of the input visible light image, and the classifier is composed of fully connected layers to generate recognition results using the feature representation;

[0013] 2-2) Use the visible light images of the source domain to perform supervised network training on the deep network. Specifically, calculate the classification cross-entropy loss function in batches, and optimize the classification cross-entropy loss function through the gradient descent algorithm to obtain the deep network model of the source domain. The calculation formula of the classification cross-entropy loss function is as follows:

[0014]

[0015] In the formula, L(·) represents the value of the cross-entropy loss function, f represents the feature extractor, h represents the classifier, represents the mathematical expectation, x s represents the source domain visible light image data matrix participating in training in a single batch, y s represents the corresponding class label matrix of x s X s represents the set of source domain visible light image matrices, Y s represents the set of corresponding class label matrices of X s ∑ represents the summation operator, K represents the number of classification categories, and log(·) represents the logarithmic operation. represents the indicator function. When k = y s the function output is 1, otherwise it is 0.

[0016] Fine-tuning the feature extraction model on the target domain, specifically including the following steps:

[0017] 3-1) Loading model parameters. Construct a feature extraction network for the target domain data with the same structure as the feature extractor in the source domain, and make the two feature extractors share the same parameter weights. Use the parameter weights of the feature extractor trained on the source domain data in (2-2) as the initial value to fine-tune the network model on the target domain;

[0018] 3-2) Constructing training samples in contrastive learning fine-tuning. For the training images on the target domain, take one image as the anchor point, and select the positive sample from the images with the same category as this anchor point in the target domain training data. The negative sample consists of two parts. One part is selected from the images with different categories from this anchor point in the target domain training data, and the other part is selected from the unlabeled data in the non-target domain. The number of available images in this part is much larger than the training images on the target domain, which can help the entire fine-tuning process;

[0019] 3-3) Constructing the features of training samples during fine-tuning. Combine the visual features and attribute features of the training samples on the target domain as the final features for using contrastive learning in the fine-tuning stage. The samples in the non-target domain only use the visual features as the final features. The visual features are obtained by extracting through the feature extractor in step 3-1). The attribute features are first manually labeled to obtain an attribute label, and then the attribute label passes through a fully connected layer to make it an attribute feature with the same dimension as the visual features. Add the visual features and attribute features in a certain proportion as the final features of the training samples during the fine-tuning stage. The calculation formula for the final features is as follows:

[0020] z i =λ vis f(x i )+λ att A i

[0021] In the formula, z represents the final features of the training samples on the target domain, f represents the feature extractor, A represents the attribute features, λ vis represents the visual feature coefficient, λ att represents the attribute feature coefficient, (·) i represents the sample information with index i;

[0022] 3-4) Calculating the contrastive loss during training. Use the contrastive learning mechanism to perform domain adaptation by optimizing the contrastive loss function and reduce the deviation of the feature representation domains of the feature extractors in the source domain and the target domain. The goal of contrastive learning is to make the feature similarity of positive sample pairs higher and the feature difference of negative sample pairs larger. The formula for the contrastive loss function is as follows:

[0023]

[0024] In the formula, L con(·) represents the value of the contrast loss function, f represents the feature extractor, i represents the anchor point, I represents the number of target domain images participating in fine-tuning, P(·) represents the positive sample of the anchor point, A(·) represents the positive and negative samples of the anchor point, z represents the final feature vector, τ represents the temperature coefficient, exp(·) represents the exponential operation, log(·) represents the logarithmic operation, ∑ represents the summation operator, |·| represents the number of samples in the data set, and (z1·z2) represents the inner product operation of vectors;

[0025] 3-5) Optimize the loss function. Optimize the contrast loss function in step 3-4) through the gradient descent algorithm to obtain the final feature extractor fine-tuned for the target domain.

[0026] The beneficial effects of the present invention are:

[0027] The present invention is a classification method for small-sample infrared images based on a deep neural network. When applying the deep network model pre-trained on visible light data (source domain) to infrared images (target domain), under the premise that the number of target domain images is very small, by combining the visual features and attribute features of the images and using the method of contrast learning, the feature extractor part in the deep network model can be fine-tuned, so that the infrared image recognition on the target domain maintains high performance. This method has the following characteristics:

[0028] 1. Fine-tune the network of the target domain, combine visual features and attribute features, and make the model learn small-sample data more fully;

[0029] 2. Use a domain adaptation method based on the contrast learning mechanism to reduce the deviation of the feature representation domains of the feature extractors in the source domain and the target domain.

[0030] 3. The images used as supplementary data for negative samples in contrast learning on non-target domains do not require class labels, reducing a lot of manual annotation work.

[0031] 4. The method is simple and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a specific implementation flowchart of the present invention.

[0033] Figure 2 It is an example of an infrared image in an embodiment of the present invention.

[0034] Figure 3 It is a structural diagram of an infrared image classification network.

[0035] Figure 4 It is a flowchart for testing the recognition performance of the model.

[0036] FIG. 5(a) and FIG. 5(b) are schematic diagrams of the visualization results of the method without fine-tuning and the feature extraction model obtained by this method in the embodiments of the present invention, respectively. Detailed implementation manners

[0037] The following further describes the specific implementation manners of the present invention in combination with the accompanying drawings and technical solutions.

[0038] The present invention provides a classification method for small-sample infrared images based on a deep neural network. The specific implementation process is as Figure 1 shown. The method of this embodiment includes the following steps:

[0039] 1) Preparation of training and test data: Three groups of data are required, namely ① a large visible light dataset as the source domain data; ② infrared images for small-sample tasks as the target domain data; ③ an image dataset of non-target domains as a supplement to negative sample data during contrast learning fine-tuning. For each round of training and testing, the images in ② need to be divided. Randomly select n images from each category as the training data for small samples, and the remaining images are used to test the results of the technical solution proposed by the present invention. The datasets in ① and ③ are both used in the training stage. Only the training data in ② needs to be labeled with both category labels and attribute labels, and the remaining datasets only need to be labeled with category labels. The specific steps are as follows:

[0040] 1-1) Collect a group of infrared images of different types. This group of infrared images contains 5 categories, namely ground facilities, trucks, tanks, airplanes, and armored vehicles, with 40 images in each category. Examples of infrared images are as Figure 2 shown;

[0041] 1-2) Label the attributes of this group of infrared images. 12 attributes are set, namely gun barrels, tracks, armor, engines, tires, cargo boxes, cockpits, houses, tents, wings, tail fins, and runways. If an image has an attribute, it is labeled 1, and the remaining items are labeled 0. Each image will have a 12-dimensional 0-1 vector representation as the attribute label;

[0042] 1-3) During algorithm design and testing, arbitrarily select five images and their attribute labels from each category in this group of infrared images as the training data for fine-tuning, and the remaining images as the test data;

[0043] 1-4) The publicly available dataset ImageNet-800 (J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, "ImageNet: A large-scale hierarchical image database," 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255.) was used as the source domain data for pre-training the deep network model. Images in the non-target domain were selected from the publicly available miniImageNet dataset (J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, "ImageNet: A large-scale hierarchical image database," 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255.).

[0044] 2) Construction and supervised training of the recognition network on the source domain: A deep network model was constructed using a residual network, and its structure included a feature extractor and a classifier. The specific structure is as shown in the upper half of the dashed box. The feature extractor extracts the feature representation of the input visible light image, and the classifier generates the recognition result using the feature representation. The deep network was trained in a supervised manner using the labeled visible light images in the source domain. The specific steps are as follows: Figure 3 The recognition network was constructed using a residual network, including a feature extractor and a classifier. The feature extractor consists of several convolutional blocks and extracts the feature representation of the input visible light image. The classifier consists of fully connected layers and generates the recognition result using the feature representation. The residual network used in this embodiment is ResNet-50, which is used as the backbone network for pre-training on the source domain. ResNet-50 is a general network structure, and its specific structure can be referred to the existing literature (Kaiming He, Xiangyu Zhang, and et al., "Deep residual learning for image recognition," in Computer Vision and Pattern Recognition, 2016, pp. 770-778.);

[0045]

[0046] 2-2) Use the visible light images in the source domain to perform supervised network training on the deep network. Specifically, calculate the classification cross-entropy loss function in mini-batch mode, and optimize the classification cross-entropy loss function through the gradient descent algorithm to obtain the deep network model of the source domain. The calculation formula of the classification cross-entropy loss function is as follows:

[0047]

[0048] In the formula, L(·) represents the value of the cross-entropy loss function, f represents the feature extractor, h represents the classifier, represents the mathematical expectation, x s represents the source domain visible light image data matrix participating in training in a single batch, y s represents the corresponding category label matrix of x s X represents the set of source domain visible light image matrices, Y s represents the set of corresponding category label matrices of X s represents X s The corresponding category label matrix set, ∑ represents the summation operator, K represents the number of classification categories, K is 800 in this embodiment, and log(·) represents the logarithmic operation. represents the indicator function. When k = y s , the function output is 1, otherwise it is 0.

[0049] 3) Fine-tuning of the feature extraction model on the target domain: For the feature extractor of the source domain deep network model obtained in step 2), use the labeled training data in the target domain and the unlabeled data in the non-target domain for fine-tuning. The labeled training data in the target domain uses visual features combined with attribute features as the final features, and the unlabeled data in the non-target domain only uses visual features as the final features. Use the loss function based on the contrast learning mechanism for domain adaptation, and fine-tune the feature extractor in the source domain deep network model, so that the feature extractor can finally migrate from the source domain to the target domain. The specific steps are as follows:

[0050] 3-1) Loading of model parameters. As shown in the lower half of the dashed box, construct a feature extraction network with the same structure as the feature extractor in the source domain for the target domain data, and make these two feature extractors share the same parameter weights. Use the parameter weights of the feature extractor trained on the source domain data in step 2-2) as the initial value to perform fine-tuning of the network model on the target domain; Figure 3

[0051] ​3-2) Construct training samples for contrastive learning fine-tuning. For training images in the target domain, one image is used as an anchor, and positive samples are selected from images of the same category as the anchor in the training data on the target domain. Negative samples are composed of two parts: one part is selected from images of a different category from the anchor in the training data on the target domain, and the other part is selected from unlabeled data in the non-target domain. This part can obtain far more images than the training images in the target domain, which can help the entire fine-tuning process;

[0052] 3-3) Construct features of the training samples during fine-tuning. The visual features and attribute features of the training samples on the target domain are combined as the final features of contrastive learning in the fine-tuning stage. The samples on the non-target domain only use visual features as the final features. The visual features are extracted by the feature extractor in step 3-1), and the attribute features are first obtained by manual annotation in step 1-2) to obtain an attribute label, and then the attribute label is passed through a fully connected layer to become an attribute feature with the same dimension as the visual feature. In this embodiment, this fully connected layer converts a 12-dimensional vector into a 2048-dimensional vector. The visual features and attribute features are added in a certain ratio as the final features of the training sample fine-tuning stage. The calculation formula for the final feature is as follows:

[0053] z i =λ vis f(x i )+λ att A i

[0054] In the formula, z represents the final features of the training samples in the target domain, f represents the feature extractor, A represents the attribute features, and λ vis Represents the visual feature coefficient, in this embodiment, λ vis is 0.6, λ att Represents the attribute characteristic coefficient, in this embodiment, λ att is 0.4, (·) i Represents the sample information with index i;

[0055] 3-4) Calculate the contrastive loss for training. Using a contrastive learning mechanism, domain adaptation is performed by optimizing the contrastive loss function to reduce the domain deviation between the source and target domain feature extractors. Metric learning is used to represent the similarity between the final features of the samples. The goal of contrastive learning is to make the feature similarity of positive sample pairs higher and the feature difference of negative sample pairs greater. The contrastive loss function formula is as follows:

[0056]

[0057] Where, L con(·) represents the value of the contrast loss function, f represents the feature extractor, i represents the anchor point, I represents the number of target domain images participating in fine-tuning, P(·) represents the positive sample of the anchor point, A(·) represents the positive and negative samples of the anchor point, z represents the final feature vector, τ represents the temperature coefficient, exp(·) represents the exponential operation, log(·) represents the logarithmic operation, ∑ represents the summation operator, |·| represents the number of samples in the data set, and (z1·z2) represents the inner product operation of vectors;

[0058] 3 - 5) Optimize the loss function. Optimize the contrast loss function in (3 - 4) through the gradient descent algorithm to obtain the final feature extractor fine-tuned for the target domain.

[0059] 4) Performance test of the feature extraction model: The remaining images on the target domain after removing the infrared images used for few-shot training are used for testing. Calculate the similarity between samples by means of metrics, and calculate the recognition accuracy to test the performance of the finally obtained feature extraction model. The specific process is as Figure 4 shown. The test process of this embodiment is as follows: Randomly select five training images for each category. The training images all obtain visual features through the fine-tuned feature extractor, and are mapped to the sample space after combining with the attribute features. A class center will be formed for each category according to the five training images. Let the test images also obtain visual features through the fine-tuned feature extractor and be directly mapped to the sample space. The nearest class center model is considered to be its category and compared with the true label. Use the T-SNE graph (L.J.P. van der Maaten and G.E.Hinton, "Visualizing High-Dimensional Data Using t-SNE." Journal of Machine Learning Research 9 (Nov), 2008, pp. 2579 - 2605.) to visualize the feature extraction models obtained by the method of the present invention and the method without fine-tuning respectively, as shown in Figures 5(a) and 5(b). As shown in Figure 5(a), it is the visualization result of the method without fine-tuning, and Figure 5(b) is the visualization result of the method of the present invention. The method without fine-tuning does not distinguish clearly between classes. After fine-tuning by the method of the present invention, the discrimination effect is better for some classes, and the features within the class are also more compact.

[0060] It can be seen from the visualization results that the fine-tuning technical solution proposed by the present invention is effective for recognition on the target domain.

Claims

1. A few-shot infrared image classification method based on a deep neural network, characterized in that It includes the following steps: 1) Training and test data preparation: Three sets of data are required, namely ① visible light image dataset, as the source domain data; ② infrared images for few-shot tasks, as the target domain data; ③ image dataset of non-target domain, as the supplement of negative sample data for contrastive learning fine-tuning; For each round of training and testing, the images in ② need to be divided. Randomly select n images from each category as the training data for few-shot, and the remaining images are used for testing results. The datasets in ① and ③ are both used in the training stage; Only the training data in ② needs to be labeled with both category labels and attribute labels, and the remaining datasets only need to be labeled with category labels; 2) Construction and supervised training of the recognition network on the source domain: Use a residual network to construct a deep network model, the structure of which includes two parts: a feature extractor and a classifier; The feature extractor extracts the feature representation of the input visible light image, and the classifier uses the feature representation to generate the recognition result; Use the labeled visible light images in the source domain to train this deep network in a supervised manner; 3) Fine-tuning of the feature extraction model on the target domain: For the feature extractor of the source domain deep network model obtained in step 2), use the labeled training data of the target domain and the unlabeled data of the non-target domain for fine-tuning; The labeled training data of the target domain uses visual features combined with attribute features as the final features, and the unlabeled data of the non-target domain only uses visual features as the final features. Use the loss function based on the contrastive learning mechanism for domain adaptation to fine-tune the feature extractor in the deep network model of the source domain, and finally enable the feature extractor to migrate from the source domain to the target domain.

2. The small-sample infrared image classification method based on a deep neural network according to claim 1, wherein The fine-tuning of the feature extraction model on the target domain in step 3) specifically includes the following steps: 3-1) Model parameter loading: Construct a feature extraction network with the same structure as the feature extractor of the source domain for the target domain data, and make these two feature extractors share the same parameter weights. Use the parameter weights of the feature extractor trained on the source domain data in step 2) as the initial value to perform fine-tuning of the network model on the target domain; 3-2) Construction of training samples in contrastive learning fine-tuning: For the training images on the target domain, take one image as the anchor point, and the positive sample is selected from the images with the same category as this anchor point in the training data of the target domain; The negative sample consists of two parts. One part is selected from the images with different categories from this anchor point in the training data of the target domain, and the other part is selected from the unlabeled data of the non-target domain; 3-3) Construction of the features of training samples during fine-tuning: Combine the visual features and attribute features of the training samples on the target domain as the final features for contrastive learning in the fine-tuning stage, and only use visual features as the final features for the samples on the non-target domain; The visual features are obtained by extracting through the feature extractor in step 3-1). First, an attribute label is obtained by manual annotation for the attribute features, and then the attribute label passes through a fully connected layer to make it an attribute feature with the same dimension as the visual features; Add the visual features and attribute features in a certain proportion as the final features in the fine-tuning stage of the training samples; The calculation formula for the final features is as follows: z i = λ vis f(x i ) + λ att A i where z represents the final feature of the training samples in the target domain, f represents the feature extractor, A represents the attribute feature, and λ vis represents the visual feature coefficient, and λ att represents the attribute feature coefficient, and (·) i represents the sample information with index i; 3-4) Calculate the contrastive loss for training: Using the contrastive learning mechanism, domain adaptation is performed by optimizing the contrastive loss function to narrow the deviation of the feature representation domains of the source domain and target domain feature extractors; the goal of contrastive learning is to make the feature similarity of positive sample pairs higher and the feature difference of negative sample pairs larger; the formula for the contrastive loss function is as follows: where L con (·) represents the value of the contrastive loss function, f represents the feature extractor, i represents the anchor point, I represents the number of target domain images participating in fine-tuning, P(·) represents the positive sample of the anchor point, A(·) represents the positive and negative samples of the anchor point, z represents the final feature vector, τ represents the temperature coefficient, exp(·) represents the exponential operation, log(·) represents the logarithmic operation, ∑ represents the summation operator, |·| represents the number of samples in the data set, and (z1·z2) represents the inner product operation of vectors; 3-5) Optimize the loss function: Optimize the contrastive loss function in step 3-4) through the gradient descent algorithm to obtain the final feature extractor fine-tuned for the target domain.

Images