Few-Shot Object Recognition Method Based on Convolutional Neural Network Metric Meta-Learning

Abstract

The present invention discloses a few-shot object recognition method based on convolutional neural network metric meta-learning. It mainly solves the problems of low recognition accuracy of the existing technology for few-shot SAR images and poor generalization ability of the feature extraction network. The solution is as follows: collect and process ground target SAR image data; build a feature extraction backbone network cascaded by four convolutional groups; use the training set to train the network and iteratively update the parameters by the stochastic gradient descent method and the backpropagation algorithm to obtain a trained feature extraction backbone network and prototype representations of images in different category training sets; calculate the Euclidean distance between the feature space of the test set samples and each category of prototype representations, and find the prototype representation with the smallest Euclidean distance from the test image to obtain the category of the input test image, thus completing the recognition of few-shot objects. The present invention effectively improves the network recognition accuracy and generalization ability in the few-shot case, and can be used for the classification of ground remote sensing images by airborne and spaceborne SARs.

Description

Technical Field

[0001] The present invention belongs to the technical field of target recognition, and particularly relates to a small-sample target recognition method, which can be used for classifying ground remote sensing images by airborne and spaceborne SAR. Background Art

[0002] The term "radar" was born about eighty years ago, and the United States was the first to apply radar to the military field. The most basic function of radar is to detect the presence of a target and determine the azimuth of the target. Since the birth of radar, it has played a huge role in both military and civilian fields. Since Wiley et al. proposed the concept of synthetic aperture radar (SAR) in 1951, with the continuous development of SAR technology, SARs of various carriers have emerged in an endless stream. As an important method for earth exploration, SAR has become one of the valuable tools in the earth remote sensing imaging system. In recent decades, with the rapid development of technology, electronic information technology has profoundly changed the application of traditional radar. Due to technical limitations, radar only has the functions of detection and ranging, making radar automatic target recognition (RATR) technology an important research direction. RATR is to process and analyze the echo signals of the target and the environment where the target is located on the basis of traditional radar target detection, extract features related to the target, and judge the characteristics, types and even models of the target according to the extracted features.

[0003] As a research direction in the field of machine learning, deep learning has received extensive attention from all walks of life due to its excellent learning ability. Convolutional neural network (CNN) is an end-to-end feedforward network, including multiple network layers, which mimics the perception process of biological neurons algorithmically and has strong adaptability to image rotation, scale change, affine transformation, etc., and is particularly suitable for image information interpretation. CNN can directly operate on the original two-dimensional image, reducing the requirements for image preprocessing, and does not require manual feature extraction, avoiding the process of feature selection, and can directly output the recognition result of the input image through the final mapping. In addition, CNN also has high flexibility, and its structure can be adjusted according to task requirements, which is convenient and practical. Combining the characteristics of SAR images, introducing CNN into SAR image target recognition has become a research hotspot in the field of SARATR. However, the improvement of the accuracy of this CNN-based image analysis technology depends on a large amount of data to train the network. In fact, in general, it is impossible to provide a large amount of labeled data. Therefore, how to use a small amount of sample data to train a task-oriented model is a problem of far-reaching significance for the practical application of deep learning and is also a very important frontier research direction in the field of deep learning in recent years.

[0004] Metric learning is an idea based on deep learning proposed in recent years, which has good effects when used for few-shot object recognition. Metric learning first learns the embedding vectors of input samples through an embedding network, and then predicts the categories of samples by solving the nearest neighbor in the embedding space to achieve the purpose of classification. The prototypical network is the most representative method in metric learning. The prototypical network uses an attention mechanism to learn an embedding network using the support set, and then uses this embedding network to predict the categories of the test set. This method believes that there is a prototype representation for each category, and the embedding vectors of samples in the corresponding category should be clustered around the prototype of this category, and this prototype is obtained by taking the mean of the embedding vectors of the support set. After obtaining the class prototype based on the embedding vectors of the support set, the classification problem is transformed into the problem of finding the nearest neighbor between the embedding vector of the test sample and the class prototype in the embedding space.

[0005] The article "Snell J, Swersky K, Zemel R. Prototypical Networks for Few-shot Learning[C]. Advances in Neural Information Processing Systems, 2017: 4078-4088." first proposed the concept of the prototypical network, and the methods in recent years are all improvements based on this. The article "Lin C, Gao F. An Extension of Prototypical Networks[C]. 4th IEEE Information Technology, Networking, Electronic and Automation Control, 2020: 421-425." introduced an attention mechanism to the prototypical network to train an embedded convolutional neural network using the support set, and used this network to extract features from the input images. The model effectively utilized the feature similarity relationship between samples during the training and prediction processes, and had better effects in text classification problems and few-shot optical image recognition compared with traditional methods, and also significantly improved in terms of convergence speed and other aspects.

[0006] The patent document with the application number CN201810980846.4 discloses a few-shot target recognition method based on deep transfer metric learning. It directly extracts the first few convolutional layers of a typical deep network to construct a feature sub-network; the input is a pair of target domain images and a pair of source domain images, and the output is the above-mentioned general features; a typical feature extraction method is used to ensure the compactness of the sample description in the feature space. However, due to the need for complex forward reasoning and a relatively complex network structure, this method cannot effectively extract SAR image features, resulting in a slow convergence speed when training the model. At the same time, since this method only uses the Euclidean distance when calculating the metric between the sample feature space and the prototype representation, it cannot well describe the true metric of the feature space, resulting in a low recognition accuracy of the recognition model for a few-shot SAR target.

[0007] The patent document with the application number CN202011388919.4 discloses a few-shot target recognition algorithm based on adaptive model-agnostic meta-learning. It uses the synthetic gradient direction to optimize the few-shot recognition model, effectively improving the model generalization ability of the first-order MAML algorithm while maintaining the basic characteristics of the first-order MAML algorithm, improving the model convergence speed, and reducing the computational complexity of the algorithm. However, due to ignoring the inter-class differences and being unable to fully utilize the information provided by the samples, this method will inevitably only converge well to "easy samples" during the gradient optimization process and cannot optimize some "hard samples", resulting in a large difference between the model learning objective and the true feature distribution of the images.

[0008] The present invention solves the technical problems of slow convergence speed of the training model, low recognition accuracy of the recognition model for a few-shot target, insufficient feature extraction ability, and insufficient model generalization ability in the end-to-end ground target SAR image target recognition framework. Summary of the Invention

[0009] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and propose a few-shot target recognition method based on convolutional neural network metric meta-learning to improve the recognition and classification accuracy of the convolutional neural network metric meta-learning model and enhance the ability to obtain local features of images.

[0010] To achieve the above purpose, the technical solution of the present invention includes the following steps:

[0011] (1) Obtain N SAR images publicly available from a public website, where N>50, and divide them into a training data set, a validation data set, and a test data set according to the ratio of 6:2:2, and use these three types of data sets to form a ground target SAR image data set;

[0012] (2) Build a feature extraction backbone network:

[0013] Construct a first convolutional group formed by cascading a first convolutional layer, a BN layer, an activation function layer, and a max pooling layer in sequence;

[0014] Construct a second convolutional group formed by sequentially connecting a second convolutional layer and two residual units

[0015] Construct a third convolutional group formed by sequentially connecting a third convolutional layer and two residual units,

[0016] Construct a fourth convolutional group formed by sequentially connecting a fourth convolutional layer and three residual units,

[0017] Cascade the first convolutional group, the second convolutional group, the third convolutional group, and the fourth convolutional group to form a feature extraction backbone network, and use the sigmoid function as the loss function of this network;

[0018] (3) Based on the training dataset, use the stochastic gradient descent method to train the feature extraction backbone network, and use the validation dataset to iteratively update the parameters of the backpropagation algorithm adopted by the feature extraction backbone network. Each round of training and parameter iteration will obtain the prototype representations of different category training set images until the set maximum number of iterations is reached or the loss function converges, obtaining the trained feature extraction backbone network and the prototype representations with labels of different category training set images;

[0019] (4) Input the test dataset into the trained feature extraction backbone network to obtain the feature space of each test set image, calculate the Euclidean distance between the test set sample feature space and each type of prototype representation, and find the prototype representation with the smallest Euclidean distance from this test image. The category corresponding to this prototype representation is the category of this input test image, completing the recognition of small sample targets.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] 1. Improve the recognition accuracy of small sample targets

[0022] Due to the design of a feature extraction backbone network composed of pointwise convolution, depth convolution, residual units, activation functions, and normalization layers, the present invention can better obtain the feature information of images without changing the size of the input matrix, and improve the recognition accuracy of small sample targets on the premise of ensuring the recognition accuracy of large-scale data.

[0023] 2. Enhance the generalization ability of the model

[0024] In the process of testing, the present invention calculates the loss of orthogonal features by processing the test set samples, and calculates the cross-entropy loss by processing the validation set samples. These two losses can more fully describe the feature space of the test images, thereby having better feature extraction capabilities for both the global and local features of the images, improving the ability to obtain local features of the images, enabling the present invention to train a network with similar or even better feature extraction effects with fewer training samples, effectively enhancing the generalization ability of the feature extraction backbone network, and thus improving the accuracy of the network in recognizing small-sample targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is the overall flowchart of the implementation of the present invention;

[0026] Figure 2 is the structural diagram of the image feature extraction backbone network in the present invention;

[0027] Figure 3 is the sub-flowchart of calculating the distance loss and the cross-entropy loss in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The following will describe in detail the embodiments and effects of the present invention with reference to the accompanying drawings.

[0029] Refer to Figure 1 , the implementation steps of this example are as follows:

[0030] Step 1: Collect and process ground target SAR images.

[0031] Obtain N publicly available SAR images from a public website, where N > 50, and divide them into a training data set, a validation data set, and a test data set according to a ratio of 6:2:2, and use these three types of data sets to form a ground target SAR image data set.

[0032] This ground target SAR image data set contains a total of ten types of targets, namely: two types of tanks T62 and T72 with different sizes; infantry fighting vehicle BMP2; armored reconnaissance vehicle BRDM2; two types of armored transports BTR60 and BTR70 with different sizes; self-propelled gun 2S1; self-propelled anti-aircraft gun ZSU234; military freight truck ZIL131 and bulldozer D7.

[0033] Step 2: Build a feature extraction backbone network.

[0034] Refer to Figure 2 , the specific implementation of this step is as follows:

[0035] 2.1) Establish a first convolutional group composed of a first convolutional layer, a BN layer, an activation function layer, and a max pooling layer cascaded in sequence, where:

[0036] The size of the convolution kernel of the first convolutional layer is set to 7×7, the number of convolution kernels is 64, and the stride is 1;

[0037] The ReLU activation function is adopted, and its expression is

[0038] The pooling size of the max pooling layer is set to 3×3, and the pooling stride is 2.

[0039] 2.2) Establish a second convolutional group composed of a second convolutional layer and two residual units connected in sequence, where:

[0040] The size of the convolution kernel of the second convolutional layer is set to 3×3, the number of convolution kernels is 128, and the stride is 1;

[0041] The two residual units are both composed of a convolutional layer with a convolution kernel size of 3×3, a number of convolution kernels of 64, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected.

[0042] 2.3) Establish a third convolutional group composed of a third convolutional layer and two residual units connected in sequence, where:

[0043] The size of the convolution kernel of the third convolutional layer is set to 3×3, and the number of convolution kernels is set to 128, and the stride is 1;

[0044] The two residual units are both composed of a convolutional layer with a convolution kernel size of 3×3, a number of convolution kernels of 64, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected.

[0045] 2.4) Establish a fourth convolutional group composed of a fourth convolutional layer and three residual units connected in sequence, where:

[0046] The size of the convolution kernel of the fourth convolutional layer is set to 3×3, the number of convolution kernels is 256, and the stride is 1;

[0047] The three residual units are all composed of a convolutional layer with a convolution kernel size of 3×3, a number of convolution kernels of 128, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected.

[0048] 2.5) Cascade the first convolutional group, the second convolutional group, the third convolutional group, and the fourth convolutional group in sequence to form a feature extraction backbone network.

[0049] 2.6) Adopt the sigmoid function as the loss function of the feature extraction backbone network, and its expression is as follows:

[0050]

[0051] where x is the network parameter.

[0052] Step 3, train the feature extraction backbone network and iteratively update the network parameters:

[0053] 3.1) Generate a txt file containing all samples and class information of the training image data in the SAR ground target image dataset, and input it together with the training set image data into the feature extraction backbone network;

[0054] 3.2) Set the maximum number of iterations to T, and initialize the parameters of the network convolutional layer and pooling layer;

[0055] 3.3) Initialize the loss function;

[0056] 3.4) Use the stochastic gradient descent algorithm to update the parameters of the feature extraction backbone network. The update formula is as follows:

[0057]

[0058] where θ is the initial parameter of the feature extraction backbone network, is the gradient of θ, θ1 is the current network parameter after being updated by the stochastic gradient descent algorithm, α is the learning rate, (x i～i+n-1 ,y i～i+n-1 ) represents a randomly selected image, x i～i+n-1 represents the sample data, y i～i+n-1 represents the label of the sample image, and n represents the number of sample images used in each update;

[0059] 3.5) Use the backpropagation algorithm to back-update the parameters updated in 3.4) to make the loss function gradually converge. The prototype representation of the training set images of different categories can be obtained in each round of training and parameter iteration. The backpropagation algorithm formula is as follows:

[0060]

[0061] where θ2 is the current network parameter updated by the backpropagation algorithm, is the gradient of θ1, and J(θ1) is the loss function of the current network;

[0062] 3.6) Repeat 3.4) and 3.5) until the maximum number of iterations T is reached or the loss function converges, and obtain the trained feature extraction backbone network and the prototype representation containing labels of the training set images of different categories.

[0063] Step 4, calculate the Euclidean distance and complete the recognition of small sample targets.

[0064] Refer to Figure 3 , and the specific implementation of this step is as follows:

[0065] 4.1) Input the test data set into the trained feature extraction backbone network to obtain the feature space of each test set image;

[0066] 4.2) Perform SVD orthogonal decomposition on the feature space of the test set samples to obtain orthogonal features, and calculate its distance loss D:

[0067]

[0068] where S k represents the set of samples of the k-th category in the training set S, x i is the M-dimensional feature vector of the i-th category, y i ∈(1,…,k) represents the i-th category, f θ (x i ) is the distance calculation function of x i based on the gradient θ;

[0069] 4.3) Calculate the similarity of each category using the features of the validation set samples, refine the feature space of the test set samples using the similarity, and obtain the support matrix of these features;

[0070] 4.4) Use the similarity matrix between the validation set samples in the feature space and the prototypes to refine the support matrix to generate an embedded feature space that can effectively cluster each test set sample, and calculate its cross-entropy loss E:

[0071]

[0072] where α is the learning rate, D k' represents the distance loss of a certain category in the test set, k' is the category of the test set sample k'∈(1,…,k), k is the number of sample categories, f θ (x) is the cross-entropy calculation function of the feature vector x of the test image based on the gradient θ;

[0073] 4.5) Solve the mean of the cross-entropy loss E and the distance loss D of an image for all categories to obtain all the Euclidean distances between the feature space of the image and the prototype representations of each category;

[0074] 4.6) Sort all the Euclidean distances of a test image, find the prototype representation with the smallest Euclidean distance to it, and the category corresponding to this prototype representation is the category of the input test image;

[0075] 4.7) Repeat 4.6) until the category corresponding to each test set image is found, and complete the classification and recognition of the few-shot target.

[0076] The effects of the present invention can be further illustrated by the following simulation experiments:

[0077] I. Experimental Conditions

[0078] The modeling platform for this experiment has an Intel 8358P CPU, 32 GB of memory, an NVIDIA GeForce RTX3090 GPU, and 24G of video memory; the operating system used in the experiment is Windows 10 (64-bit);

[0079] The compilation environment used is Pycharm Community 2020.1, the deep learning framework is Pytorch 1.7.1, the GPU computing platform is CUDA 11.0, and the GPU acceleration library is cuDNN 8.0.5. In this example, the initial learning rate is set to 0.001 and the number of training epochs is 100.

[0080] II. Experimental Contents

[0081] Under the above conditions, three groups of small-sample training tasks are established. For each training task, 5, 10, and 15 samples are randomly selected from each class of samples in 10 types of ground target SAR images, and the recognition success rates of the feature extraction backbone network under different training task settings are recorded. Comparative experiments on small-sample SAR image recognition are carried out using the method of the present invention and the existing AGC network, PN fully connected network, PN-Conv4 network, and matching network in the same simulation environment. The results are shown in Table 1;

[0082] Table 1 Comparative Experimental Results of Different Networks on the Ground Target SAR Image Dataset

[0083]

[0084] As can be seen from Table 1: For the Conv4 and matching networks, the performance when there are 15 samples per class is more than 20% higher than that when there are 5 samples per class. Therefore, the Conv4 and matching networks have the ability to recognize small-sample SAR images, and as the number of samples increases, their recognition ability is significantly improved; while for AGC, due to the sample dimension being much higher than the sample quantity, the covariance matrix estimation is poor, and its performance is much lower than that of the above two networks; the PN-FC fully connected classifier has an identification accuracy of about 10% when the number of samples per class does not exceed 15, so it does not have the ability to recognize small samples. Compared with the Conv4 and matching networks, the method of the present invention has an accuracy improvement of 14%, 10%, and 6% respectively in the experiments with 5, 10, and 15 samples per class. Thus, under the conditions of small-sample SAR images of ground targets, the present invention has higher effectiveness and generalization ability and higher recognition accuracy compared with the existing AGC network, PN fully connected network, PN-Conv4 network, and matching network.

Claims

1. A few-shot object recognition method based on convolutional neural network metric meta-learning, characterized in that It includes the following steps: (1) Obtain N publicly available SAR images from public websites, where N > 50, and divide them into a training dataset, a validation dataset, and a test dataset according to a ratio of 6:2:

2. Use these three types of datasets to form a ground target SAR image dataset; (2) Build a feature extraction backbone network: Build a first convolutional group composed of a first convolutional layer, a BN layer, an activation function layer, and a max pooling layer cascaded in sequence; Build a second convolutional group composed of a second convolutional layer and two residual units connected in sequence Build a third convolutional group composed of a third convolutional layer and two residual units connected in sequence, Build a fourth convolutional group composed of a fourth convolutional layer and three residual units connected in sequence, Cascade the first convolutional group, the second convolutional group, the third convolutional group, and the fourth convolutional group in sequence to form a feature extraction backbone network, and use the sigmoid function as the loss function of this network; (3) Based on the training dataset, use the stochastic gradient descent method to train the feature extraction backbone network, and use the validation dataset to iteratively update the parameters of the backpropagation algorithm adopted by the feature extraction backbone network. Each round of training and parameter iteration will obtain the prototype representations of different category training set images until the set maximum number of iterations is reached or the loss function converges, obtaining a trained feature extraction backbone network and the prototype representations with labels of different category training set images; (4) Input the test dataset into the trained feature extraction backbone network to obtain the feature space of each test set image. Calculate the Euclidean distance between the test set sample feature space and each category prototype representation, and find the prototype representation with the smallest Euclidean distance from the test image. The category corresponding to this prototype representation is the category of the input test image, completing the recognition of small sample targets.

2. The method according to claim 1, wherein , The ground target SAR image dataset formed in step (1) contains ten types of targets, namely: two types of tanks T62 and T72 with different sizes; infantry fighting vehicle BMP2; armored reconnaissance vehicle BRDM2; two types of armored transport vehicles BTR60 and BTR70 with different sizes; self-propelled howitzer 2S1; self-propelled anti-aircraft gun ZSU234; military freight truck ZIL131 and bulldozer D7.

3. The method according to claim 1, wherein , In step (2), for the convolutional layer, activation function layer, and max pooling layer that make up the first convolutional group, their parameters are as follows: For the convolutional layer, the convolutional kernel size is 7×7, the number of convolutional kernels is 64, and the stride is 1; The activation function uses the ReLU activation function, and its expression is: For the max pooling layer, the pooling size is 3×3 and the stride is 2.

4. The method according to claim 1, wherein , In step (2), for the second convolutional layer and two residual units that make up the second convolutional group, their structural parameters are as follows: For the second convolutional layer, the convolutional kernel size is 3×3, the number of convolutional kernels is 128, and the stride is 1; For the two residual units, each is composed of a convolutional layer with a convolutional kernel size of 3×3, a number of convolutional kernels of 64, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected.

5. The method according to claim 1, wherein , In step (2), for the third convolutional layer and two residual units that make up the third convolutional group, their structural parameters are as follows: The third convolutional layer has a convolutional kernel size of 3×3, 128 convolutional kernels, and a stride of 1 Each of the two residual units is composed of a convolutional layer with a convolutional kernel size of 3×3, 64 convolutional kernels, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected together.

6. The method according to claim 1, wherein , in step (2), the fourth convolutional layer and the four residual units in the fourth convolutional group have the following structural parameters: The fourth convolutional layer has a convolutional kernel size of 3×3, 256 convolutional kernels, and a stride of 1 Each of the three residual units is composed of a convolutional layer with a convolutional kernel size of 3×3, 128 convolutional kernels, and a stride of 1, a ReLU activation function, and a BN batch normalization layer connected together.

7. The method according to claim 1, wherein , in step (2), the sigmoid function is used as the loss function of the feature extraction backbone network, which is expressed as follows: where x is the network parameter.

8. The method according to claim 1, wherein , in step (3), the feature extraction backbone network is trained to iteratively update the parameters as follows: 3a) Generate a txt file containing all samples and class information of the training image data in the SAR ground target image dataset, and input it together with the training set image data into the constructed feature extraction backbone network; 3b) Initialize the parameters of each layer of the network; 3c) Use the stochastic gradient descent algorithm to update the parameters of the feature extraction backbone network. The update formula is as follows: θ1 = θ - α·▽ θ J(θ|x i～i+n-1 ,y i～i+n-1 ) Among them, θ is the initialization parameter of the feature extraction backbone network, ▽ θ is the gradient of the parameter θ, θ1 is the current network parameter after being updated by the stochastic gradient descent algorithm, α is the learning rate, (x i～i+n-1 , y i～i+n-1 ) represents a randomly selected image, x i～i+n-1 represents the sample data, y i～i+n-1 represents the label of the sample image, and n represents the number of sample images used in each update; 3d) Use the backpropagation algorithm to back-update the parameters updated in 3c) to make the loss function gradually converge. Different prototype representations of the training set images of different classes will be obtained in each round of training and parameter iteration. The backpropagation algorithm formula is as follows: θ2 = θ1 - α·▽ θ1 J(θ1) where θ2 is the current network parameter updated by the backpropagation algorithm, and ▽ θ1 is the gradient of θ1, and J(θ1) is the loss function of the current network; 3e) Repeat 3c) and 3d) until the maximum number of iterations T is reached or the loss function converges, and obtain the trained feature extraction backbone network and the prototype representations with labels of the training set images of different classes.

9. The method according to claim 1, wherein , in step (4), calculate the Euclidean distance between the feature space of the test set samples and each class of prototype representation as follows: 4a) Perform SVD orthogonal decomposition on the features of the test set samples to obtain orthogonal features, and calculate the distance loss D; 4b) Calculate the similarity of each class using the features of the validation set samples, and refine the feature space of the test set samples using the similarity to obtain the support matrix of these features; 4c) Then refine the support matrix through the similarity matrix between the validation set samples in the feature space and the prototype to generate an embedded feature space that can effectively cluster each test set sample, and calculate its cross-entropy loss E; 4d) Solve the mean of the cross-entropy loss E and the distance loss D of each image, and this mean is the Euclidean distance between its feature space and the prototype representation.

10. The method according to claim 9, characterized in that , the calculation formulas for the distance loss D and the cross-entropy loss E are as follows: Among them, S k represents the sample set of the k-th category in the training set S, and x i is the M-dimensional feature vector of the i-th category, and y i ∈(1,…,k) represents the i-th category, and f θ (·) is the distance calculation function based on the gradient θ; D k' represents the distance loss of a certain category in the test set, k' is the category of the test set sample k'∈(1,…,k), and k is the number of sample categories.

Images