SAR multi-angle inter-class difference feature fusion identification method based on small samples

Abstract

The multi-angle inter-class difference feature fusion identification method comprises the following steps of extracting multi-angle features and different-class features; carrying out inter-class similarity calculation on the multi-angle features and the heterogeneous features; extracting inter-class difference features between the multi-angle features and the heterogeneous features; performing feature enhancement on the inter-class difference features according to inter-class similarity; designing a multi-view weighted loss function according to the inter-class similarity; and constructing a training data set according to network characteristics. According to the method, the inter-class difference features between the multi-angle features and the heterogeneous features can be extracted, the inter-class difference features are enhanced according to the similarity between the multi-angle features and the heterogeneous features, the features learned by the network are enriched, the inter-class distance is increased, the data amplification effect is improved, and finally the model recognition performance is improved.

Description

Technical Field

[0001] The present invention relates to signal and information processing technology, and in particular to a SAR multi-angle inter-class difference feature fusion recognition method based on small samples. Background Art

[0002] Synthetic aperture radar (SAR) is widely used in military and civilian fields because it is not affected by the external natural environment. Therefore, automatic recognition of SAR targets has become a current research hotspot. With the development of deep learning, it has become the mainstream to realize automatic classification and recognition by autonomously learning target features. Although the application field of SAR is becoming more and more extensive, considering the limitations of cost and actual situation, it is still difficult to obtain SAR images. SAR image recognition faces the problem of inaccurate recognition under small sample conditions. When the number of samples is scarce, it is difficult for current algorithms to achieve good recognition results. Currently, small sample SAR target recognition is usually achieved through data enhancement. Traditional methods include flipping, translation, cutting, mirroring and random erasing. For example, Random Erasing Data Augmentation proposed by Zhu Zhong et al. randomly selects a rectangular area in the erased image and erases its pixels with random values. In this process, training images with different degrees of occlusion are generated. Such methods can achieve a certain data augmentation effect, but the data augmentation effect is very limited when the original data is scarce, and the network training effect is not ideal. With the rise of generative adversarial networks (GANs), some scholars use generative networks to generate new data. For example, the DCGAN network proposed by Alec Radfor et al. combines GAN with CNN to achieve image generation and target classification. Chongxuan Li et al. proposed the Triple GAN network, which consists of a generator, a classifier, and a discriminator. It generates new data to improve network training performance and achieve target classification. However, the training of generative networks also requires a large amount of data. When the data is small, the quality of the generated new data cannot be guaranteed, resulting in low recognition performance.

[0003] Due to the particularity of SAR imaging mechanism, very small changes in azimuth and elevation angles will cause great changes in target features, which brings challenges to target recognition. To address this problem, a multi-angle SAR fusion recognition algorithm was proposed. This type of algorithm inputs SAR images of the same target at different angles into the network in parallel, extracts multi-scale features at different convolutional layers, and realizes multi-angle SAR image fusion recognition. Since the feature extraction and fusion of different angles are performed at different feature layers, the data between each parallel input can be repeatedly combined and used to achieve the effect of data amplification. The research results show that multi-angle image recognition can provide more classification information. At present, many related literatures based on multi-angle recognition have been published in the field of SAR ATR. For example, the DCNN network proposed by Jifang Pei et al. has multiple parallel inputs and achieves good recognition results with a small number of samples. This method can improve the accuracy of target recognition to a certain extent, and in theory, the performance of the model can be improved by increasing the number of parallel inputs. However, since multiple feature fusion operations between parallel input images will lead to great feature redundancy, too many parallel branches will cause overfitting of the model, resulting in poor generalization performance of the model. In addition, when the amount of data is very small, multiple repeated combinations will cause more serious feature redundancy and overfitting.

[0004] Based on the above analysis, the current SAR image recognition has the following two problems: 1. The current data enhancement method based on small samples cannot obtain high-quality effective data; 2. The multi-angle fusion recognition algorithm will produce a large feature redundancy when the samples are very small, resulting in poor recognition effect. Summary of the invention

[0005] The present invention will be described in detail below with reference to the accompanying drawings.

[0006] In view of the shortcomings of the prior art, the present invention proposes a SAR multi-angle inter-class difference feature fusion recognition method based on a small sample, which specifically includes the following steps:

[0007] Step 1: Multi-angle heterogeneous feature extraction;

[0008] The network has four parallel inputs, of which input angle 1, input angle 2 and input angle 3 are SAR images of the same category at different angles, which are used to extract the image features of the three inputs; other categories of inputs are used to extract heterogeneous image features;

[0009] Step 2: Multi-angle heterogeneous feature similarity calculation;

[0010] Calculate the cosine similarity between multi-angle features and heterogeneous features, and enhance the extracted inter-class features according to the size of the similarity;

[0011]

[0012] In the formula, Represents the extracted multi-angle features, where i represents the i-th input and l represents the features extracted by the l-th convolution layer; represents heterogeneous features, where 4 represents the fourth input and l represents the features extracted by the lth convolution layer; express and The inter-class similarity matrix of ||·||2 represents the L2 norm; the mean S of the similarity matrix is ​​taken as the measure of the similarity between the two features;

[0013]

[0014] In the formula, sim represents the similarity matrix, m,n represents the mth row and nth column in sim, S m,n represents the similarity of each channel between the extracted multi-angle features and heterogeneous features; t represents the dimension of each feature; S represents the average similarity between the multi-angle features and heterogeneous features after averaging;

[0015] Each round of deep learning network training is divided into many batches, each batch has Bt images, and the feature similarity of all images in each batch is calculated to obtain a similarity set Normalize the elements in the collection to obtain a 0-1 distribution;

[0016]

[0017]

[0018] In the formula, It represents the similarity set obtained by calculating all the data in the batch, S1…S Bt are the inter-class similarities of the corresponding multi-angle features and heterogeneous features in each batch; Bt represents the number of images in each batch, S q represents the qth element in the set, α q Indicates S q The normalized value, S min express The minimum value, S max express The maximum value in ;

[0019] Step 3: Extract and enhance inter-class difference features;

[0020] After calculating the similarity of the input multi-angle features and heterogeneous features, the inter-class difference features of the two are extracted, and then the inter-class difference features with large similarity are enhanced, and the inter-class difference features with small similarity are directly extracted, and then the subsequent multi-angle feature fusion is performed; the details are as follows;

[0021] Assumptions represents the features extracted from the i-th input layer l, Represents the features extracted from the lth layer of heterogeneous input; extracts inter-class features by feature subtraction;

[0022]

[0023] In the formula, g i l Represents the inter-class difference features extracted from the lth layer of the i-th multi-angle input; after extracting the inter-class difference features, according to the calculated similarity α i The size of is used to enhance the inter-class features; Formula (6) corresponds to Figure 1 The “-” in the middle means the feature extraction of inter-class difference;

[0024]

[0025] In the formula, Represents the enhanced inter-class features, using the similarity size S obtained by direct calculation i As a basis for judgment, the normalized value α i As a weight to enhance the inter-class features; β represents the judgment threshold. i When S < β, it indicates that the multi-angle features and heterogeneous features are not similar, and no feature enhancement is performed. i When ≥β, it indicates that the two are highly similar. At this time, according to the similarity α calculated after normalization i Enhance its features and increase the distance between classes;

[0026] After feature enhancement, a convolutional layer is introduced to reduce the feature dimension;

[0027]

[0028] In the formula, f 2D (·) represents a two-dimensional convolution operation, Represents the inter-class difference features obtained after convolution dimensionality reduction;

[0029] Step 4: Fusion of multi-angle and inter-class difference features;

[0030] After the inter-class features are extracted, the inter-class difference features and the extracted multi-angle features are fused;

[0031]

[0032] In the formula, represents feature concatenation, y l+1 represents the features fused at layer l+1, represents the multi-angle features extracted by the j-th (i≠j) multi-angle input in the l+1th convolution layer, Indicates the yth r Inter-class features extracted from multi-angle input;

[0033] Step 5: Loss function design;

[0034] Use inter-class similarity as a penalty term to adjust the final prediction loss and propose a multi-view weighted loss function; add a new classifier at the end of the network to predict the category of heterogeneous input and calculate the classification loss; use the similarity between multi-view and heterogeneous features as weights to adjust the heterogeneous loss;

[0035] The total loss of the network is the sum of the multi-view loss and the heterogeneous loss:

[0036]

[0037] In the formula, λ represents the weight for calculating heterogeneous loss, p r represents the true category of heterogeneous input, Indicates the predicted category of heterogeneous input;

[0038] Use inter-class similarity as a penalty term to adjust the final prediction loss;

[0039]

[0040] Where S view1,j Indicates input view Figure 1 and the inter-class similarity of input heterogeneity, S view2,j Indicates input view Figure 2 and inter-class similarity with heterogeneous inputs;

[0041]

[0042] In the formula, y r represents the true value of the image, represents the predicted value of the image, u represents the number of images, Indicates multi-angle prediction loss;

[0043] The joint loss function of the final network is:

[0044]

[0045] Step 6: Construction of training data set;

[0046] Taking the MSTAR dataset as an example, each category is divided into three angle combinations according to the azimuth: (0-120), (120-240), (240-360), a total of three sets, represented as Where w, z, d represent the number of images in the set, and e represents the e-th category. By repeatedly combining data, theoretically, w×z×d combinations of training data can be obtained for network training.

[0047] However, since SAR images of the same target at different angles have more feature redundancy, a heterogeneous feature extraction branch, i.e., other category input, is added, and each combination of multi-angle data can be randomly combined with other categories; each combination of multi-angle data can be randomly combined with all other remaining categories. Assuming that the data set has a total of x categories, there are a total of w×z×d×(w+z+d)(x-1) data combinations available to the network.

[0048] The advantages of the present invention are as follows:

[0049] Compared with DCNN and previous data enhancement algorithms, the designed multi-angle inter-class difference feature extraction network introduces a heterogeneous feature extraction branch based on multi-angle feature fusion. This branch extracts and fuses multi-angle features extracted from inputs of the same category but at different angles. This alleviates the feature redundancy caused by multi-angle feature fusion in the DCNN algorithm, enriches the features learned by the network, and improves the network's data amplification capabilities.

[0050] The high similarity between heterogeneous features and multi-angle features is an important reason for reducing the performance of the recognition algorithm. When extracting inter-class difference features, the present invention can enhance the inter-class difference features according to the inter-class similarity of heterogeneous features and multi-angle features, increase the inter-class distance, and improve the model performance.

[0051] In order to further increase the inter-class distance and reduce the inter-class similarity, the present invention adds a new classifier at the end of the network to predict the category of the heterogeneous input and calculate the classification loss. The similarity between multi-view and heterogeneous features is used as a weight to adjust the heterogeneous loss. The total loss of the network is the sum of the multi-view loss and the heterogeneous loss. By combining the loss calculation with the inter-class similarity, the network can purposefully reduce the inter-class similarity during the learning process, making the classification of similar objects more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The network structure diagram of the multi-angle inter-class difference feature fusion recognition method is shown;

[0053] Figure 2 A schematic diagram of SAR image comparison of different types of targets is shown, where (a1), (a2), and (a3) ​​represent SAR images of BMP2 infantry fighting vehicles at different angles, and (b1), (b2), and (b3) represent SAR images of BTR70 armored transport vehicles at different angles;

[0054] Figure 3 The schematic diagram of multi-angle heterogeneous feature similarity calculation and inter-class difference feature extraction and enhancement is shown;

[0055] Figure 4 The schematic diagram of data enhancement is shown;

[0056] Figure 5 The characteristic splicing principle diagram is shown;

[0057] Figure 6 The following is a visualization of the recognition results when there are 20 valid data in each category: Figure 6 (a) shows the clustering visualization results of the traditional single-input recognition network. Figure 6 (b) shows the clustering visualization results of the dual-input multi-angle fusion recognition network. Figure 6 (c) shows the clustering visualization results of the three-input multi-angle fusion recognition network. Figure 6 (d) shows the clustering visualization results of the network after introducing heterogeneous feature extraction into the three-input multi-angle fusion recognition network. Figure 6 (e) shows the clustering visualization result of the network after introducing inter-class difference feature enhancement based on heterogeneous feature extraction. Figure 6 (f) shows Figure 6 (e) The network clustering visualization results of the complete algorithm after adding the improved loss function based on the experimental results;

[0058] Figure 7 The confusion matrix diagram is shown when there are 20 valid data for each category. DETAILED DESCRIPTION

[0059] The present invention will be further described below in conjunction with embodiments and drawings.

[0060] The present invention proposes a multi-angle inter-class difference feature fusion recognition method, the network architecture is as follows Figure 1 As shown in the figure, Conv represents the convolution layer, 'S' represents the inter-class similarity calculation, '-' represents the inter-class difference feature extraction, 'E' represents the inter-class difference feature enhancement, 'Cat' represents the feature concatenation. 'fc' represents the fully connected layer. It includes the following steps:

[0061] Step 1: Multi-angle heterogeneous feature extraction.

[0062] Figure 1 The network structure diagram of the present invention is shown. Different from the traditional network input, the network proposed in the present invention has four parallel inputs, where input angle 1, input angle 2 and input angle 3 are SAR images of the same category at different angles, which are used to extract the image features of the three inputs. Other categories of inputs are used to extract heterogeneous image features (feature extraction from SAR images is known to those skilled in the art and will not be repeated).

[0063] Step 2: Multi-angle heterogeneous feature similarity calculation.

[0064] SAR images of different targets at different angles may have high similarities. Take the moving and stationary target acquisition and recognition dataset, i.e., the MSTAR dataset, as an example. This dataset was collected and released by Sandia National Laboratories in the United States and contains 10 types of targets. Figure 2 As shown, (a1), (a2), and (a3) ​​respectively represent SAR images of a BMP2 tank at different angles, and (b1), (b2), and (b3) respectively represent SAR images of a BTR70 tank at different angles, and both have high similarity. Various studies have shown that too high inter-class similarity is one of the important reasons for poor network recognition performance. Therefore, it is meaningless to extract inter-class difference features from multi-angle features and heterogeneous features with high similarity, and the obtained inter-class difference features contain very little effective information. If inter-class difference features are directly extracted from features of different categories with high similarity, it will have a negative impact on network training. Therefore, the present invention first determines the similarity between multi-angle features and heterogeneous features. Figure 1 Here, S represents the similarity calculation between multi-angle features and heterogeneous features. Figure 3 S1 represents the similarity calculation between classes.

[0065] Figure 3 In the above figure, S1 represents the inter-class similarity calculation, where "1, 2, 3, 4, 5" represent multi-angle features, and "6, 7, 8, 9" represent other category features. After calculating the similarity between multi-angle features and other category features, S2 is performed. First, the inter-class difference features of multi-angle and other categories are extracted, and then the similarity between the corresponding multi-angle features and other category features is determined. i Whether it exceeds the threshold β, if it exceeds the threshold, it is considered to be an invalid inter-class difference feature, and its feature needs to be enhanced according to formula (7). If it is less than the threshold β, the subsequent feature fusion operation is performed.

[0066] Studies have shown that cosine similarity can characterize the similarity between features. Therefore, the present invention calculates the cosine similarity between multi-angle features and heterogeneous features, and enhances the extracted inter-class features according to the size of the similarity.

[0067]

[0068] In the formula, Represents the extracted multi-angle features, where represents the i-th input, i represents the feature extracted by the l-th convolution layer. l represents heterogeneous features, where 4 represents the fourth input. Represents the features extracted by the lth layer of convolution. l represents and The inter-class similarity matrix. Represents the ||·||2 norm. Cosine similarity can represent the similarity between two features. Since the extracted features are all tensors, and cosine similarity calculates the similarity of matrices, a similarity matrix is ​​obtained after calculation. The present invention takes the mean S of the similarity matrix as a measure of the similarity between two features.

[0069]

[0070] Where sim represents the similarity matrix, sim represents the sim-th row and m-th column in m,n, and n represents the similarity of each channel between the extracted multi-angle features and heterogeneous features. m,n represents the dimension of each feature. t represents the average similarity between the multi-angle features and heterogeneous features after averaging. Formula (1) (2) (3) corresponds to Figure 1 The "S" here refers to the inter-class similarity calculation.

[0071] Each round of training of the deep learning network is divided into many batches, each batch has S images. The present invention calculates the feature similarity of all images in each batch to obtain a similarity set Bt. Then the elements in the set are normalized. The role of batch normalization is to obtain a 0-1 distribution, which facilitates the selection of weights during subsequent feature enhancement.

[0072]

[0073] In the formula, It represents the similarity set obtained by calculating all the data in the batch, S1…S Bt They are the inter-class similarities of the corresponding multi-angle features and heterogeneous features in each batch, respectively. represents the number of images in each batch, and Bt represents the Sth image in the set. q elements, q represents α q The normalized value, S q Indicates S min The minimum value, Indicates S max The maximum value in .

[0074] Step 3: Extract and enhance inter-class difference features.

[0075] After calculating the similarity between the current multi-angle features and heterogeneous features, inter-class difference features are extracted. Figure 3 As shown in S2, represents the inter-class similarity of multi-angle features and heterogeneous features, β represents the judgment threshold, when S iWhen S < β, it indicates that the multi-angle features and heterogeneous features are not similar, and no feature enhancement is performed. i When ≥β, it indicates that the two are highly similar and their features need to be enhanced to increase the distance between classes. The details are as follows.

[0076] After calculating the similarity of the input multi-angle features and heterogeneous features, the inter-class difference features of the two are extracted, and then the inter-class difference features with large similarity are enhanced, and the inter-class difference features with small similarity are directly extracted, and then the subsequent multi-angle feature fusion is performed. The details are as follows.

[0077] Assumptions Indicates The features extracted from the i-th layer of the input, l represents the heterogeneous input In order to retain the differences between multi-angle features and heterogeneous features to a greater extent, the present invention extracts inter-class features by feature subtraction.

[0078]

[0079] In the formula, g i l Represents the inter-class difference features extracted from the lth layer of the i-th multi-angle input. After extracting the inter-class difference features, according to the calculated similarity α i The size of is used to enhance the inter-class features. Formula (6) corresponds to Figure 1 The “-” in the middle refers to the extraction of inter-class difference features.

[0080]

[0081] In the formula, Represents the enhanced inter-class features. The present invention uses the similarity size obtained by direct calculation As the basis for judgment, the normalized value S i As a weight to enhance the inter-class features. i Represents the judgment threshold. When β, it indicates that the multi-angle features and heterogeneous features are not similar, and no feature enhancement is performed. When S i When <β, it indicates that the two are highly similar. At this time, according to the similarity S calculated after normalization i ≥β enhances its features and increases the distance between classes. Formula (7) corresponds to Figure 1 The "E" in the figure refers to the enhancement of inter-class difference features. In order to reduce the feature dimension and avoid the huge increase in computational complexity caused by excessive dimension, the present invention introduces a convolutional layer after feature enhancement to reduce the feature dimension.

[0082]

[0083] In the formula, f 2D(·) represents a two-dimensional convolution operation, Represents the inter-class difference features obtained after convolution dimensionality reduction.

[0084] Step 4: Fusion of multi-angle and inter-class difference features.

[0085] After the inter-class features are extracted, the inter-class difference features and the extracted multi-angle features are fused.

[0086]

[0087] In the formula, It represents feature splicing, which directly stacks two features of the same dimension together, and can retain all the original features (the splicing technology is well known to those skilled in the art, for example Figure 6 shown), Indicates that in y l+1 The features obtained by layer fusion, l+1 represents the The multi-angle features extracted by the i≠j-th convolution layer of the multi-angle input, l+1 represents the inter-class features extracted by the i-th multi-angle input. Formula (9) corresponds to Figure 1 The "Cat" in it refers to feature splicing.

[0088] Step 5: Loss function design.

[0089] In order to further increase the inter-class distance and reduce the inter-class similarity, the present invention adds a new classifier at the end of the network to predict the category of the heterogeneous input and calculate the classification loss. The similarity between multi-view and heterogeneous features is used as a weight to adjust the heterogeneous loss. The total loss of the network is the sum of the multi-view loss and the heterogeneous loss. By combining the loss calculation with the inter-class similarity, the network can purposefully reduce the inter-class similarity during the learning process, making the classification of similar objects more accurate.

[0090]

[0091] In the formula, λ represents the weight for calculating heterogeneous loss, p r represents the true category of heterogeneous input, Represents the predicted category of heterogeneous input (the real category of the image can be obtained through the label file, and the predicted category of the image is the prediction result of the proposed method, which is known to those skilled in the art). In order to further increase the inter-class distance and reduce the inter-class similarity, the present invention uses the inter-class similarity as a penalty term to adjust the final prediction loss. Since network optimization is achieved by minimizing the loss, combining loss settlement with inter-class similarity can reduce the inter-class similarity while reducing the loss.

[0092]

[0093] Where S view1,j Indicates input view Figure 1 and the inter-class similarity of input heterogeneity, S view2,j Indicates input view Figure 2 and inter-class similarity with heterogeneous inputs.

[0094]

[0095] In the formula, y r represents the true value of the image, represents the predicted value of the image (the true value of the image can be obtained through the label file, and the predicted value of the image is the prediction result of the proposed method, which is known to those skilled in the art), u represents the number of images, Represents the multi-angle prediction loss.

[0096] The joint loss function of the final network is:

[0097]

[0098] Step 6: Construction of training data set.

[0099] Depend on Figure 1 It can be seen from the backbone network shown that the network proposed by the present invention has four input branches at the same time. These four input branches realize feature fusion at different layers. Therefore, different input combinations will eventually result in different features. Therefore, the network has a strong data amplification effect. In the multi-angle branch, in order to avoid feature redundancy caused by too high similarity of SAR images with similar angles. Figure 4 As shown, taking the MSTAR dataset as an example, each category is divided into three angle combinations according to the azimuth: (0-120), (120-240), (240-360), a total of three sets, respectively represented as Where w, z, d represent the number of images in the set, and e represents the e-th category. By repeatedly combining data, theoretically, w×z×d combinations of training data can be obtained for network training.

[0100] However, since SAR images of the same target at different angles have more feature redundancy, the present invention adds a heterogeneous feature extraction branch, i.e., other category input, and each combination of multi-angle data can be randomly combined with other categories. This method can not only improve the feature redundancy caused by the fusion of multi-angle features of the same category, but also bring about a powerful data amplification effect. Figure 4As shown, each combination of multi-angle data can be randomly combined with all other remaining categories. Assuming that the data set has x categories, the network proposed by the present invention has w×z×d×(w+z+d)(x-1) types of available data combinations. Taking MSTAR data as an example, assuming that there are ten categories, 30 images in each category, and a total of 300 SAR images as original data, assuming that three angle sets are evenly distributed in each category, that is, w=z=d=10, then the available data combinations for each category are 10×10×10×30×9=270000 types, and there are 2700000 types in ten categories, which is 9000 times larger than the available data of the basic 300 original images. Considering that such a large number of repeated combinations of data will inevitably cause feature redundancy, the actual data amplification effect is not so good, but even if the amplification effect is half, it still has a huge advantage over other data amplification methods.

[0101] Experimental verification

[0102] (1) Construction of MSTAR training dataset

[0103] In order to verify the performance of the algorithm proposed in the present invention under small sample conditions, the data with a pitch angle of 17° is used as the training set in the MSTAR data, and the data with a pitch angle of 15° is used as the test set, with the ratios of 10:3, 3:1, and 2:1 respectively. 100, 60, and 20 original data are selected for each category of the training set, and the selected data are randomly selected from the overall data. The data distribution is shown in the table. The total data volume is 1000, 600, and 200 SAR images respectively. 30, 20, and 10 original data are selected for each category of the test set as the test set, with a total number of 300, 200, and 100. Among them. The network structure proposed in the present invention can achieve a great data utilization effect with a very small amount of data, but if the data is reused too much, it will cause feature redundancy, which will cause model overfitting. Therefore, the data selected for different original training data is different. The traditional single-input network model obviously does not have a data amplification effect, so the training data and the original data are the same. Although the multi-angle fusion recognition network itself can realize data reuse, multiple fusions of the same category will cause obvious feature redundancy. Table 1 shows a comparison table of available training data for different networks, and the specific experimental data distribution is shown in the table.

[0104] Table 1 Training data used in the experiment

[0105]

[0106] As can be seen from the table, the number of training sets and test sets of MSTAR data in the multi-angle network model of the present invention are (5000, 1000), (1800, 600), and (1000, 400) respectively. Because when the original data is large enough, the original features it possesses are rich enough, so when the data is reused, more data combinations can be obtained without worrying about the negative impact of feature redundancy; if the original features are few, then when the data is reused, too many combinations will produce serious feature redundancy.

[0107] The multi-angle heterogeneous feature fusion network proposed in the present invention introduces heterogeneous feature extraction on the basis of multiple angles. The introduction of heterogeneous feature extraction enables heterogeneous features to be fused with multi-angle features at will. In this case, the data reuse capability of the model is greatly improved. And the feature redundancy caused by reuse is also greatly reduced. Finally, the number of training set test sets of the algorithm proposed in the present invention is (15000, 2500), (9000, 1700), (2600, 600).

[0108] (2) Experimental environment setup

[0109] In this section, the present invention will discuss the experimental results of the proposed method under different data sets and different training data amounts. The SGD optimizer is used for training, and the weight decay is 0.000001. The momentum is set to 0.9. A total of 200 epochs are trained, and the learning rate is 0.0001. The experimental environment is pytorch2.0, and the graphics card is a Tesla V100 graphics card. All experiments are carried out in the same environment.

[0110] (3) Experimental results analysis

[0111] 1) Analysis of SOC experimental results on the MSTAR dataset

[0112] Tables 2 and 3 respectively show the ablation experimental results of the method proposed in the present invention when the original training samples are 100 and 20 SAR images of each category. V1-1 represents the experimental results of the traditional single-input target recognition network, V1-2 represents the experimental results of adding the proposed heterogeneous feature fusion recognition network on the basis of V1-1, V1-3 represents the experimental results of adding the inter-class feature enhancer on the basis of V1-2, and V1-4 represents the experimental results of adding the multi-angle-heterogeneous joint loss function on the basis of V1-3. V2-1 represents the experimental results of the multi-angle target recognition network with two inputs, V2-2 represents the experimental results of adding the proposed heterogeneous feature fusion recognition network on the basis of V2-1, V2-3 represents the experimental results of adding the inter-class feature enhancer on the basis of V2-2, and V2-4 represents the experimental results of adding the multi-angle-heterogeneous joint loss function on the basis of V2-3. V3-1 represents the experimental results of the 3-input multi-angle target recognition network, V3-2 represents the experimental results of V3-1 plus the proposed heterogeneous feature fusion recognition network, V3-3 represents the experimental results of V3-2 plus the inter-class feature enhancer, and V3-4 represents the experimental results of V3-2 plus the multi-angle-heterogeneous joint loss function.

[0113] Table 2 Ablation experiment: recognition performance under different experimental conditions for the original training samples with 100 SAR images per class 1, 2, 3 represent inter-class difference feature fusion, inter-class difference feature enhancer, and multi-angle-heterogeneous joint loss function respectively.

[0114]

[0115] Table 3 Ablation experiment: recognition performance under different experimental conditions for the original training samples with 20 SAR images per class 1, 2, 3 represent heterogeneous feature fusion, differential feature enhancer, and multi-angle-heterogeneous joint loss function, respectively.

[0116]

[0117] The experimental results in Table 2 are the recognition results of 100 images of each category. The recognition rate of the complete version of the method proposed in the present invention is as high as 99.53%. Taking V3-1 as the benchmark, the recognition rate of the proposed heterogeneous feature fusion structure, namely the V3-2 version, is 99.27, which is 2.27% higher than the comparison algorithm. The recognition rate of the V3-3 version obtained after heterogeneous feature enhancement based on the heterogeneous feature fusion network is 99.33, which is 0.06% higher. The recognition rate of the complete version reaches 99.53, which is 0.2% higher than V3-3.

[0118] Table 3 shows the experimental results of the method proposed in the present invention when there are 20 images in each category. The recognition rate of the complete version is 96.80%, which is only 2.73% lower than the recognition rate of 100 images in each category, and the amount of data used for training is only 1 / 5 of the original. Compared with the recognition rate of 86.08% of the basic version, the advantages of the three improvement measures proposed in the present invention are more obvious. After introducing heterogeneous feature fusion on the basis of V3-1, the recognition rate increased by 7.19%, reaching 93.27%. The introduction of heterogeneous feature enhancers increased the recognition rate by 2.33%, reaching 95.60%. The recognition rate of the final complete version is 96.80%, which is 10.72% higher than the baseline version.

[0119] like Figure 6 As shown, the present invention conducts a visual analysis of the test results of the validation set of MSTAR data of 20 images in each category, and (a), (b), (c), (d), (e), and (f) correspond to the experimental results of V1-1, V2-1, V3-1, V3-2, V3-3, and V3-4 in Table IV respectively. It can be clearly seen from the visualization results that the features of each category of the comparison methods V1-1, V2-1, and V3-1 on the 1500 validation sets are seriously mixed and not well separated. The methods V3-2, V3-3, and V3-4 proposed in the present invention have better visualization effects of clustering of each category, which proves the effectiveness of the proposed method. Figure 7 This is the confusion matrix of the method proposed in the present invention on the validation set. The confusion matrix shows the details of the recognition errors. From the data of the confusion matrix, it can be seen that the real label categories of the images with recognition errors are relatively concentrated.

[0120] 2) Analysis of EOC experimental results on the MSTAR dataset

[0121] There are great differences in SAR imaging results under different working conditions. These changes will affect the performance of the recognition model. The MSTAR dataset contains datasets under different working conditions. In order to better illustrate the superior performance of the algorithm proposed in this invention, a more complex scenario, namely extended operating conditions (EOCs), is used to verify the performance of the model. In the SOC experiment, the training images and test images were taken at depression angles of 17° and 15°, respectively. The EOC working conditions are more complex, specifically including three different configuration modes: EOC-D, EOC-C and EOC-V. The training samples and test samples are randomly selected from different configuration modes, and the specific distribution is shown in Table 4.

[0122] Table 4 Algorithm recognition effect under different original training data under EOC working conditions

[0123]

[0124] Table 4 shows the recognition results of EOC when the training samples of each category are 20, 60, and 100 images respectively. The experiment is conducted under the full configuration. When there are 20 images in each category, the minimum recognition rate is 93.24%, indicating that the proposed method has good robustness in practical applications, can cope with different scene changes, has low dependence on data, and can obtain good training results with less data. When there are 100 images in each category, the recognition rate is above 98%, which fully demonstrates that the proposed algorithm can meet actual needs.

[0125] 3) Analysis of experimental results of different algorithms on the MSTAR dataset

[0126] In this section, the present invention uses different methods to compare with the method proposed in the present invention under different training samples, mainly based on four types of algorithms for comparison, namely 1) traditional algorithm; 2) data enhancement algorithm; 3) multi-angle fusion recognition algorithm. The specific algorithm model is given below.

[0127] Traditional recognition algorithms include PCA+SVM and Adaboost. From the experimental results in Table 5, it can be seen that this type of algorithm has high data requirements and poor model performance under small sample conditions. The highest recognition rate is only 96.43% when there are 20 images in each category. Even if all MSTAR data is used, the recognition rate is only 94.32%. There are many ways to enhance data, such as traditional enhancement methods such as rotation, flipping, cropping, etc. At present, the GAN network is deeply loved by researchers because of its excellent performance. Therefore, this paper selects the GAN network as a comparison algorithm. Four GAN-derived recognition algorithms are mainly selected for comparative experiments. According to the experimental results in the table, compared with the traditional algorithm, the model performance is significantly improved after data enhancement. The recognition rate of 20 images in each category reaches 85.23%, and the recognition rate of 80 data reaches 94.91. However, the overall recognition rate still needs to be improved. This is because the training of the GAN network also requires a large amount of data. It is difficult to obtain a good data enhancement effect when the amount of data is small, and the quality of the generated experience data is not high. The main comparison algorithm of the present invention is the DCNN algorithm. Considering that SAR images at different angles have different features, the algorithm simultaneously inputs multi-angle SAR images into the network, so that the network can simultaneously learn the features of different angles, and different data combinations can obtain different features, so the network has a good data amplification effect. Among them, 2V-DCNN, 3V-DCNN, and 4V-DCNN represent network models with 2 inputs, 3 inputs, and 4 inputs, respectively. The more parallel input data, the better the data amplification effect. It can be seen from the experimental results that when there are 20 valid data in each category, 4V-DCNN obtains a recognition accuracy rate of 88.67%. Although it has been improved compared with the data enhancement algorithm, the recognition rate is still very low. The main reason is that the repeated combination of a small amount of data leads to huge feature redundancy, so that although there are a lot of training data in theory, feature redundancy causes the model to overfit. And finally, the algorithm proposed in the present invention has a recognition rate of up to 96.80% when there are 20 images in each category. Compared with the DCNN series algorithm, the algorithm proposed in the present invention introduces heterogeneous feature extraction, inter-class feature enhancement and joint training loss function. In the ablation experiments in Table 4-7, experiments V2-1 and V3-1 correspond to 2V-DCNN and 3V-DCNN in DCNN. According to the experimental results, when training 20 images of each category, after introducing heterogeneous feature extraction on the basis of 2V-DCNN and 3V-DCNN, the model recognition accuracy increased by 7.73% and 7.19% respectively. It can be seen that the heterogeneous feature extraction branch proposed in the present invention enables the model to learn the feature differences between each category, and effectively solves the feature redundancy caused by data reuse, greatly alleviating the overfitting of the model.

[0128] Table 5 Comparison of recognition results of different algorithms

[0129]

Claims

1. A SAR multi-angle inter-class difference feature fusion recognition method based on small samples, characterized by: The specific steps include: Step 1: Multi-angle heterogeneous feature extraction; The network has four parallel inputs, of which input angle 1, input angle 2 and input angle 3 are SAR images of the same category at different angles, which are used to extract the image features of the three inputs; other categories of inputs are used to extract heterogeneous image features; Step 2: Multi-angle heterogeneous feature similarity calculation; Calculate the cosine similarity between multi-angle features and heterogeneous features, and enhance the extracted inter-class features according to the size of the similarity; In the formula, Represents the extracted multi-angle features, where i represents the i-th input and l represents the features extracted by the l-th convolution layer; represents heterogeneous features, where 4 represents the fourth input and l represents the features extracted by the lth convolution layer; express and The inter-class similarity matrix of ||·||2 represents the L2 norm; the mean S of the similarity matrix is ​​taken as the measure of the similarity between the two features; In the formula, sim represents the similarity matrix, m,n represents the mth row and nth column in sim, S m,n represents the similarity of each channel between the extracted multi-angle features and heterogeneous features; t represents the dimension of each feature; S represents the average similarity between the multi-angle features and heterogeneous features after averaging; Each round of deep learning network training is divided into many batches, each batch has Bt images, and the feature similarity of all images in each batch is calculated to obtain a similarity set S; Normalize the elements in the collection to obtain a 0-1 distribution; In the formula, It represents the similarity set obtained by calculating all the data in the batch, S1…S Bt are the inter-class similarities of the corresponding multi-angle features and heterogeneous features in each batch; Bt represents the number of images in each batch, S q represents the qth element in the set, α q Indicates S q The normalized value, S min express The minimum value, S max express The maximum value in ; Step 3: Extract and enhance inter-class difference features; After calculating the similarity of the input multi-angle features and heterogeneous features, the inter-class difference features of the two are extracted, and then the inter-class difference features with large similarity are enhanced, and the inter-class difference features with small similarity are directly extracted, and then the subsequent multi-angle feature fusion is performed; The details are as follows; Assumptions represents the features extracted from the i-th input layer l, Represents the features extracted from the lth layer of heterogeneous input; extracts inter-class features by feature subtraction; In the formula, g i l represents the inter-class difference features extracted from the lth layer of the i-th multi-angle input; After extracting the inter-class difference features, according to the calculated similarity α i The size of is used to enhance the inter-class features; Formula (6) corresponds to the "-" in Figure 1, that is, the inter-class difference feature extraction; In the formula, Represents the enhanced inter-class features, using the similarity size S obtained by direct calculation i As a basis for judgment, the normalized value α i Use it as weight to enhance inter-class features; β represents the judgment threshold. When S i When S < β, it indicates that the multi-angle features and heterogeneous features are not similar, and no feature enhancement is performed. i When ≥β, it indicates that the two are highly similar. At this time, according to the similarity α calculated after normalization i Enhance its features and increase the distance between classes; After feature enhancement, a convolutional layer is introduced to reduce the feature dimension; In the formula, f 2D (·) represents a two-dimensional convolution operation, Represents the inter-class difference features obtained after convolution dimensionality reduction; Step 4: Fusion of multi-angle and inter-class difference features; After the inter-class features are extracted, the inter-class difference features and the extracted multi-angle features are fused; In the formula, represents feature concatenation, y l+1 represents the features fused at layer l+1, represents the multi-angle features extracted by the j-th (i≠j) multi-angle input in the l+1th convolution layer, Indicates the yth r Inter-class features extracted from multi-angle input; Step 5: Loss function design; Use inter-class similarity as a penalty term to adjust the final prediction loss and propose a multi-view weighted loss function; add a new classifier at the end of the network to predict the category of heterogeneous input and calculate the classification loss; use the similarity between multi-view and heterogeneous features as weights to adjust the heterogeneous loss; The total loss of the network is the sum of the multi-view loss and the heterogeneous loss: In the formula, λ represents the weight for calculating heterogeneous loss, p r represents the true category of heterogeneous input, Indicates the predicted category of heterogeneous input; Use inter-class similarity as a penalty term to adjust the final prediction loss; Where S view1,j represents the inter-class similarity between input view 1 and input heterogeneity, S view2,j Represents the inter-class similarity between input view 2 and input heterogeneity; In the formula, y r represents the true value of the image, represents the predicted value of the image, u represents the number of images, l mulit Indicates multi-angle prediction loss; The joint loss function of the final network is: Step 6: Construction of training data set; Taking the MSTAR dataset as an example, each category is divided into three angle combinations according to the azimuth: (0-120), (120-240), (240-360), a total of three sets, represented as Where w, z, d represent the number of images in the set, and e represents the e-th category. By repeatedly combining data, theoretically, w×z×d combinations of training data can be obtained for network training. However, since SAR images of the same target at different angles have more feature redundancy, a heterogeneous feature extraction branch, i.e., other category input, is added, and each combination of multi-angle data can be randomly combined with other categories; each combination of multi-angle data can be randomly combined with all other remaining categories. Assuming that the data set has a total of x categories, there are a total of w×z×d×(w+z+d)(x-1) data combinations available to the network.

Images