SAR image scene classification method based on online gradient boosting

Abstract

The present invention discloses a SAR image scene classification method based on online gradient boosting, which is specifically implemented by adding an embedding layer at the end of CNN, dividing the embedding layer into multiple sub-embeddings, and adding a classifier after each sub-embedding; the classifier is trained by online gradient boosting during the training process, and a regressor is added between each sub-embedding, and the embedded features in different sub-embeddings are mapped to the same feature space by the regressor, and a metric function based on the Pearson correlation coefficient is learned in the space. The regressor is ignored during testing, and the prediction of each classifier is integrated by weighted voting. The SAR image scene classification method based on online gradient boosting proposed by the present invention can enable CNN to focus on learning complex SAR scene images, and also enable CNN to learn more diverse information between SAR scenes of the same category; and enable CNN to extract more discriminative features from SAR images.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a SAR image scene classification method based on online gradient boosting. Background Art

[0002] With the continuous development of SAR imaging technology, a large number of high-resolution SAR images are used for ground observation. SAR systems can observe a variety of different types of scenes, such as residential areas of different densities, skyscrapers, trees, farmland, road traffic, etc., and have high military and civilian value. Compared with the booming SAR imaging technology, SAR image interpretation technology has developed slowly. SAR image scene classification is one of the key factors in SAR image interpretation. Usually, a good SAR scene classification system mainly includes two parts: feature extraction of SAR images and training an effective and robust classifier based on the extracted features. Compared with the training of the classifier, extracting sufficiently discriminative features from SAR images is crucial for SAR to effectively distinguish different SAR scene images.

[0003] In the early days, some manual feature extraction methods such as texture descriptors and Gabor filters were usually used to extract SAR image features. However, these methods can only extract some low-level features from SAR images and need to be combined with prior knowledge, so they are difficult to apply to feature extraction of a large number of SAR images. In addition, some machine learning algorithms such as bag-of-words models can extract mid-level features from SAR images, but the disadvantage of such methods is that the amount of calculation is too large. In addition, the widespread application of deep learning technology in recent years has led to the use of many deep neural networks to extract SAR image features, such as deep belief networks (DBN), autoencoders (AE), convolutional neural networks (CNN) and their variants. Compared with the low-level and mid-level features extracted from SAR images by traditional methods, deep learning-based methods can extract deep features from SAR images to better describe SAR images. In addition, when faced with a large amount of SAR data, deep learning-based methods can also process more accurately and quickly. However, the special imaging mechanism of SAR images makes it difficult for deep neural networks to directly extract sufficiently discriminative features for SAR image classification tasks. A method is needed to extract discriminative information from SAR images for SAR image classification by combining deep learning technology.

[0004] Due to the special imaging mechanism of SAR images, there are more backscatter responses in SAR images of complex scenes compared to SAR images of simpler scenes, which leads to a more complex spatial structure of SAR images. In addition, since SAR systems are mostly airborne or satellite-borne, even SAR scene images of the same type have extremely rich information. In response to the above problems, the present invention proposes a SAR image scene classification method based on online gradient boosting and Pearson correlation coefficient. The method applies a greater weight to SAR images of complex scenes by means of online gradient boosting so that the model focuses on learning samples of this type. In addition, the present invention also introduces a metric function based on the Pearson correlation coefficient to enable the model to learn more diverse information of SAR scenes of the same category. By combining the two methods, the present invention enables CNN to learn more discriminative features for SAR image scene classification. Summary of the invention

[0005] The purpose of the present invention is to provide a SAR image scene classification method based on online gradient boosting, which enables CNN to focus on learning more complex SAR scene images through the online gradient boosting algorithm, and enables CNN to learn more diverse information between similar SAR scenes through a metric function based on the Pearson correlation coefficient, aiming to improve the accuracy of SAR image scene classification.

[0006] The technical solution adopted by the present invention is: a SAR image scene classification method based on online gradient boosting, and the specific operation steps are as follows:

[0007] Step 1: Use the random sampling method to construct the SAR data set, and use random cropping and random horizontal flipping to expand the data set; adjust the SAR scene image of 200×200 pixels to 128×128 pixels by bilinear interpolation method;

[0008] Step 2: Add an embedding layer at the end of the convolutional neural network (CNN) and divide it into multiple non-overlapping sub-embeddings that share weights. Add a classifier after each sub-embedding.

[0009] Step 3: Add a regressor between every two sub-embeddings, and add a gradient flipping layer between each regressor and the sub-embedding. The feature vectors in the two sub-embeddings are mapped to the same feature space through the regressor, and a total metric function based on the Pearson correlation coefficient is learned in the space.

[0010] Step 4: Iteratively train each classifier in step 2 by using the online gradient boosting algorithm. During forward propagation, the loss function of each classifier is weighted and accumulated to obtain the loss of the entire ensemble. The ensemble loss is combined with the metric function based on the Pearson correlation coefficient described in step 3 to obtain a new loss function.

[0011] Step 5: During back propagation, first update the weight of each classification loss according to the integrated loss described in step 4, and then the gradient flipping layer described in step 3 flips the gradient sign of the metric function based on the Pearson correlation coefficient;

[0012] Step 6: After training, the prediction of each sub-embedded component is input into the softmax function, and the probability results output by the softmax function are weighted integrated through the weighted voting method to obtain the final integrated prediction result.

[0013] The present invention is also characterized in that:

[0014] In step 1, the SAR dataset is constructed using the random sampling method. The dataset is divided into multiple small batches before training. Then the model trains these small batch training sets in sequence. When sampling each small batch, ensure that each batch contains 80 SAR scene images, and these 80 images come from 16 different scene classes, and each class contains 5 images.

[0015] Step 2 is as follows:

[0016] For the SAR dataset containing N samples in step 1 where x i represents the data sample, y i Represents the label of each sample; CNN is used to extract features φ(x i ), and pass it to the embedding layer to obtain the corresponding sub-embedding vector {f1(x i ),f1(x i ),...,f K (x i )}, and obtain the corresponding classifier based on each sub-embedding vector K is the number of sub-embeddings and classifiers, and the number of sub-embeddings and classifications is the same.

[0017] The total metric function L based on the Pearson correlation coefficient described in step 3 metric for:

[0018]

[0019] where x n ,1≤n≤N represents the nth sample in the SAR data set,λ weight Represents the weight parameter, Lpearson(i,j) Represents the metric function based on the Pearson correlation coefficient between sub-embeddings i and j. For sub-embeddings i and j, by adding a regressor between them, the embedding vector in sub-embedding j is mapped to the space where sub-embedding i is located, and the metric function based on the Pearson correlation coefficient is learned in this space:

[0020]

[0021] Among them, f i (x n ) and f j (x n ) represent the embedding vectors in sub-embedding i and j respectively; Reg(j,i) represents the regressor that maps the embedding vector in sub-embedding j to the space of sub-embedding i, d j represents the dimension of the space where sub-embedding j is located; r(·,·) represents the Pearson correlation coefficient of two feature vectors;

[0022] For vector and The Pearson correlation coefficient between the two is:

[0023]

[0024] In the present invention, f i (x n ) is equivalent to vector a, Reg(j,i)f j (x n ) is equivalent to vector b;

[0025] in, and Represent the mean of vectors a and b respectively, r(a,b) ranges from [-1,1]. When r(a,b) = 0, it means that vectors a and b have no linear relationship. In order to avoid the situation where the weight parameter in the regressor and embedding layer is 0, which causes the metric function based on the Pearson correlation coefficient to be 0, the metric function L between each two sub-embeddings is pearson(i,j) Apply constraint L weight :

[0026]

[0027] The operator [] + represents max(0,·), that is, in b T Find the maximum value between b-1 and 0; Represents the weight in the embedding layer, w i ∈W and b represent the weight and bias in the regressor Reg(·), respectively.

[0028] Step 4 uses the online gradient boosting algorithm to iteratively train the classifier; assuming that there are K classifiers in total, the prediction of each classifier is iteratively obtained by weighted summation during forward propagation; for the kth classifier, its prediction is:

[0029]

[0030] Among them, the weight η of the kth classifier k =2 / k+1; represents the prediction of the first k classifiers for the i-th SAR sample;

[0031] The loss of the entire ensemble during forward propagation is:

[0032]

[0033] Among them, w represents weight, the initial value is 1, and the weight is the weight of the kth classifier corresponding to the i-th sample; Represents the loss function of each classifier. The loss function used by each classifier in the present invention is the cross entropy loss function:

[0034]

[0035] in, <y i ,logf(x i )> represents the i-th sample label y i and logf(x i )

[0036] Combine the loss of the entire integration with the metric function based on the Pearson correlation coefficient described in step 3 to obtain a new loss function:

[0037] L total =L ensemble +λ div ·L metric (8)

[0038] Among them, div Represents the weight parameter.

[0039] In step 5, during back propagation, the new loss function L is used total Update the model weights until the L total When the loss no longer changes significantly, training stops; specifically, for the integrated loss L ensemble , calculate the residual based on the loss obtained from the ensemble prediction of the first k-1 classifiers, and use the residual value as the weight to reweight the training samples of the kth classifier;

[0040]

[0041] Among them, w represents weight, its initial value is 1, weight is the weight of the k-th classifier corresponding to the i-th sample, which is calculated by the negative gradient of the first k-1 ensemble prediction losses;

[0042] New loss function L total L metric By flipping the sign of the gradient during the back-propagation process, due to the effect of the gradient flipping layer, the process of minimizing the Pearson metric function during training will in turn reduce the correlation between the sub-embeddings, thereby improving the diversity of the embedding layer; specifically, the weight parameter θ in the sub-embedding i, j during back-propagation i ,θ j The update process is:

[0043]

[0044]

[0045] where μ represents the learning rate of sub-embeddings i and j.

[0046] Step 6: When predicting, the predictions of each classifier are integrated by weighted voting:

[0047] Where F(x i ) represents the integrated output, and α k is the weight corresponding to the prediction of the kth classifier in the integration process, calculated as:

[0048]

[0049] The beneficial effect of the present invention is that the SAR image scene classification method based on online gradient boosting of the present invention uses an ensemble learning method to make the model focus on learning SAR images containing complex scenes. In addition, the method constrains the sub-embedding by a metric function based on the Pearson correlation coefficient, thereby increasing the diversity of features in the embedding layer so that the model can learn rich intra-class information in the SAR image. Experiments show that the SAR image scene classification method based on online gradient boosting proposed by the present invention has excellent performance in the SAR image scene classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the SAR image scene classification method based on online gradient boosting of the present invention;

[0051] Figure 2It is a learning process of a metric function based on the Pearson correlation coefficient of the SAR image scene classification method based on online gradient boosting of the present invention;

[0052] Figure 3 This is the learning process of the online gradient boosting method of the SAR image scene classification method based on online gradient boosting of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention is based on the SAR image scene classification method based on online gradient boosting, such as Figure 1 The specific steps are as follows:

[0055] Step 1. In order to allow the inventive method to learn more relationships between similar and different types of SAR scene images, the data set is divided into multiple small batches before training, and then the model trains these small batch training sets in sequence. When sampling each small batch, ensure that each batch contains 80 SAR scene images, and these 80 images come from 16 different scene classes, and each class contains 5 images. The present invention constructs a total of 4,000 batches of the above type as training sets. In addition, the present invention uses random cropping and random horizontal flipping to expand the data set. And the SAR scene image with a size of 200×200 pixels is adjusted to a size of 128×128 pixels by the bilinear interpolation method.

[0056] Step 2: For the data set constructed in step 1, CNN is used to extract features φ(x i ), and pass it to the embedding layer to obtain the corresponding sub-embedding vector {f1(x i ),f1(x i ),...,f K (x i )}. And add a classifier after each sub-embedding

[0057] Step 3: Figure 2 Add a regressor between the sub-embeddings to learn the Pearson correlation coefficient-based metric function:

[0058]

[0059] where x n ,1≤n≤N represents the nth sample in the SAR data set,λ weight Represents the weight parameter. pearson(i,j)It is obtained by the regressor between each two sub-embeddings. The regressor consists of two parts: a hidden layer and a mapping layer. For sub-embedding i and sub-embedding j, the purpose of the regressor is to map the embedding vector in sub-embedding j to the space where sub-embedding i is located, and learn a metric function based on the Pearson correlation coefficient in this space:

[0060]

[0061] Among them, f i (x n ) and f j (x n ) represent the embedding vectors in sub-embedding i and j respectively; Reg(j,i) represents the regressor that maps the embedding vector in sub-embedding j to the space of sub-embedding i, d j represents the dimension of the space where sub-embedding j is located; r(·,·) represents the Pearson correlation coefficient of two feature vectors;

[0062] For vector and The Pearson correlation coefficient between the two is:

[0063]

[0064] In the present invention, f i (x n ) is equivalent to vector a, Reg(j,i)f j (x n ) is equivalent to vector b;

[0065] in, and Represent the mean of vectors a and b respectively. The value range of r(a,b) is [-1,1]. When r(a,b)=0, it means that there is no linear relationship between vectors a and b.

[0066] Step 3.1: In order to avoid the situation where the weight parameter in the regressor and embedding layer is 0, which causes the metric function based on the Pearson correlation coefficient to be 0, the metric function L between each two sub-embeddings is pearson(i,j) Apply constraint L weight :

[0067]

[0068] The operator [] + represents max(0,·), that is, in b T Find the maximum value between b-1 and 0; Represents the weight in the embedding layer, w i ∈W and b represent the weight and bias in the regressor Reg(·), respectively.

[0069] Step 4: Figure 3 The training process uses an online gradient boosting algorithm to iteratively train the classifier. When forward propagating for K classifiers, the prediction of each classifier is iteratively calculated and weighted summed. For the kth classifier, its prediction is:

[0070]

[0071] The weight η of the kth classifier is k =2 / k+1. represents the prediction of the k-th classifier for the i-th SAR sample.

[0072] Step 4.1, the loss of the entire integration during the forward propagation process is:

[0073]

[0074] Where w represents the weight, the initial value is 1, the weight is the weight of the k-th classifier corresponding to the i-th sample. Represents the loss function of each classifier. In this method, the cross entropy loss function is used:

[0075]

[0076] in, <y i ,logf(x i )> represents the i-th sample label y i and logf(x i )

[0077] Combine the loss of the entire integration with the metric function based on the Pearson correlation coefficient described in step 3 to obtain a new loss function:

[0078] L total =L ensemble +λ div ·L metric (8)

[0079] Among them, div Represents the weight parameter.

[0080] Step 5: During back propagation, use the new loss function L total Weighting the model Update until the L total When the loss no longer changes significantly, the training stops; for the integrated loss L ensemble, when backpropagating, the residual is calculated by the loss obtained by the integrated prediction of the first k-1 classifiers, and the residual value is used as the weight to reweight the training samples of the kth classifier.

[0081]

[0082] Step 5.1: The present invention updates the weight parameters When , the negative gradient calculation can be achieved by calculating the pseudo residual. Specifically, the weight The pseudo residual is updated as follows:

[0083]

[0084] Step 5.2: For L metric In the back propagation process, this method adds a gradient flip layer between the sub-embedding and the regressor, which causes the gradient sign to be flipped to negative during the back propagation process; at this time, the weight parameter θ in the sub-embedding i, j i ,θ j The update process is:

[0085]

[0086]

[0087] where μ represents the learning rate of sub-embeddings i and j.

[0088] Step 6: Integrate the predictions of each classifier using weighted voting:

[0089]

[0090] Where F(x i ) represents the integrated output, and α k is the weight corresponding to the prediction of the kth learner in the ensemble, calculated as:

[0091]

[0092] Finally, SAR image scene classification is achieved.

[0093] The effects of the present invention can be specifically illustrated through simulation experiments.

[0094] 1. Experimental conditions

[0095] The experiment in this invention mainly uses the deep learning framework PyTorch to implement the Adam algorithm to optimize the loss function during training, with an initial learning rate of 0.001 and a weight decay value of 10. -4. In addition, 0.001 is used as the initial learning rate to optimize the regressor between sub-embeddings. During the training process, the "triangular2" strategy is also used to periodically adjust the learning rate. In this strategy, the minimum learning rate is set to the initial learning rate 0.001, the maximum learning rate is set to 0.01, and the learning rate increase and decrease steps are set to 1400. In the "triangular2" strategy, the difference between the maximum learning rate and the minimum learning rate is halved after each cycle, so the difference between the maximum learning rate and the minimum learning rate will become smaller in the later stage of training. Other environment configurations of the experiment of the present invention are shown in Table 1:

[0096] Table 1 Experimental environment configuration

[0097]

[0098] 2. Experimental content

[0099] First, in order to verify the effectiveness of the experiment, a convolutional neural network is designed in this experiment, and the network is mainly composed of the deep separable structure in MobileNet. The network architecture is shown in Table 2:

[0100] Table 2 Network architecture

[0101]

[0102] Each convolutional layer is followed by a linear activation function ReLU. For the fully connected layer, a Dropout layer is also applied to it and its activation function value changes to 0 with a probability of 0.2. In the experiment, the last two layers are used as classifiers, and the rest are used as the network backbone to extract SAR image features.

[0103] In the experimental process, the present invention directly uses the above structure for SAR image scene classification and uses it as a baseline method.

[0104] In addition, the experiment of the present invention also compares whether to add a metric function based on the Pearson correlation coefficient and the impact of the sub-embedding division on the SAR image scene classification.

[0105] 3. Experimental results

[0106] In the present invention, the classification accuracy is highest when the embedding layer is divided into 4 sub-embeddings of size 32, as shown in Table 3:

[0107] Table 3 Accuracy of different methods (%)

[0108]

[0109]

[0110] As can be seen from the table, compared with the baseline, using two classifiers to learn through the online gradient boosting algorithm can improve the SAR scene classification accuracy. In addition, when a metric function based on the Pearson correlation coefficient is applied to each sub-embedding, the classification accuracy is further improved.

[0111] The present invention applies the idea of ​​online gradient boosting to the SAR image scene classification problem, so that scene images with more complex structures can obtain higher weights, so that they can be focused on by CNN; in addition, the present invention also proposes a metric function based on the Pearson correlation coefficient to constrain different sub-embeddings, which allows the model to learn more rich intra-class information between similar SAR scenes. The present invention combines the above two methods to enable CNN to extract sufficiently discriminative feature information from SAR images, so as to be better used for SAR image scene classification.

Claims

1. A SAR image scene classification method based on online gradient boosting, characterized in that: The specific steps are as follows: Step 1: Use the random sampling method to construct the SAR data set, and use random cropping and random horizontal flipping to expand the data set; adjust the SAR scene image of 200×200 pixels to 128×128 pixels by bilinear interpolation method; Step 2: Add an embedding layer at the end of the convolutional neural network CNN and divide it into multiple non-overlapping sub-embeddings that share weights. Add a classifier after each sub-embedding. Step 2 is as follows: For the SAR dataset containing N samples in step 1 ,in represents a data sample, Represents the label of each sample; extracts features from SAR images using CNN , passing it into the embedding layer to obtain the corresponding sub-embedding vector , and obtain the corresponding classifier based on each sub-embedding vector , K is the number of sub-embeddings and classifiers, and the number of sub-embeddings and classifications is the same; Step 3: Add a regressor between every two sub-embeddings, and add a gradient flipping layer between each regressor and the sub-embedding. The feature vectors in the two sub-embeddings are mapped to the same feature space through the regressor, and a total metric function based on the Pearson correlation coefficient is learned in the space. The overall metric function based on the Pearson correlation coefficient described in step 3 for: （1） in Represents the SAR dataset n samples, represents the weight parameter, Represents sub-embedding i and j The Pearson correlation coefficient-based metric between i and sub-embedding j , by adding a regressor between the two, the sub-embedding j The embedding vectors in the map are mapped to sub-embeddings i The space where is located, and the metric function based on the Pearson correlation coefficient is learned in this space: （2） in, and Represent sub-embeddings i and j The embedding vector in ; Represents the sub-embed j The embedding vectors in the map are mapped to sub-embeddings i The regressor in the space, Represents sub-embedding j The dimension of the space in which you live; Represents the Pearson correlation coefficient of two eigenvectors; For vector and The Pearson correlation coefficient between the two is: （3） is equivalent to vector a, Equivalent to vector b; in, and represent the mean of vectors a and b respectively, The value range is [-1,1]. = 0, it means that vectors a and b have no linear relationship. In order to avoid the situation where the weight parameter in the regressor and embedding layer is 0, which causes the metric function based on the Pearson correlation coefficient to be 0, the metric function between each two sub-embeddings is Imposing constraints : （4） The operator express , that is, in Find the maximum value between and 0; represents the weight in the embedding layer, and b Represents the regressor Medium weights and biases; Step 4: Iteratively train each classifier in step 2 by using the online gradient boosting algorithm. During forward propagation, the loss function of each classifier is weighted and accumulated to obtain the entire integrated loss. The integrated loss is combined with the metric function based on the Pearson correlation coefficient described in step 3 to obtain a new loss function. Step 5: During back propagation, firstly, the weight corresponding to each classifier is updated according to the integrated loss described in step 4, and then the gradient flipping layer described in step 3 flips the gradient sign of the metric function based on the Pearson correlation coefficient; Step 6: After training, the prediction of each sub-embedded component is input into the softmax function, and the probability results output by the softmax function are weighted integrated through the weighted voting method to obtain the final integrated prediction result.

2. The SAR image scene classification method based on online gradient boosting according to claim 1 is characterized in that: In step 1, the SAR dataset is constructed using the random sampling method. The dataset is divided into multiple small batches before training. Then the model trains these small batch training sets in sequence. When sampling each small batch, ensure that each batch contains 80 SAR scene images, and these 80 images come from 16 different scene classes, and each class contains 5 images.

3. The SAR image scene classification method based on online gradient boosting according to claim 1 is characterized in that: Step 4 uses the online gradient boosting algorithm to iteratively train the classifier; assuming that there are K classifiers in total, the prediction of each classifier is iteratively obtained by weighted summation during forward propagation; for the kth classifier, its prediction is: （5） Among them, k The weight of the classifier ; Before the Representative k The classifier is i Prediction of SAR samples; The loss of the entire ensemble during forward propagation is: （6） in, Represents weight, the initial value is 1, weight It corresponds to i The sample k The weight of each classifier; represents the loss function of each classifier, （7） in, Representative i Sample labels and The inner product of Combine the loss of the entire integration with the metric function based on the Pearson correlation coefficient described in step 3 to obtain a new loss function: （8） in, Represents the weight parameter.

4. The SAR image scene classification method based on online gradient boosting according to claim 3 is characterized in that: In step 5, during back propagation, the new loss function is used Update the model weights until When the loss no longer changes significantly, training stops; specifically, for the integrated loss , according to the previous k -1 classifier ensemble prediction loss to calculate the residual, and use the residual value as the weight for the k Reweight the training samples of each classifier; （9） in, Represents weight, its initial value is 1, weight It is k The classifier corresponds to i The weight of the samples is calculated by k -1 negative gradient of the integrated prediction loss is calculated; New loss function In By flipping the sign of the gradient during back propagation, the process of minimizing the Pearson metric function during training will in turn reduce the correlation between sub-embeddings due to the effect of the gradient flip layer, thereby improving the diversity of the embedding layer; specifically, during back propagation, the sub-embedding i , j Medium weight parameter The update process is: （11） （12） in Represents sub-embedding i and j The learning rate.

5. The SAR image scene classification method based on online gradient boosting according to claim 4 is characterized in that: Step 6: When predicting, the predictions of each classifier are integrated by weighted voting: （13） in represents the integrated output, and It is the first k The weight corresponding to the prediction of a classifier is calculated as: （14）。

Images