SAR image classification method based on attention mechanism and residual relational network

Abstract

The present invention discloses a SAR image classification method based on an attention mechanism and a residual relational network, comprising: constructing a classification model, the classification model being a 3D residual relational network model based on an attention mechanism; selecting a training set; training the classification model using the training set to obtain a trained classification model; obtaining a polarimetric synthetic aperture radar image to be classified; inputting the original input information of each pixel in the polarimetric synthetic aperture radar image to be classified into the trained classification model to obtain a classification result. The present invention is aimed at the problem of small samples in polarimetric SAR image classification, improves the application of deep learning methods in polarimetric SAR images, and combines the spatial neighborhood information of polarimetric SAR data to effectively realize the classification of polarimetric SAR images under small samples by deep learning.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a SAR image classification method based on an attention mechanism and a residual relational network. Background Art

[0002] Synthetic Aperture Radar (SAR) is a coherent imaging radar that works in the microwave band. It is an active remote sensing sensor that plays an important role in the field of remote sensing because it can perform large-scale imaging of ground targets around the clock and in all weather conditions. Single-frequency, single-polarization SAR images can make up for the shortcomings of optical remote sensing, eliminate interference from clouds, fog, vegetation, and leaves, and have a certain degree of penetration into the ground surface. They have unique advantages in surveying and mapping, military, environmental monitoring, disaster monitoring, and ocean monitoring.

[0003] Traditional polarimetric SAR image classification methods mainly include fully supervised learning and semi-supervised learning methods. The former requires all samples to be labeled, which has high requirements on samples. The latter only requires some samples to have labels and others to have no labels. For these two methods, feature extraction and classification technology are two basic elements. The performance of polarimetric SAR image classification depends largely on feature extraction.

[0004] Classic polarimetric SAR classification methods include: Jiao et al. proposed a method combining Wishart distribution and network, which realized the rapid classification of polarimetric SAR images and the expansion of deep learning in the field of polarimetric SAR classification. Adam Santoro et al. proposed a memory-enhanced network based on long short-term memory network (LSTM). The memory-enhanced network trains data as a sequence, and the previous label is also used as network input, and external storage is added to store the previous input. This allows the label and input to establish a connection when the direction propagates after the next input, so that the subsequent input can obtain related images through external memory for comparison and achieve better prediction. Zhou et al. proposed a polarimetric SAR image classification method based on CNN with pixel neighborhood as the basic classification unit. The above methods are all based on supervised learning, so there are some defects: supervised learning requires a large number of labeled samples, and the annotation of polarimetric SAR images requires rich experience and professional knowledge to complete, and it takes a lot of energy and material resources to complete high-quality annotation. Summary of the invention

[0005] The embodiment of the present invention provides a SAR image classification method based on an attention mechanism and a residual relationship network, so as to solve the problem that the supervised classification method in the prior art has high sample requirements and consumes energy and material resources.

[0006] On the one hand, an embodiment of the present invention provides a SAR image classification method based on an attention mechanism and a residual relational network, comprising:

[0007] Build a classification model, which is a 3D residual relational network model based on the attention mechanism;

[0008] Select a training set;

[0009] The classification model is trained using the training set to obtain a trained classification model;

[0010] Acquiring a polarimetric synthetic aperture radar image to be classified;

[0011] Selecting a pixel block centered on each pixel in the polarimetric synthetic aperture radar image;

[0012] Extract feature information of each pixel in the pixel block, and use the feature information corresponding to each pixel in the pixel block as the original input information of the central pixel;

[0013] The original input information of each pixel in the polarimetric synthetic aperture radar image is input into the trained classification model to obtain the classification result.

[0014] The SAR image classification method based on the attention mechanism and residual relation network in the present invention has the following advantages:

[0015] (1) The present invention aims to improve the application of deep learning methods in polarimetric SAR image classification by targeting the small sample problem in polarimetric SAR image classification. By combining the spatial neighborhood information of polarimetric SAR data, the present invention effectively realizes the classification of polarimetric SAR images using deep learning under small sample conditions.

[0016] (2) The present invention improves the deficiency caused by the small number of labeled samples by adding an attention mechanism (SENet) to the relationship network and combining it with a meta-learning method. It can be used for small sample SAR image classification, automatically obtain the importance of features, and filter out unimportant feature information.

[0017] (3) The simulation results show that the small sample SAR image classification method based on attention mechanism and relational network adopted in the present invention has excellent classification effect and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a SAR image classification method based on an attention mechanism and a residual relationship network provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart of the attention mechanism provided for an embodiment of the present invention;

[0021] Figure 3 It is a PauliRGB composite image of C-band polarimetric SAR data of Flevoland Province, the Netherlands, used in a simulation in an embodiment of the present invention;

[0022] Figure 4 To simulate the real ground feature marking of the polarization SAR image used in the embodiment of the present invention;

[0023] Figure 5 The classification results of the simulation using Wishart, SVM, CNN, 3D-CNN, RN, RN-SENet, RN-Res-SENet, 3D-RN-Res and other classification methods in the embodiment of the present invention are as follows;

[0024] Figure 6 The following is a simulation of a classification result using the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Figure 1 A flowchart of a SAR image classification method based on an attention mechanism and a residual relationship network provided in an embodiment of the present invention. An embodiment of the present invention provides a SAR image classification method based on an attention mechanism and a residual relationship network, including:

[0027] S100, constructing a classification model, where the classification model is a 3D residual relational network model based on an attention mechanism.

[0028] Exemplarily, the core of the 3D residual network is 3D-CNN (three-dimensional convolutional neural network). 3D-CNN is built based on 3D convolution operations, and its convolution kernel is a three-dimensional cube. Similarly, polarimetric SAR image data is also three-dimensional data of the entire neighborhood and central pixel. Using 3D-CNN for polarimetric SAR image classification is more conducive to extracting spatial and channel characteristics of polarimetric SAR data.

[0029] The classification model of the present invention includes an input module, a feature embedding module and a relationship module. Experiments have shown that when the number of network layers of the model increases, the model has stronger feature learning ability and feature representation ability. However, as the network depth (i.e., the number of layers) increases, the convolutional neural network will suffer from the problem of degradation. In view of this, a residual network is introduced, and the classification accuracy can be improved by increasing the depth of the model.

[0030] Based on the above 3D residual relationship network, Figure 2 As shown in Figure 2, the specific process of the attention mechanism is as follows:

[0031] a) X is the data input to the input module, with a size of C'×H'×W';

[0032] b), X undergoes convolution operation (F tr ) after which the output U is C×H×W;

[0033] c) Input U into the GlobalAveragePooling layer for processing, and the output is a data size of 1×1×C:

[0034]

[0035] d) After the output data Z of 1×1×C passes through two levels of fully connected layers, the output is limited to the range of [0,1] through the Sigmod activation function, and the output scale is:

[0036] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0037] e) Multiply U by scale, that is, multiply each layer of U's C channels by scale as input data for the next level (i.e., the relationship module):

[0038]

[0039] The structure of the classification model is:

[0040] Feature embedding module: input layer → data preprocessing layer → first residual block → first maximum pooling layer → second residual block → second maximum pooling layer → third residual block → average pooling layer → first fully connected layer → output layer.

[0041] Relation module: input layer → first convolution layer → first pooling layer → second convolution layer → first fully connected layer → second fully connected layer.

[0042] Among them, the number of feature maps of the input layer in the feature embedding module and the relationship module is set to 15, the number of feature maps of the data preprocessing layer is set to 15, and there are three convolutional layers in the first residual block, among which the number of feature maps of the first convolutional layer is set to 1, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the second convolutional layer is set to 6, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the third convolutional layer is set to 6, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the first maximum pooling layer is set to 6, the pooling kernel size is 1×3×3, and the step is 1×2×2. There are three convolutional layers in the second residual block, among which the number of feature maps of the fourth convolutional layer is set to 32, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The feature map of the fifth convolutional layer The number of images is set to 32, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the sixth convolution layer is set to 32, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the second maximum pooling layer is set to 32, the pooling kernel size is 1×3×3, and the step is 1×2×2. There are three convolutional layers in the third residual block, of which the number of feature maps of the seventh convolution layer is set to 64, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the eighth convolution layer is set to 64, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the ninth convolution layer is set to 64, the size of the convolution kernel is 3×3×3, and the sliding step is 1. The number of feature maps of the average pooling layer is set to 64, the pooling kernel size is 1×1×1, and the feature dimension of the first fully connected layer is set to 64.

[0043] In the relationship module, the number of feature maps of the first convolutional layer is set to 128, the size of the convolution kernel is 1×3×3, and the sliding step is 1. The number of feature maps of the first pooling layer is set to 64, the size of the pooling kernel is 1×3×3, and the step is 1×2×2. The number of feature maps of the second convolutional layer is set to 64, the size of the convolution kernel is 1×3×3, and the sliding step is 1. The feature dimension of the first fully connected layer is set to 640, and the feature dimension of the second fully connected layer is set to 8. The output of the relationship module is the output of the entire classification model, and the number of its output layers is set to 15, corresponding to the number of categories of the classification results.

[0044] S110, selecting a training set.

[0045] Exemplarily, according to the real ground object labels, 7 labeled data are randomly selected for each ground object category of the polarimetric SAR image data as training samples, and the remaining labeled data are used as test samples.

[0046] In the subsequent classification model training, before each iterative training starts, the training samples are converted into episodes. The episodes include support sets and query sets. All episodes constitute the training set.

[0047] S120: Use the training set to train the classification model to obtain a trained classification model.

[0048] Exemplarily, a meta-learning method may be used to train the classification model. Specifically, the meta-learning method includes:

[0049] a) Selecting samples with labels not equal to zero from the training samples according to the classification category to obtain the above training samples;

[0050] b) Expanding the selected samples to obtain an expanded sample set; specifically, the training samples can be expanded once by using a superpixel segmentation method, and then the expanded training samples are expanded twice by using the similarity between pixels to obtain an expanded sample set. After obtaining the expanded sample set, all samples with non-zero labels can also be used as test samples;

[0051] c) Sampling is performed in the expanded sample set. Each sampling obtains a support set and a query set. The support set and the query set form a round set. The number of categories selected from the training samples is recorded as C, and the number of support sets is recorded as K. Then:

[0052] x i ∈{Support set}

[0053] x j ∈{Query set}

[0054] Each round set is regarded as a C-way K-shot classification task. Usually, a complete training task (epoch) includes the process of establishing round sets multiple times. A round set needs to be established before each iterative training, and after each iterative training, the difference between the classification results obtained by training and the actual classification results can be used to update the parameters of the classification model until the maximum number of iterations is reached or the expected training effect is achieved:

[0055]

[0056] After obtaining the round set, the support set and query set in the round set are added to the feature embedding module, and the above-mentioned attention mechanism is used to extract the corresponding feature data;

[0057] After extracting the feature data of the support set and query set, the feature data of the support set and query set are concatenated and input into the relationship module to calculate the relationship score:

[0058] r i,j =g φ (C(f φ (x i ),f φ (x j ))),i=1,2,…,15

[0059] In the above relationship scores, f φ represents the embedded feature, C(x,y) represents the operation of concatenating x and y, and g φ Indicates the calculation of relationship score, r i,j Represents the relationship score.

[0060] After completing the training of the classification model, the trained classification model is also tested using the test sample. During the test process, the test sample is input into the trained classification model to obtain the classification result, and the classification result is compared with the true category of the test sample to obtain the classification accuracy of the classification model. Among them, the classification accuracy includes the category accuracy and the overall accuracy. The category accuracy is the ratio of the number of correctly classified pixels to the total number of pixels of the corresponding category in the test sample, and the overall accuracy is the ratio of the total number of correctly classified pixels to the total number of pixels in the test sample. When the classification accuracy reaches the expected level, it can be considered that the performance of the trained classification model has met the requirements.

[0061] S130: Obtain a polarimetric synthetic aperture radar image to be classified.

[0062] S140: Select a pixel block centered on each pixel in the polarimetric synthetic aperture radar image.

[0063] Exemplarily, before selecting a pixel block centered on each pixel in the polarimetric synthetic aperture radar image, the method further includes:

[0064] a) Filter the polarimetric synthetic aperture radar image. Specifically, the refined polarimetric LEE filtering method can be used to filter out the speckle noise in the polarimetric synthetic aperture radar image, and the filter window size can be set to 15×15. For the specific refined polarimetric LEE filtering method, please refer to Yang Dahai and Ma Debao. Improved polarimetric Lee filtering algorithm based on polarization vector similarity coefficient [J]. Journal of Information Engineering University, 2010, 11(06): 737-740.

[0065] b) Perform Pauli decomposition on the filtered polarization synthetic aperture radar image to obtain the corresponding pseudo-color image. Pauli decomposition is a target decomposition method that decomposes the scattering matrix of the original data into a linear combination of a single scattering mechanism, a dihedral scattering mechanism rotated 0° around the axis, and a dihedral scattering mechanism rotated 45° around the axis according to the scattering characteristics of the ground object target.

[0066] c) After obtaining the pseudo color image, a pixel block centered on each pixel in the pseudo color image is selected. Specifically, each pixel can be centered and pixels within a range of 15×15 can be selected as a pixel block.

[0067] S150, extracting feature information of each pixel in the pixel block, and using the feature information corresponding to each pixel in the pixel block as original input information of the central pixel.

[0068] Exemplarily, the method for extracting feature information is:

[0069] a) The pixels in the pseudo-color image are a 3×3 coherence matrix T:

[0070]

[0071] b) Using the polarization coherence matrix T, extract the 6-dimensional feature information L1:

[0072] L1=[A,B,C,D,E,F]

[0073] A=10lg(T 11 +T 12 +T 13 )

[0074] B=T 22 / (T 11 +T 22 +T 33 )

[0075] C=T 33 / (T 11 +T 22 +T 33 )

[0076]

[0077]

[0078]

[0079] Where A represents the total scattered energy of all polarization channels, B and C are T 22 and T 33 The energy ratio of , D, E, and F are the correlation coefficients.

[0080] S160, inputting the original input information of each pixel in the polarimetric synthetic aperture radar image into the trained classification model to obtain a classification result.

[0081] Exemplarily, after obtaining the classification results, red, green and blue are used as the three primary colors and are colored according to the three primary color coloring method. The same colors are used for the same categories in the classification results to obtain a colored classification result map, and the colored classification result map is output.

[0082] Experimental Description

[0083] To illustrate the beneficial effects of the classification method in the present invention, the following experiments can be used for verification.

[0084] 1. Experimental methods: including (1) the existing supervised Wishart method; (2) the traditional CNN classification method; (3) the method of the present invention, i.e., the small sample SAR image classification method based on the attention mechanism relationship network. Among them, the existing supervised Wishart method, SVM method, traditional CNN classification method, and 3D-CNN method are all classic methods that are widely cited in polarimetric SAR data classification.

[0085] 2. Experimental results

[0086] Figure 3 This is a PauliRGB composite image of the C-band polarimetric SAR data of Flevoland, the Netherlands, used in the simulation 1 of this embodiment, with a size of 750×1024. Figure 4 is the real ground feature mark of the polarimetric SAR image used in the simulation of this embodiment. Figure 4 For each category, 7 labeled samples are randomly selected as training samples, and the remaining labeled data are used as test samples. The simulation experiment methods and results are shown in Table 1.

[0087] The existing supervised Wishart classification method is used to Figure 3 The classification results are shown in Figure 5 (c) in the figure. Using the traditional SVM classification method Figure 3 The classification results are shown in Figure 5 (d) in the figure. The classic CNN classification method is used to classify Figure 3 The classification results are shown in Figure 5 (e) in the figure. 3D-CNN classification method is used to classify Figure 3 The classification results are shown in Figure 5 (f) in the figure. Figure 3 The classification results are shown in Figure 5 (g) in the figure. The RN-SENet classification method is used to classify Figure 3 The classification results are shown in Figure 5 (h) in the figure. The RN-Res classification method is used to classify Figure 3 The classification results are shown in Figure 5(i) in the figure. The RN-Res-SENet classification method is used to classify Figure 3 The classification results are shown in Figure 5 (j) in the figure. The 3D-RN-Res classification method is used to classify Figure 3 The classification results are shown in Figure 5 (k) in the above. Figure 3 The classification results are shown in Figure 5 (l) or Figure 6 The category accuracy and overall accuracy obtained by each classification method are shown in Table 1.

[0088] Table 1 Experimental results of 10 methods for polarimetric SAR image classification

[0089]

[0090] From Table 1 and Figure 4-6 It can be seen that the accuracy of the existing supervised Wishart classification method is 80.49%; the accuracy of the traditional SVM classification method is 61.59%; the accuracy of the classic CNN method is 78.98%; and the accuracy of the classification using 3D-CNN is 80.15%; the accuracy of the RN classification method is 89.16%; the accuracy of the RN-SENet classification method is 90.77%; the accuracy of the RN-Res classification method is 92.37%; the accuracy of the RN-Res-SENet classification method is 93.57%; the accuracy of the 3D-RN-Res classification method is 93.08%; the accuracy of the method of the present invention is 95.88%. It can be seen that the method of the present invention has achieved the highest accuracy. The classification results of the method of the present invention are superior to those of the supervised wishart classification method, the SVM method, and the traditional CNN and 3D-CNN classification methods in terms of accuracy and visual effect. Moreover, the homogeneous regions of the method of the present invention are more complete than those of the supervised Wishart classification method, the SVM classification method, the traditional CNN, and the 3D-CNN classification methods, etc. This fully demonstrates that the method of the present invention has better small sample SAR data classification performance.

[0091] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0092] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A SAR image classification method based on attention mechanism and residual relation network, characterized in that: include: Constructing a classification model, wherein the classification model is a 3D residual relational network model based on an attention mechanism; Select a training set; The classification model is trained using the training set to obtain a trained classification model; the classification model includes a feature embedding module and a relationship module; Acquiring a polarimetric synthetic aperture radar image to be classified; Selecting a pixel block centered on each pixel in the polarimetric synthetic aperture radar image; Extracting feature information of each pixel in the pixel block, and using the feature information corresponding to each pixel in the pixel block as original input information of the central pixel; Inputting the original input information of each pixel in the polarimetric synthetic aperture radar image into the trained classification model to obtain a classification result; The selecting of the training set comprises: According to the real object labels, multiple labeled data are selected as training samples; Each time the classification model performs training iteration, the training samples are converted into a round set, the round set includes a support set and a query set, and all the round sets constitute the training set; Each time the classification model performs training iteration, the training samples are converted into a round set, including: Selecting samples whose labels are not zero from the training samples according to the classification category; Expanding the selected samples to obtain an expanded sample set; Sampling is performed in the expanded sample set, and each sampling obtains a support set and a query set, wherein the support set and the query set constitute the round set; After obtaining the round set, adding the support set and the query set in the round set to the feature embedding module, and extracting corresponding feature data using an attention mechanism; after extracting the feature data of the support set and the query set, concatenating the feature data of the support set and the query set, and inputting them into the relationship module to calculate a relationship score; Before selecting a pixel block centered on each pixel in the polarimetric synthetic aperture radar image, the method further includes: Performing filtering processing on the polarimetric synthetic aperture radar image; Perform Pauli decomposition on the filtered polarimetric synthetic aperture radar image to obtain the corresponding pseudo-color image; After the pseudo-color image is obtained, a pixel block centered on each pixel in the pseudo-color image is selected.

2. The SAR image classification method based on attention mechanism and residual relation network according to claim 1, characterized in that: The classification result is colored by using a three-primary-color coloring method to obtain a colored classification result.

3. The SAR image classification method based on attention mechanism and residual relation network according to claim 1, characterized in that: In each training iteration, after converting the training samples into round sets, the following steps are also included: Performing convolution processing on the support set and the query set to obtain a convolution result; Performing average pooling processing on the convolution result to obtain a pooling result; Performing two full connection processes on the pooling result, limiting the output range to the range of [0,1], and obtaining a full connection result; The convolution result and the full connection result are multiplied to obtain the feature data of the support set and the query set.

4. The SAR image classification method based on attention mechanism and residual relation network according to claim 1, characterized in that: After the training of the classification model is completed, the trained classification model is tested using a test sample.

Images