A spatial sequence polarimetric SAR image classification method based on attention mechanism

Abstract

The present invention discloses a spatial sequence polarimetric SAR image classification method based on an attention mechanism, comprising constructing a classification model, the classification model comprising an input layer, a deep network model, an attention layer, a flattening layer, a classifier, and an output layer connected in sequence, wherein the deep network model is composed of multiple spatially reinforced long short-term memory network layers; training the classification model to obtain a trained classification model; extracting a polarimetric feature vector for each pixel in the polarimetric synthetic aperture radar image to be classified; selecting a pixel block centered around each pixel of the polarimetric synthetic aperture radar image to be classified, and establishing a temporal feature vector based on the polarimetric feature vector corresponding to each pixel in the pixel block; and inputting the temporal feature vector into the trained classification model to obtain a classification result. The present invention utilizes the cross-connection method in the LSTM model to strengthen the relationship between pixel spatial information based on the neighborhood relationship between pixels in the PolSAR image. Furthermore, to improve classification accuracy, an attention mechanism is introduced to select more effective classification features from high-level features, thereby strengthening the influence of key point pixels on the classification results and improving classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a spatial sequence polarization SAR image classification method based on an attention mechanism. Background Art

[0002] Polarimetric Synthetic Aperture Radar (SAR) is a high-resolution microwave imaging system. Unaffected by weather and day / night conditions, polarimetric SAR has found important applications in military, remote sensing, agriculture, water conservancy, and other fields. In recent years, polarimetric SAR image classification has become a key research topic in polarimetric SAR image processing.

[0003] In traditional PolSAR image classification methods, a classifier is used after the feature extractor, which can generally be divided into unsupervised and supervised classification, such as clustering methods, support vector machines (SVM), Wishart maximum likelihood (Wishart ML), neural networks (NN), etc. However, all of these machine learning classifiers need to be connected to a task-specific feature extractor, because the classification performance of these classifiers depends largely on the discriminative power of the extracted features. In addition, the influence of speckle noise increases the difficulty of feature extraction from PolSAR data and weakens the effective relationship between the feature extractor and classifier. Therefore, it is necessary to develop an end-to-end classification method from raw PolSAR data to the final classification result, which can automatically extract features and perform classification under a unified framework.

[0004] The emergence of recurrent neural networks (RNNs) has provided an effective approach for analyzing problems involving ordered sequence data. RNNs are specialized neural networks with a "memory" property that performs linear recurrence over time. Their concept was inspired by the idea that human cognition is based on past experience and memory. Because the W constant is shared across all time steps within a single RNN layer, a series of multiplication operations during network optimization can cause exploding or vanishing gradients. Furthermore, during forward propagation, the initial input data has less and less influence on subsequent steps, leading to long-range dependencies and, consequently, the RNN's loss of memory.

[0005] RNNs have been used to model dependencies in sequential data using feedback connections between them. These dependencies can be used to reflect the spatial relationships between pixels in PolSAR image classification. This deep model not only enables end-to-end classification but also captures spatial information. Long short-term memory (LSTM) networks, a typical RNN architecture, use LSTM units instead of the RNN's hidden layers to learn and memorize long-term information. Summary of the Invention

[0006] The embodiment of the present invention provides a spatial sequence polarimetric SAR image classification method based on an attention mechanism. The spatial information between pixels is increased through a spatial sequence, and the cross-connection method in the LSTM model is used to strengthen the relationship between pixel spatial information. The attention mechanism is introduced into the LSTM network to select important information and improve classification performance, avoiding the feature selection process.

[0007] On the one hand, an embodiment of the present invention provides a spatial sequence polarimetric SAR image classification method based on an attention mechanism, comprising:

[0008] Build a classification model, which includes an input layer, a deep network model, an attention layer, a flat layer, a classifier, and an output layer connected in sequence. The deep network model consists of multiple spatially reinforced long short-term memory network layers.

[0009] Train the classification model to obtain a trained classification model;

[0010] Extracting the polarization feature vector of each pixel in the polarization synthetic aperture radar image to be classified;

[0011] A pixel block is selected with each pixel of the polarization synthetic aperture radar image to be classified as the center, and a time series feature vector is established according to the polarization feature vector corresponding to each pixel in the pixel block;

[0012] Input the time series feature vector into the trained classification model to obtain the classification result.

[0013] The spatial sequence polarimetric SAR image classification method based on the attention mechanism in the present invention has the following advantages:

[0014] (1) A new PolSAR image classification method based on spatially enhanced LSTM network can extract polarization and spatial features from adjacent pixels, solving the problem of insufficient spatial information extraction of polarization SAR images by traditional LSTM.

[0015] 2) The proposed method consists of an automatic feature extractor, selector, and classifier, which does not require feature extraction and selection and can achieve end-to-end classification. The proposed method can input PolSAR image sequences and output PolSAR classification results.

[0016] (3) The classification effect is excellent, the accuracy is high, and the regional consistency of the classification results is also good. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a spatial sequence polarimetric SAR image classification method based on an attention mechanism provided in an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a spatially reinforced long short-term memory network model provided by an embodiment of the present invention;

[0020] Figure 3 A pseudo-color image of the polarimetric synthetic aperture radar image provided in an embodiment of the present invention after Pauli decomposition;

[0021] Figure 4 The real ground feature marker corresponding to the polarimetric synthetic aperture radar image provided by the embodiment of the present invention;

[0022] Figure 5 is the classification result obtained by using the Wishart classification method;

[0023] Figure 6 is the classification result obtained by using the support vector machine classification method;

[0024] Figure 7 is the classification result obtained by using the convolutional neural network classification method;

[0025] Figure 8 is the classification result obtained by using the long short-term memory network classification method;

[0026] Figure 9 This is the classification result obtained by using the long short-term memory network and attention mechanism classification method;

[0027] Figure 10 This is the classification result obtained by using the spatial reinforcement long short-term memory network classification method;

[0028] Figure 11 This is the classification result obtained by the classification method provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] Figure 1 The present invention provides a flow chart of a method for classifying spatial sequence polarimetric SAR images based on an attention mechanism. The present invention provides a method for classifying spatial sequence polarimetric SAR images based on an attention mechanism, the method comprising:

[0031] S100. Construct a classification model. The classification model includes an input layer, a deep network model, an attention layer, a flat layer, a classifier, and an output layer connected in sequence. The deep network model is composed of multiple spatially reinforced long and short-term memory network layers.

[0032] Exemplarily, the number of the above-mentioned Space Enhanced Long-Short Term Memory (SELSTM) network layers is 3, namely the first SELSTM layer, the second SELSTM layer and the third SELSTM layer.

[0033] The input format of the input layer is (N, step, dim), where N represents the number of samples, step represents the time step, and dim represents the dimension of each time step. In this embodiment, the input layer is set to (N, 25, 6). The size of the hidden layer within the first SELSTM layer is set to 32, the size of the hidden layer within the second SELSTM layer is set to 64, and the size of the hidden layer within the third SELSTM layer is set to 128. The number of categories in the classifier is set to 15, and the output layer outputs the final determined category for each pixel.

[0034] In order to extract more sufficient spatial information, this paper proposes a spatial enhancement LSTM (SELSTM) network structure based on neighborhood spatial sequence to enhance the impact of spatial information. The specific structure is as follows Figure 2 As shown, in the embodiment of the present invention, the spatial enhanced long short-term memory network layer has a forget gate, an input gate and an output gate.

[0035] The SELSTM layer first removes some information from the memory cell c through the forget gate t The forget gate reads the output of the previous moment and the input of the current moment, and then uses the Sigmoid activation function to decide which information to keep and which information to discard. The implementation principle of the forget gate is shown in the following formula:

[0036] ft =σ(w f ·[h k ,h t-1 ,x t ]+b f )

[0037] Among them, σ is the Sigmoid activation function, w f is the weight matrix corresponding to the forget gate, b f is the bias corresponding to the forget gate, h k represents the output at time k, k represents the kth element of the extracted pixel sequence, and this pixel is spatially adjacent to the pixel input at time t, h t-1 Represents the output at the previous moment, x t Represents the input at the current moment.

[0038] Then, the SELSTM layer passes the input gate in the memory cell c t The input gate is implemented in two parts: First, it uses the Sigmoid activation function to determine which values need to be updated. Second, it uses the tanh function to construct a vector containing the new candidate values. By combining the first and second steps, the state of the old memory cell is updated and then input into the new memory cell. The implementation principle of the input gate is shown in the following formula:

[0039] i t =σ(w i ·[h k ,h t-1 ,x t ]+b i )

[0040]

[0041]

[0042] Among them, w i is the weight matrix corresponding to the input gate, b i is the bias corresponding to the input gate, w c Indicates updating memory unit c t The weight matrix, b c To update the memory unit c t The corresponding bias.

[0043] Finally, the SELSTM layer determines which information needs to be output through the output gate. The SELSTM layer first determines the memory unit c through the Sigmoid activation function. t What information needs to be output, and then the memory unit c is processed by the tanh function tProcessing is performed by converting the value to between -1 and 1, and finally letting the memory unit c processed by the tanh function t The output gate is multiplied by the value of the Sigmoid activation function, so that the information that determines the output can be obtained. The implementation principle of the output gate is shown in the following formula:

[0044] o t =σ(w o ·[h k ,h t-1 ,x t ]+b o )

[0045] h t =o t .tanh(c t )

[0046] Among them, w o represents the weight matrix of the output gate, b o represents the bias of the output gate, h t Represents the output at time t, which consists of the output gate and memory unit c t Joint decision.

[0047] After the three SELSTM layers in the deep network model output vectors at each moment, the attention layer obtains the vectors output by the third SELSTM layer at each moment and combines the outputs h at different moments t As the input of the attention layer, the outputs of multiple moments are weighted summed to enhance the model's ability to focus on key features from long input sequences.

[0048] The input of the attention layer of the present invention is composed of the output {h1,h2,h3,...,hi} of the third layer SELSTM. The attention layer generates an attention weight matrix and feature representation U according to the importance of different features. The formula is as follows

[0049] e ti =V T tanh(W w h ti +b w )

[0050]

[0051]

[0052] Among them, W, V, b are network learnable parameters, e represents the attention score function, and U represents the feature vector calculated by the attention mechanism as the input of the next layer.

[0053] Finally, the output of the attention layer is passed through the flattening layer to flatten the multidimensional data into one dimension, and then classified by the Softmax classifier. The classification calculation formula is:

[0054]

[0055] Among them, W represents the weight, x represents the input vector, and b represents the bias term.

[0056] S110: Train the classification model to obtain a trained classification model.

[0057] For example, before training, a sample polarization synthetic aperture radar image with ground object markers is first obtained; then a portion of pixels in the sample polarization synthetic aperture radar image is selected as training samples; then a time series feature vector corresponding to each pixel in the training sample is established; finally, the time series feature vector corresponding to each pixel in the training sample is input into the classification model, the classification model is trained, and a trained classification model is obtained.

[0058] After selecting a subset of pixels from a sample polarimetric synthetic aperture radar image as training samples, other pixels from the sample polarimetric synthetic aperture radar image are used as test samples. After obtaining a trained classification model, a time-series feature vector corresponding to each pixel in the test sample is constructed. This time-series feature vector corresponding to each pixel in the test sample is input into the trained classification model to obtain a classification result. The performance of the trained classification model is then determined based on the error between the classification result and the ground feature label of the corresponding pixel.

[0059] In an embodiment of the present invention, based on real ground object labels, 5% of the labeled pixels for each ground object category in the sample polarimetric synthetic aperture radar image are randomly selected as training samples, and the remaining 95% of the labeled pixels are used as test samples.

[0060] During the training process, the time series feature vector corresponding to each pixel in the training sample is used as the input of the classification model of the present invention, and the result of the forward propagation of the classification model is used as the output of the model. By solving the error between the output and the correct category of the real object label of each pixel, and backpropagating the error, the network parameters of the classification model are optimized to obtain a trained classification model. After the classification of the test sample is completed, the classification result obtained by classification is compared with the real object label. For each category, the ratio of the number of correctly classified pixels to the total number of pixels of the corresponding category in the test sample is used as the accuracy of this category; the ratio of the total number of correctly classified pixels to the total number of pixels in the test sample is used as the overall classification accuracy of the sample polarimetric SAR image.

[0061] S120: Extracting a polarization feature vector of each pixel in the polarization synthetic aperture radar image to be classified.

[0062] Exemplarily, extracting a polarization feature vector of each pixel in a polarization synthetic aperture radar image to be classified includes: filtering the polarization synthetic aperture radar image to be classified to obtain a filtered polarization synthetic aperture radar image to be classified; performing Pauli decomposition on the filtered polarization synthetic aperture radar image to be classified to obtain a corresponding pseudo-color image; and extracting a polarization feature vector of each pixel in the pseudo-color image.

[0063] In an embodiment of the present invention, the purpose of filtering the polarimetric SAR image to be classified is to remove speckle noise and obtain a filtered polarimetric SAR image to be classified. Polarimetric SAR images are typically filtered using the existing refined polarimetric LEE filtering method, with a filter window size of 7×7. Pauli decomposition is a target decomposition method that decomposes the scattering matrix of the raw data into a linear combination of a single scattering mechanism, a dihedral scattering mechanism rotated 0° about the axis, and a dihedral scattering mechanism rotated 45° about the axis, based on the scattering characteristics of the ground object.

[0064] The process of extracting the polarization feature vector of each pixel in the polarization synthetic aperture radar image to be classified is as follows:

[0065] a) Read each pixel of the filtered polarimetric SAR image to be classified. The pixel is a 3×3 covariance matrix:

[0066]

[0067] Among them, S HH Represents the scattered component of horizontal transmission and horizontal reception, S VV Indicates the scattered component of vertical transmission and vertical reception, S HV represents the scattered component of horizontal transmission and vertical reception, <·> represents the inner product, |·| represents the modulus, and ·* represents that this matrix is a conjugate matrix.

[0068] b) The covariance matrix C is a complex conjugate matrix in which each element value is a complex number and the elements on the diagonal are real numbers. After converting the complex data into real values, the polarization eigenvector of each pixel in the polarimetric SAR image to be classified can be expressed as:

[0069] feature=[c 11 ,c 22 ,c 33 ,real(c 12 ),imag(c 12 ),real(c 13 ),imag(c 13 ),real(c 23 ),imag(c 23 )]

[0070] Among them, real() represents the real part and imag() represents the imaginary part.

[0071] S130 , selecting a pixel block with each pixel of the polarization synthetic aperture radar image to be classified as the center, and establishing a time series feature vector according to the polarization feature vector corresponding to each pixel in the pixel block.

[0072] Exemplarily, a 5×5 pixel block is selected with each pixel of the to-be-classified polarimetric synthetic aperture radar image as the center, and then a time series feature vector is established according to the polarimetric feature vector corresponding to each pixel in the pixel block.

[0073] S140: Input the time series feature vector into the trained classification model to obtain a classification result.

[0074] For example, the classification result is a classified polarimetric SAR image. After obtaining the classification result, red, green, and blue are used as three primary colors and the classified polarimetric SAR image is colored according to the three primary coloring method. The same color is used for the same category in the classification result, thereby obtaining a colored classification result map.

[0075] Experimental Description

[0076] To illustrate the effect of the polarimetric SAR image classification method based on SELSTM and attention mechanism in the present invention, the following experiments can be used to verify it.

[0077] 1. Experimental methods: including (1) Wishart classification method; (2) SVM classification method; (3) CNN (convolutional neural network) classification method; (4) LSTM classification method; (5) classification method combining LSTM and attention mechanism; (6) SELSTM classification method; (7) classification method combining SELSTM and attention mechanism in the present invention.

[0078] 2. Experimental results

[0079] Figure 3 This is the Pauli pseudo-color image of the polarimetric SAR data of Flevoland, the Netherlands, used in the simulation of this experiment. The size is 750×1024. Figure 4 is the real ground feature marker corresponding to the polarimetric SAR image used in this experiment. Figure 4 We randomly select 5% of the real object labels for each category as training samples, and use the remaining 95% of the labeled data as test data. The simulation experiment method and results are shown in Table 1.

[0080] Simulation 1, using Wishart classification method Figure 3 Classification is performed, and the classification results are shown in Figure 5 The accuracy of each category and the overall accuracy are shown in Table 1.

[0081] Simulation 2, using SVM classification method to Figure 3 Classification is performed, and the classification results are shown in Figure 6 The accuracy of each category and the overall accuracy are shown in Table 1.

[0082] Simulation 3, using CNN classification method to Figure 3 Classification is performed, and the classification results are shown in Figure 7 The accuracy of each category and the overall accuracy are shown in Table 1.

[0083] Simulation 4, using LSTM classification method Figure 3 Classification is performed, and the classification results are shown in Figure 8 The accuracy of each category and the overall accuracy are shown in Table 1.

[0084] Simulation 5: Classification method combining LSTM and attention mechanism Figure 3 Classification is performed, and the classification results are shown in Figure 9 The accuracy of each category and the overall accuracy are shown in Table 1.

[0085] Simulation 6, using SELSTM classification method Figure 3 Classification is performed, and the classification results are shown in Figure 10 The accuracy of each category and the overall accuracy are shown in Table 1.

[0086] Simulation seven, using the classification method of the present invention to Figure 3 Classification is performed, and the classification results are shown in Figure 11 The accuracy of each category and the overall accuracy are shown in Table 1.

[0087] Table 1 The accuracy of various classification methods

[0088]

[0089]

[0090] From Table 1 and Figure 5-11As can be seen, the accuracy rate of the Wishart classification method was 78.15%; the SVM classification method was 86.82%; the CNN classification method was 96.03%; the LSTM classification method was 92.01%; the classification method combining LSTM and attention mechanism was 95.47%; the SELSTM classification method was 96.26%; and the accuracy rate of the proposed method was 98.61%. Clearly, the proposed method achieved the highest accuracy rate. The classification results of the proposed method outperformed existing methods in both accuracy and visual quality. Furthermore, the proposed method produced more complete homogeneous regions and better regional consistency than existing methods. This fully demonstrates the superior performance of the proposed method in polarimetric SAR image classification.

[0091] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0092] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A spatial sequence polarimetric SAR image classification method based on attention mechanism, characterized by: include: Constructing a classification model, wherein the classification model includes an input layer, a deep network model, an attention layer, a flat layer, a classifier, and an output layer connected in sequence, wherein the deep network model is composed of multiple spatially reinforced long short-term memory network layers; Training the classification model to obtain a trained classification model; Extracting the polarization feature vector of each pixel in the polarization synthetic aperture radar image to be classified; Selecting a pixel block with each pixel of the polarization synthetic aperture radar image to be classified as the center, and establishing a time series feature vector according to the polarization feature vector corresponding to each pixel in the pixel block; Inputting the time series feature vector into the trained classification model to obtain a classification result; The spatial reinforcement long short-term memory network layer has a forget gate, an input gate and an output gate; The forget gate is used to discard some information from the memory unit. The forget gate reads the output of the previous moment and the input of the current moment, and then determines the information to be retained and the information to be discarded through the activation function; The input gate is used to determine the new information that needs to be stored in the memory cell. The input gate first determines the value that needs to be updated through the activation function, and then constructs a vector including the new candidate value through the tanh function. The input gate updates the state of the memory cell by combining these two steps and then inputs the information into the new memory cell. The output gate is used to determine the information to be output. The output gate first determines the information to be output in the memory unit through the activation function, then processes the memory unit through the tanh function, and finally multiplies the value processed by the tanh function and the output value of the activation function to obtain the information to be output; The attention layer generates an attention weight matrix and feature representation U based on the importance of different features. The formula is as follows: e ti =V T tanh(W w h ti +b w ) Among them, W, V, b are network learnable parameters, e represents the attention score function, and U represents the feature vector calculated by the attention mechanism as the input of the next layer.

2. The spatial sequence polarimetric SAR image classification method based on the attention mechanism according to claim 1 is characterized in that: The extracting of the polarization feature vector of each pixel in the polarization synthetic aperture radar image to be classified includes: performing Pauli decomposition on the polarization synthetic aperture radar image to be classified to obtain a corresponding pseudo color image; and extracting the polarization feature vector of each pixel in the pseudo color image.

3. The spatial sequence polarimetric SAR image classification method based on the attention mechanism according to claim 2 is characterized in that: The step of performing Pauli decomposition on the polarimetric synthetic aperture radar image to be classified to obtain a corresponding pseudo-color image includes: Filtering the polarimetric synthetic aperture radar image to be classified to obtain a filtered polarimetric synthetic aperture radar image to be classified; Pauli decomposition is performed on the filtered polarimetric synthetic aperture radar image to be classified to obtain a corresponding pseudo-color image.

4. The method for spatial sequence polarimetric SAR image classification based on attention mechanism according to claim 3, characterized in that: The polarimetric synthetic aperture radar image to be classified is filtered using a refined polarimetric LEE filtering method.

5. The spatial sequence polarimetric SAR image classification method based on the attention mechanism according to claim 1, characterized in that: The training of the classification model to obtain a trained classification model includes: Acquire sample polarimetric synthetic aperture radar images with ground object markers; Selecting a portion of pixels in the sample polarimetric synthetic aperture radar image as training samples; Establishing a time series feature vector corresponding to each pixel in the training sample; The temporal feature vector corresponding to each pixel in the training sample is input into the classification model, and the classification model is trained to obtain the trained classification model.

6. The method for spatial sequence polarimetric SAR image classification based on attention mechanism according to claim 5, characterized in that: After selecting a portion of pixels in the sample polarimetric synthetic aperture radar image as training samples, the method further includes: using other pixels in the sample polarimetric synthetic aperture radar image as test samples; After obtaining the trained classification model, the method further includes: Establishing a temporal feature vector corresponding to each pixel in the test sample; The time series feature vector corresponding to each pixel in the test sample is input into the trained classification model to obtain a classification result, and the performance of the trained classification model is determined based on the error between the classification result and the ground feature label of the corresponding pixel.

Images