Polarimetric SAR terrain classification method based on hybrid convolutional neural network cross pseudo supervision

Abstract

The invention discloses a polarized SAR terrain classification method based on hybrid convolutional neural network cross pseudo supervision in the technical field. The method comprises the following steps: 1) converting triangular elements on a polarized coherence matrix of each pixel in an image into a 6-dimensional complex vector and a 6-dimensional real number vector through real number conversion; 2) extracting an image block with the size of w * w * 6 as an input sample of the network by taking each pixel in the image as a center; 3) using random values to initialize weights in the CV-CNN and the CNN; 4) inputting the label samples and the label-free samples into the cross pseudo-supervised network, and performing optimization by using a small-batch stochastic gradient descent algorithm; and 5) after the training is finished, inputting a neighborhood block corresponding to each pixel in the image into the CV-CNN and the CNN, and fusing the prediction probability values of the two networks to complete the classification of the image, thereby solving the problem of training sample scarcity in the polarized SAR image classification problem.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a polarimetric SAR ground object classification method. Background Art

[0002] Synthetic Aperture Radar (SAR) is an active microwave imaging system that can achieve all-day and all-weather earth observation. SAR image classification is the basis of SAR image interpretation and plays a key role in urban planning, agricultural monitoring, natural disaster analysis, situational awareness and other fields. Compared with SAR, polarimetric SAR can transmit and receive electromagnetic waves of different polarization modes, thereby providing SAR images with richer target information and obtaining more accurate classification results. However, the diversity and complexity of polarimetric information also bring difficulties in extraction and utilization, which brings higher challenges to interpretation problems such as polarimetric SAR image classification.

[0003] Consistency regularization is a commonly used technique in machine learning that improves the performance and generalization of a model by encouraging it to make consistent predictions for similar input data. The basic idea of ​​this technique is to penalize the model when it makes different predictions for similar input data. Consistency regularization is one of the commonly used operations in semi-supervised learning, which allows the model to access both labeled and unlabeled data. Labeled data is used to train the model in a supervised manner, while unlabeled data is used to promote consistency in the model's predictions between different versions of the same input.

[0004] Deep learning is a machine learning method that simulates the structure of the human brain's neural network and learns and abstracts data through multi-level neural networks. It is based on large-scale data and powerful computing power, and continuously adjusts network parameters through the back-propagation algorithm to achieve recognition and prediction of complex patterns. Deep learning has been widely used in image recognition, speech recognition, natural language processing and other fields, and has achieved remarkable results. Its advantage is that it can learn feature representation from raw data without manual feature extraction, and it has high flexibility and generalization ability. However, deep learning also has challenges such as large data demand, high model complexity, and poor interpretability, and it is necessary to comprehensively consider factors such as model design and data quality. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a polarimetric SAR feature classification method based on hybrid convolutional neural network cross pseudo-supervision, which solves many difficult problems such as the difficulty in classification in the case of small samples of polarimetric SAR images.

[0006] The object of the present invention is achieved by: a polarimetric SAR ground object classification method based on hybrid convolutional neural network cross pseudo supervision, comprising the following steps:

[0007] Step 1) converting the upper triangular elements of the polarization coherence matrix of each pixel in the image into a 6-dimensional complex vector and a 6-dimensional real vector by real number conversion;

[0008] Step 2) Extract an image block of size w×w×6 with each pixel in the image as the center as the input sample of the network;

[0009] Step 3) Initialize the weights in CV-CNN and CNN using random values;

[0010] Step 4) Input the labeled samples and unlabeled samples into the cross pseudo-supervised network and optimize them using a mini-batch stochastic gradient descent algorithm;

[0011] Step 5) After the training is completed, the neighborhood block corresponding to each pixel in the image is input into CV-CNN and CNN, and the predicted probability values ​​of the two networks are fused to complete the image classification.

[0012] Further, step 1) converts the upper triangular elements of the polarization coherence matrix of each pixel in the image into a 6-dimensional complex vector and a 6-dimensional real vector by real number conversion. The implementation process is as follows: the 6-dimensional complex vector converted from the polarization coherence matrix is ​​used as the feature of the polarization SAR image, wherein the polarization coherence matrix can be expressed by the following formula:

[0013]

[0014] Then the 6-dimensional complex vector can be expressed as v_cv = [T 11 ,T 12 ,T 13 ,T 22 ,T 23 ,T 33 ], using unitary transformation to convert complex-valued matrices into real-valued matrices, a fully symmetric transformation is proposed by the following formula:

[0015]

[0016] Where R is a Hermitian and generalized symmetric matrix. T is the polarization coherence matrix, and J is a matrix with 1s on the sub-diagonal and zeros elsewhere. The unitary matrix U is defined as follows:

[0017]

[0018] URU H The six different parameters of are taken as 6-dimensional real-valued vectors, and the parameters of the 6-dimensional real-valued vector are expressed as follows:

[0019]

[0020] Furthermore, the input samples in step 2) are: around each pixel in the polarimetric SAR image, an image block of size w×w×6 is extracted as the input of the spatial feature extraction module. For the area beyond the boundary, it is expanded by filling with 0, and the field extraction is performed on the real and complex image data respectively as the input samples.

[0021] Further, step 3) is as follows:

[0022] The whole model mainly consists of two branch networks, namely, feature extraction modules based on CV-CNN and CNN, fully connected layers, normalized exponential functions, supervised learning modules and cross pseudo-supervision modules. The two neural networks follow the principle of the same structure to process polarimetric SAR image data in different data domains. A six-layer network structure is used to extract data features, including three convolutional layers and three maximum pooling layers. Each convolutional layer is followed by a BN operation and a ReLU activation function. The top layer of the two neural networks is composed of a fully connected layer, which contains a ReLU activation function.

[0023] Further, step 4) is as follows:

[0024] In the supervision module, the labeled samples are subjected to the real samples and the loss function of the supervision part is simply calculated according to formula (5)

[0025]

[0026] Where y1 and y2 are the one-hot outputs of the two networks; in the cross-pseudo-supervision module, the cross-pseudo-supervision loss functions of labeled samples and unlabeled samples are calculated according to equations (6) and (7) respectively. and Finally, the total loss function of the network is obtained by formula (8);

[0027]

[0028]

[0029] Furthermore, the implementation process of step 5) is as follows: after the training is completed, all the tested samples are sent to two perturbation networks respectively, and the predicted probability values ​​obtained based on the two networks are weighted and summed using the mean weight to obtain the final classification result.

[0030] Compared with the prior art, the beneficial effect of the present invention is that the present invention uses labeled and unlabeled samples to train the network through a cross-pseudo-supervision method, and the outputs of the two networks on the labeled samples are supervised by the corresponding labels. Each network estimates a classification result, i.e., a pseudo-label, for the unlabeled sample. The pseudo-label is used as an additional signal to supervise the other classification network. Cross-pseudo-supervision encourages the two perturbation networks to learn the same input consistently and expand the training data using unlabeled samples with pseudo-labels. This can avoid the problem that a large amount of computing resources are consumed by the clustering algorithm to generate pseudo-labels and the quality of the generated pseudo-labels is greatly affected by the number of cluster centers. In addition, real convolution and complex convolution operations are embedded in the network to capture different polarimetric SAR image features. In the experiment, three widely used polarimetric SAR data sets were used to evaluate the effectiveness of the proposed semi-supervised method. The experimental results show that compared with other comparative methods, the proposed semi-supervised model has better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0032] Figure 1 It is a flow chart of the present invention.

[0033] Figure 2 These are the classification results of the true value map range of the AIRSAR L-band image of the Flevoland area, including (a) Ground-truth; (b) CNN; (c) CV-CNN; (d) CNN-CPS; (e) CV-CNN-CPS; and (f) this method.

[0034] Figure 3 These are the classification results of the true value map range of the ESAR L-band image of the Oberpfaffenhofen area, including (a) Ground-truth; (b) CNN; (c) CV-CNN; (d) CNN-CPS; (e) CV-CNN-CPS; and (f) this method. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] like Figure 1 A method for classifying objects in polarimetric SAR remote sensing images based on cross pseudo-supervision of a hybrid convolutional network is shown, comprising the following steps:

[0037] Step 1: Convert the upper triangular elements of the polarization coherence matrix of each pixel in the image into a 6-dimensional complex vector and a 6-dimensional real vector by real number conversion.

[0038] The 6-dimensional complex vector obtained by transforming the polarization coherence matrix is ​​used as the feature of the polarization SAR image. The polarization coherence matrix can be expressed as follows:

[0039]

[0040] Then the 6-dimensional complex vector can be expressed as v_cv = [T 11 ,T 12 ,T 13 ,T 22 ,T 23 ,T 33 ], in order to ensure the same data dimension, we use unitary transformation to convert the complex value matrix into a real value matrix; this method takes advantage of a property that the polarization coherence matrix is ​​not only a Hermitian matrix, but also symmetric. However, the polarization coherence matrix T is a Hermitian matrix, but not a generalized symmetric matrix; therefore, a fully symmetric transformation is proposed by the following formula:

[0041]

[0042] Where R is a Hermitian and generalized symmetric matrix. T is the polarization coherence matrix, and J is a matrix with 1s on the sub-diagonal and zeros elsewhere. The unitary matrix U is defined as follows:

[0043]

[0044] URU H The six different parameters of are taken as 6-dimensional real-valued vectors. Then the parameters of the 6-dimensional real-valued vector are expressed as follows:

[0045]

[0046] Step 2: Extract an image block of size w×w×6 with each pixel in the image as the center as the input sample of the network;

[0047] There is a strong spatial correlation in the neighborhood space of polarimetric SAR remote sensing image pixels. Therefore, in order to extract the spatial features of each pixel in polarimetric SAR remote sensing images, it is necessary to consider the neighborhood space of the pixel. An image block of size w×w×6 is extracted around each pixel in the polarimetric SAR image as the input of the spatial feature extraction module. For areas beyond the boundary, 0 padding is used to expand. Field extraction is performed on real and complex image data respectively as input samples.

[0048] Step 3: Initialize the weights in CV-CNN and CNN with random values ​​that are subject to Gaussian distribution with a mean of 0 and a standard deviation of 0.1; the bias term is initialized to 0.

[0049] like Figure 1 The whole model shown mainly includes two branch networks, which are based on CV-CNN and CNN feature extraction modules, fully connected layers, normalized exponential functions, supervised learning modules, and cross-pseudo-supervisory modules. Among them, the two neural networks follow the principle of the same structure, process polarimetric SAR image data in different data domains, and use a six-layer network structure to extract data features, including 3 convolutional layers and 3 maximum pooling layers. Each convolutional layer is followed by a BN operation and a ReLU activation function. The top layer of both neural networks consists of a fully connected layer, and a ReLU activation function is included after the fully connected layer. The specific structure of the hybrid convolutional neural network is shown in Table 1.

[0050] Step 4: Input the labeled samples and unlabeled samples into the cross pseudo-supervised network and optimize them using the mini-batch stochastic gradient descent algorithm;

[0051] In the supervision module, the labeled samples are subjected to the real samples and the loss function of the supervision part is simply calculated according to formula (5)

[0052] Table 1 shows the structure of the hybrid convolutional neural network

[0053]

[0054]

[0055]

[0056] Where y1 and y2 are the one-hot outputs of the two networks; in the cross-pseudo-supervision module, the cross-pseudo-supervision loss functions of labeled samples and unlabeled samples are calculated according to equations (6) and (7) respectively. and Finally, the total loss function of the network is obtained by equation (8).

[0057]

[0058] Step 5: After training, the neighborhood blocks corresponding to each pixel in the image are input into CV-CNN and CNN, and the predicted probability values ​​of the two networks are fused to complete the image classification.

[0059] After the training, all the tested samples are sent to two perturbation networks respectively. The predicted probability values ​​obtained by the two networks are weighted and summed using the mean weight to obtain the final classification result.

[0060] To verify the classification results of the Flevoland dataset under different label sample ratios, 0.1%, 0.2%, 0.4% and 0.8% label samples were randomly selected as training sets for different land features, and the rest were used as test sets. The numerical results on the Flevoland dataset are shown in Table 2. When the training samples are only 0.1% (an average of 10 types of each land feature), 94.07% classification OA can be achieved. When the training sample used is 0.8%, the method is significantly better than the comparison method when the training samples are the same, and when the number of samples is very small, the present invention can still maintain a good classification accuracy.

[0061] Table 2 Comparison of classification accuracy of different label sample ratios in Flevoland

[0062]

[0063] For the polarimetric SAR image acquired by AIRSAR in the L band of Flevoland, 0.8% of the labeled samples were used as training samples to test the image. The experimental results are shown in Figure 2 and as shown in Table 3. The results show that the present invention effectively utilizes the information of unlabeled samples, improves the problem of insufficient learning ability of the network model in the case of small samples, and improves the classification performance.

[0064] Table 3 Comparison of classification accuracy of different methods in Flevoland region

[0065]

[0066] For the polarimetric SAR image of the Oberpfaffenhofen area, 1% of the labeled samples were used as training samples to test the image. The experimental results are shown in Figure 3 As shown in Table 4. The results further prove that the method of the present invention is effective, which effectively utilizes the information of unlabeled samples, greatly improves the classification results, and increases the classification accuracy.

[0067] Table 4 Comparison of classification accuracy of different methods in Oberpfaffenhofen area

[0068]

[0069] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision, characterized in that: The following steps are involved: Step 1) converting the upper triangular elements of the polarization coherence matrix of each pixel in the image into a 6-dimensional complex vector and a 6-dimensional real vector by real number conversion; Step 2) Extract an image block of size w×w×6 with each pixel in the image as the center as the input sample of the network; Step 3) Initialize the weights in CV-CNN and CNN using random values; Step 4) Input the labeled samples and unlabeled samples into the cross pseudo-supervised network and optimize them using a mini-batch stochastic gradient descent algorithm; Step 5) After the training is completed, the neighborhood block corresponding to each pixel in the image is input into CV-CNN and CNN, and the predicted probability values ​​of the two networks are fused to complete the image classification.

2. The polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision according to claim 1 is characterized in that: Step 1) The upper triangular elements of the polarization coherence matrix of each pixel in the image are converted into a 6-dimensional complex vector and a 6-dimensional real vector by real number conversion. The implementation process is as follows: the 6-dimensional complex vector converted from the polarization coherence matrix is ​​used as the feature of the polarization SAR image, where the polarization coherence matrix can be expressed by the following formula: Then the 6-dimensional complex vector can be expressed as v_cv = [T 11 ,T 12 ,T 13 ,T 22 ,T 23 ,T 33 ], using unitary transformation to convert complex-valued matrices into real-valued matrices, a fully symmetric transformation is proposed by the following formula: Where R is a Hermitian and generalized symmetric matrix; T is a polarization coherence matrix, J is a matrix with 1s on the sub-diagonal and zeros elsewhere; and the unitary matrix U is defined as follows: URU H The six different parameters of are taken as 6-dimensional real-valued vectors, and the parameters of the 6-dimensional real-valued vector are expressed as follows:

3. The polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision according to claim 1 or 2, characterized in that: The input samples in step 2) are: around each pixel in the polarimetric SAR image, an image block of size w×w×6 is extracted as the input of the spatial feature extraction module. For the area beyond the boundary, it is expanded by filling with 0, and the field extraction is performed on the real and complex image data respectively as the input samples.

4. The polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision according to claim 1 or 2, characterized in that: Step 3) is as follows: The whole model mainly consists of two branch networks, namely, feature extraction modules based on CV-CNN and CNN, fully connected layers, normalized exponential functions, supervised learning modules and cross pseudo-supervision modules. The two neural networks follow the principle of the same structure to process polarimetric SAR image data in different data domains. A six-layer network structure is used to extract data features, including three convolutional layers and three maximum pooling layers. Each convolutional layer is followed by a BN operation and a ReLU activation function. The top layer of the two neural networks is composed of a fully connected layer, which contains a ReLU activation function.

5. The polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision according to claim 1 or 2, characterized in that: Step 4) is as follows: In the supervision module, the labeled samples are subjected to the real samples and the loss function of the supervision part is simply calculated according to formula (5) Where y1 and y2 are the one-hot outputs of the two networks; in the cross-pseudo-supervision module, the cross-pseudo-supervision loss functions of labeled samples and unlabeled samples are calculated according to equations (6) and (7) respectively. and Finally, the total loss function of the network is obtained by formula (8); 6. The polarimetric SAR object classification method based on hybrid convolutional neural network cross pseudo supervision according to claim 1 or 2, characterized in that: The implementation process of step 5) is as follows: after the training is completed, all the tested samples are sent to two perturbation networks respectively, and the predicted probability values ​​obtained based on the two networks are weighted summed using the mean weight to obtain the final classification result.

Images