End-to-end polarimetric SAR image superpixel segmentation method, system and device based on full convolutional neural network, and medium

Abstract

The invention discloses an end-to-end polarimetric SAR image superpixel segmentation method, system and device based on a full convolutional neural network, and a medium. The method comprises the following steps: inputting polarimetric features and position information; constructing a super-pixel segmentation model of the polarized SAR image based on the full convolutional neural network, and outputting a super-pixel segmentation result; training a superpixel segmentation model; inputting the processed polarization features associated with the position information into the trained super-pixel segmentation model to obtain a super-pixel segmentation result, processing the super-pixel segmentation result, and using the processed super-pixel segmentation result as the input of a polarized SAR image classification model to train the polarized SAR image classification model; classifying the polarimetric SAR images by using the trained polarimetric SAR image classification model to obtain a polarimetric SAR image classification result combined with superpixel segmentation; the system, the equipment and the medium are used for implementing the method. The method has the advantages that a high-quality superpixel segmentation result is quickly generated without a traditional clustering method, and the method is easy to combine with a subsequent polarized SAR image classification deep learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to an end-to-end polarimetric SAR image superpixel segmentation method, system, device and medium based on a fully convolutional neural network. Background Art

[0002] In recent years, polarimetric SAR images have been widely used in military and civilian fields. Polarimetric SAR image classification, as a type of polarimetric SAR image interpretation, refers to the use of algorithms or models to extract and process the scattering, spatial and other information of the pixel itself to obtain its category, thereby obtaining the actual type of ground object corresponding to the pixel. Since pixel-based classification technology only marks each pixel independently and ignores the influence of the spatial relationship between pixels, the speckle noise caused by the coherent imaging mechanism seriously affects the classification performance of polarimetric SAR images. Therefore, polarimetric SAR classification methods based on superpixels have attracted widespread attention.

[0003] The patent application document with publication number CN110827290A discloses a watershed-based polarimetric SAR image superpixel segmentation method. This method uses an improved watershed segmentation method to solve the superpixel segmentation problem of polarimetric SAR images. However, before superpixel segmentation, it is necessary to use polarization information to perform gradient calculation. After obtaining the superpixel segmentation result, post-processing and elimination of noise areas are required. The workload is too large, and it is not combined with deep learning methods, which affects the classification efficiency and classification accuracy to a certain extent.

[0004] Junjun Yin et al. proposed a traditional clustering superpixel segmentation method for solving the superpixel segmentation problem of polarimetric SAR images in "SLIC Superpixel Segmentation for Polarimetric SAR Images, IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022, Art no. 5201317, doi: 10.1109 / TGRS.2020.3047126". This method uses a modified SLIC clustering function to perform superpixel segmentation of polarimetric SAR images. The steps of this method are: (1) initialization; (2) local K-means clustering; (3) post-processing. This method has achieved good performance on three widely used polarimetric SAR image datasets; however, this method is limited by traditional clustering-based methods, has poor superpixel segmentation performance, and cannot be combined with deep learning methods.

[0005] Lei Wang et al. proposed a traditional clustering superpixel segmentation method for solving the problem of superpixel segmentation of polarimetric SAR images in "PolSAR-SSN: An end-to-end superpixel sampling network for PolSAR image Classification, IEEE Geosci. Remote Sens. Lett., 2022, vol. 19, pp. 1-5, Art no. 4505305, doi: 10.1109 / LGRS.2022.3152794". This method uses a deep neural network to extract the features of polarimetric SAR images and perform superpixel segmentation of polarimetric SAR images. Although end-to-end training of polarimetric SAR image segmentation is achieved, the method only uses a deep neural network for polarimetric feature extraction and fails to completely get rid of the dependence on the traditional clustering superpixel segmentation algorithm. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an end-to-end polarimetric SAR image superpixel segmentation method, system, device and medium based on a fully convolutional neural network, use polarization features as the input of a superpixel segmentation model of a polarimetric SAR image based on a fully convolutional neural network to make full use of polarization information, construct a fully convolutional neural network through an encoder-decoder structure to generate superpixel segmentation results, adopt a flexible loss function to achieve end-to-end training, and promote the application of superpixel segmentation in polarimetric SAR image interpretation; the present invention can efficiently obtain polarimetric SAR image superpixel segmentation results by combining polarimetric feature extraction with the superpixel segmentation process, and can be easily combined with a polarimetric SAR image classification network to improve image classification performance.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] An end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network comprises the following steps:

[0009] Step 1: Select a polarization coherence matrix as the polarization feature of each pixel in the polarimetric SAR image, process the polarization coherence matrix, extract the elements of the upper triangular part of the matrix, decompose the real part and the imaginary part, associate the position information of each pixel with the polarization feature, and obtain the processed polarization feature associated with the position information;

[0010] Step 2: construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through an encoder-decoder structure with skip connections, use the polarization features processed in step 1 and associated with position information as the input of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network, adopt a superpixel segmentation strategy on a regular network, and output a superpixel segmentation result;

[0011] Step 3, for the super-pixel segmentation task of polarimetric SAR images, a loss function is designed to train the super-pixel segmentation model of polarimetric SAR images based on a fully convolutional neural network constructed in step 2;

[0012] Step 4, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3 to obtain superpixel segmentation results, and processing the superpixel segmentation results as input to the polarimetric SAR image classification model to train the polarimetric SAR image classification model;

[0013] Step 5: Classify the polarimetric SAR image using the polarimetric SAR image classification model trained in step 4 to obtain a polarimetric SAR image classification result combined with superpixel segmentation.

[0014] The polarization characteristics of the position information after processing in step 1 are as follows:

[0015] t p =[T 11 ,T 22 ,T 33 ,Re[T 12 ],Im[T 12 ],Re[T 13 ],Im[T 13 ],Re[T 23 ],Im[T 23 ],row,col]

[0016] Where, T ij Represents the elements in the polarization coherence matrix, i, j = 1, 2, 3, Re[.], Im[.] represent the real part and imaginary part respectively, row, col represent the position coordinates of the pixel.

[0017] The structure of the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network in step 2 includes an encoder, a decoder and a prediction layer;

[0018] The encoder consists of five layers of convolutional neural networks, with the kernel size set to 3×3 and the number of channels set to 16, 32, 64, 128 and 256 respectively. During the convolution process, the input of the first layer of the encoder convolutional neural network is padded according to the size of the feature map;

[0019] The decoder consists of four deconvolution layers, with a kernel size of 4×4 and the number of channels set to 16, 32, 64, and 128, respectively. The feature maps are upsampled step by step, and the output features of each deconvolution layer in the decoder are associated with the output features of the convolution layer with the same number of channels in the encoder to obtain the final superpixel prediction. The activation function Leaky ReLU is used for all convolution layers in the encoder and all deconvolution layers in the decoder.

[0020] The prediction layer is a convolutional layer, the kernel size is set to 3×3, the number of channels is set to 9, the output result of the decoder is input into the prediction layer, and the output result of the prediction layer generates a superpixel association matrix through softmax.

[0021] The specific process of the superpixel segmentation strategy on the rule network in step 2 includes:

[0022] Step 2.11, segment the input image using a regular grid with a configurable size, and consider each grid as an initial superpixel;

[0023] Step 2.12, assign each pixel to one of the superpixels, and record this mapping relationship in the pixel-superpixel association matrix H, calculated as follows:

[0024]

[0025] In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, D is the distance metric, when a pixel belongs to a superpixel, the corresponding value in H is 1, otherwise it is 0; only the mapping relationship between the pixel and its surrounding nine superpixels is considered;

[0026] Step 2.13, the pixel-superpixel correlation matrix H of step 2.12 is processed by soft correlation mapping, so that the final superpixel soft correlation matrix Q is differentiable, and the calculation formula is as follows:

[0027] Q = exp(-D(I P ,S i ))

[0028] In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, and D is the distance metric.

[0029] The specific method of step 2 includes:

[0030] Step 2.21, using the polarization features processed in step 1 and associated with the position information as the input of the superpixel segmentation model of the polarization SAR image based on the fully convolutional neural network, extracting the polarization features of the polarization SAR image layer by layer through the encoder to obtain a feature map;

[0031] Step 2.22, upsampling the feature map obtained in step 2.21 through the decoder, and associating it with the output features of the convolutional layer in the encoder with the same number of channels to obtain the output result of the encoder;

[0032] In step 2.23, the output result of the encoder in step 2.22 is input to the prediction layer, and the output result of the prediction layer is used to generate a superpixel association matrix through softmax to obtain the final superpixel prediction.

[0033] The loss function designed in step 3 is:

[0034] Assume f(p) is the feature corresponding to a given pixel p, p = [x, y] T Used to represent the position of a given pixel, based on the superpixel soft correlation matrix Q obtained in step 2.13, calculate the center c of any superpixel s s =(u s ,l s ), where u s is the feature vector, l s is the position vector; the calculation method is as follows:

[0035]

[0036] Among them, N p is the set of superpixels around p, q s (p) is the predicted probability of association between p and the superpixel; the reconstructed feature f'(p) and position p' of any pixel are calculated as follows:

[0037]

[0038] In the formula, u s is the feature vector of the pixel, l s is the position vector of the pixel; finally, the loss function consists of two terms. The first term encourages the trained superpixel segmentation model of polarimetric SAR images based on fully convolutional neural networks to group pixels with similar features; the second term enforces the spatial compactness of superpixels, selects the one-hot encoding vector of the ground truth label, and uses cross entropy as the distance metric. The loss function is calculated as follows:

[0039]

[0040] where lm is a configurable weight, ΔS is the superpixel sampling interval and is related to the size of a regular grid of configurable size, and E(·,·) is the cross entropy loss function.

[0041] The specific method of step 4 includes:

[0042] The polarimetric SAR image classification model adopts a three-layer convolutional neural network, whose convolution kernel sizes are 3×3, 2×2 and 2×2, and the number of output channels are 16, 32, and 64 respectively;

[0043] Step 4.1, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3, obtaining a superpixel segmentation result with irregular superpixel blocks, and calculating the average value of the features of each irregular superpixel block;

[0044] Step 4.2, expand the superpixel block into a rectangle, and fill the blank part with the average value of the features obtained in step 4.1;

[0045] Step 4.3, adjust the filled rectangle in step 4.2 to a set size and use it as the input of the polarimetric SAR image classification model;

[0046] In step 4.4, the polarimetric SAR image classification model is trained using the Adam optimizer and cross entropy loss at the set learning rate and number of iterations to obtain a trained polarimetric SAR image classification model.

[0047] The present invention also provides an end-to-end polarimetric SAR image superpixel segmentation system based on a fully convolutional neural network, comprising:

[0048] The polarization feature input module is used to select the polarization coherence matrix as the polarization feature of each pixel in the polarization SAR image, and after processing the polarization coherence matrix, associate the position information of each pixel with the polarization feature to obtain the processed polarization feature associated with the position information;

[0049] The segmentation module of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network is used to construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through a fully convolutional neural network with an encoder-decoder structure with skip connections, use the processed polarization features with associated position information as the input of the superpixel segmentation model of polarimetric SAR images based on the fully convolutional neural network, adopt the superpixel segmentation strategy on the regular network, and output the superpixel segmentation result;

[0050] The super-pixel segmentation model training module of polarimetric SAR images based on a fully convolutional neural network is used to design a loss function for the super-pixel segmentation task of polarimetric SAR images and train the super-pixel segmentation model of polarimetric SAR images based on a fully convolutional neural network.

[0051] The polarimetric SAR image classification model training module is used to input the processed polarimetric features associated with the position information into the trained super-pixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network to obtain the super-pixel segmentation results, and use the processed super-pixel segmentation results as the input of the polarimetric SAR image classification model to train the polarimetric SAR image classification model.

[0052] The polarimetric SAR image classification module is used to classify polarimetric SAR images using the trained polarimetric SAR image classification model to obtain polarimetric SAR image classification results combined with superpixel segmentation.

[0053] The present invention also provides an end-to-end polarimetric SAR image superpixel segmentation device based on a fully convolutional neural network, comprising:

[0054] Memory: a computer program storing the above-mentioned end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network, which is a computer-readable device;

[0055] Processor: used to implement the end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network when executing the computer program.

[0056] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the end-to-end polarization SAR image superpixel segmentation method based on a fully convolutional neural network.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention constructs a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through step 2, and uses a deep learning method instead of a traditional clustering method to achieve end-to-end superpixel segmentation. The present invention can directly obtain superpixel segmentation results by a deep neural network without the participation of a traditional clustering superpixel segmentation algorithm; compared with the prior art, the problem of the superpixel segmentation algorithm's dependence on the traditional clustering superpixel segmentation algorithm and the difficulty in combining it with a deep learning method is solved.

[0059] 2. The present invention uses a fully convolutional neural network with an encoder-decoder structure with jump connections in step 2 to construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network. It adopts a superpixel segmentation strategy on a regular network and combines feature extraction and segmentation processes to directly obtain superpixel segmentation results. Compared with the prior art, it solves the problem of large workload and can obtain superpixel segmentation results more efficiently and accurately.

[0060] 3. The present invention obtains a superpixel segmentation result by constructing a superpixel segmentation model of polarization SAR images based on a full convolutional neural network in step 2, and obtains a polarization SAR image classification result by processing the superpixel segmentation result in step 4 as the input of the polarization SAR image classification model. The constructed superpixel segmentation model of polarization SAR images based on a full convolutional neural network can be better combined with the polarization SAR image classification model based on deep learning, and can effectively ensure the accuracy of ground object classification while reducing the dimension.

[0061] 4. The present invention selects a polarization coherence matrix as the polarization feature of each pixel in the polarization SAR image through step 1, and after processing the polarization coherence matrix, associates the position information of each pixel with the polarization feature, and obtains the processed polarization feature with associated position information as input. Compared with the prior art, the present invention solves the problem of performance degradation caused by using pseudo-color images as input, and can better extract the features of polarization SAR images and obtain high-precision super-pixel segmentation results.

[0062] In summary, the present invention selects the polarization coherence matrix of the associated position information as input, constructs a fully convolutional neural network with an encoder-decoder structure with jump connections, obtains the polarization SAR image classification result by processing the superpixel segmentation result as the input of the polarization SAR image classification model, and realizes end-to-end polarization SAR image superpixel segmentation considering polarization features. The present invention has the advantages of quickly generating high-quality superpixel segmentation results without the need for traditional clustering methods and being easy to combine with the subsequent polarization SAR image classification deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0064] Figure 2 This is a structural diagram of the superpixel segmentation model of polarization SAR images based on a fully convolutional neural network in the present invention.

[0065] Figure 3 This is a structural diagram of the polarization SAR image classification model of the present invention.

[0066] Figure 4This is a comparison chart of the superpixel segmentation results of the polarimetric SAR image of the San Francisco area in the United States between the present invention and the prior art SLIC and PolSAR-SSN; wherein, Figure 4 (a) is the real annotation map of the polarimetric SAR image of the San Francisco area in the United States. Figure 4 (b) is a superpixel segmentation map obtained by simulation using the method of the present invention, Figure 4 (c) is the superpixel segmentation map obtained by SLIC simulation in the prior art. Figure 4 (d) is the superpixel segmentation map obtained by PolSAR-SSN simulation of the existing technology.

[0067] Figure 5 This is a comparison chart of the superpixel segmentation results of the polarimetric SAR image of the Oberpfaffenhofen area in Germany between the present invention and the prior art SLIC and PolSAR-SSN; wherein, Figure 5 (a) is the real annotation map of the polarimetric SAR image of Oberpfaffenhofen, Germany. Figure 5 (b) is a superpixel segmentation map obtained by simulation using the method of the present invention, Figure 5 (c) is the superpixel segmentation map obtained by SLIC simulation in the prior art. Figure 5 (d) is the superpixel segmentation map obtained by PolSAR-SSN simulation of the existing technology.

[0068] Figure 6 This is a comparison chart of the superpixel segmentation results of the polarimetric SAR image of the Weihe River area in Xi'an between the present invention and the prior art SLIC and PolSAR-SSN; Figure 6 (a) is the real annotation map of the polarimetric SAR image of the Weihe River area in Xi’an. Figure 6 (b) is a superpixel segmentation map obtained by simulation using the method of the present invention, Figure 6 (c) is the superpixel segmentation map obtained by SLIC simulation in the prior art. Figure 6 (d) is the superpixel segmentation map obtained by PolSAR-SSN simulation of the existing technology.

[0069] Figure 7 This is a comparison chart of the image classification results of the polarimetric SAR images of the San Francisco area in the United States, in the present invention and the prior art SLIC and PolSAR-SSN; wherein, Figure 7 (a) is the classification map obtained by the pixel-level convolutional neural network method. Figure 7 (b) is a classification diagram obtained by simulation using the method of the present invention, Figure 7 (c) is the classification diagram obtained by SLIC simulation of the prior art. Figure 7 (d) is the classification diagram obtained by PolSAR-SSN simulation of the existing technology.

[0070] Figure 8 This is a comparison chart of the image classification results of the polarimetric SAR image of the Oberpfaffenhofen area in Germany between the present invention and the prior art SLIC and PolSAR-SSN; wherein, Figure 8 (a) is the classification map obtained by the pixel-level convolutional neural network method. Figure 8 (b) is a classification diagram obtained by simulation using the method of the present invention, Figure 8 (c) is the classification diagram obtained by SLIC simulation of the prior art. Figure 8 (d) is the classification diagram obtained by PolSAR-SSN simulation of the existing technology.

[0071] Fig. 9 This is a comparison chart of the image classification results of the polarimetric SAR image of the Weihe River area in Xi'an, the present invention and the prior art SLIC and PolSAR-SSN; wherein, Fig. 9 (a) is the classification map obtained by the pixel-level convolutional neural network method. Fig. 9 (b) is a classification diagram obtained by simulation using the method of the present invention, Fig. 9 (c) is the classification diagram obtained by SLIC simulation of the prior art. Fig. 9 (d) is the classification diagram obtained by PolSAR-SSN simulation of the existing technology. DETAILED DESCRIPTION

[0072] The technical solution adopted by the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0073] Reference Figure 1 , an end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network, comprising the following steps:

[0074] Step 1: Select the polarization coherence matrix as the polarization feature of each pixel in the polarimetric SAR image, process the polarization coherence matrix, extract the elements of the upper triangular part of the matrix, and after decomposing the real part and the imaginary part, associate the position information of each pixel with the polarization feature to obtain the processed polarization feature associated with the position information; as the input of the subsequent superpixel segmentation method;

[0075] The polarization characteristics of the position information after processing in step 1 are as follows:

[0076] t p =[T 11 ,T 22 ,T 33 ,Re[T 12 ],Im[T 12 ],Re[T 13 ],Im[T 13 ],Re[T 23],Im[T 23 ],row,col]

[0077] Where, T ij Represents the elements in the polarization coherence matrix, i, j = 1, 2, 3, Re[.], Im[.] represent the real part and imaginary part respectively, row, col represent the position coordinates of the pixel.

[0078] Step 2: construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through an encoder-decoder structure with skip connections, use the polarization features processed in step 1 and associated with position information as the input of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network, adopt a superpixel segmentation strategy on a regular network, and output a superpixel segmentation result;

[0079] Reference Figure 2 , the structure of the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network in step 2 includes an encoder, a decoder and a prediction layer;

[0080] The encoder for polarimetric SAR image feature extraction consists of a five-layer convolutional neural network with a kernel size of 3×3 and the number of channels set to 16, 32, 64, 128, and 256, respectively. During the convolution process, the input of the first layer of the encoder convolutional neural network is appropriately padded according to the size of the feature map;

[0081] The decoder consists of four deconvolution layers with a kernel size of 4×4 and the number of channels set to 16, 32, 64, and 128, respectively. The feature maps are upsampled step by step, and the output features of each deconvolution layer in the decoder are associated with the output features of the convolution layer with the same number of channels in the encoder to obtain the final superpixel prediction. The activation function LeakyReLU is used for all convolution layers in the encoder and all deconvolution layers in the decoder.

[0082] The prediction layer is a convolutional layer with a kernel size of 3×3 and a number of channels of 9. The output of the decoder is input into the prediction layer, and the output of the prediction layer generates a superpixel association matrix through softmax.

[0083] The specific process of the superpixel segmentation strategy on the rule network in step 2 includes:

[0084] Step 2.11, segment the input image using a regular grid with a configurable size, and consider each grid as an initial superpixel;

[0085] Step 2.12, assign each pixel to one of the superpixels, and record this mapping relationship in the pixel-superpixel association matrix H, calculated as follows:

[0086]

[0087] In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, D is the distance metric, when a pixel belongs to a superpixel, the corresponding value in H is 1, otherwise it is 0; only the mapping relationship between the pixel and its surrounding nine superpixels is considered;

[0088] Step 2.13, the pixel-superpixel correlation matrix H of step 2.12 is processed by soft correlation mapping, so that the final superpixel soft correlation matrix Q is differentiable, and the calculation formula is as follows:

[0089] Q = exp(-D(I P ,S i ))

[0090] In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, and D is the distance metric;

[0091] The specific method of step 2 includes:

[0092] Step 2.21, using the polarization features processed in step 1 and associated with the position information as the input of the superpixel segmentation model of the polarization SAR image based on the fully convolutional neural network, extracting the polarization features of the polarization SAR image layer by layer through the encoder to obtain a feature map;

[0093] Step 2.22, upsampling the feature map obtained in step 2.21 through the decoder, and associating it with the output features of the convolutional layer in the encoder with the same number of channels to obtain the output result of the encoder;

[0094] In step 2.23, the output result of the encoder in step 2.22 is input to the prediction layer, and the output result of the prediction layer is used to generate a superpixel association matrix through softmax to obtain the final superpixel prediction.

[0095] Step 3: Design a flexible loss function for the superpixel segmentation task of polarimetric SAR images, use the Adam optimizer and cross entropy loss, and train the superpixel segmentation model of polarimetric SAR images based on the fully convolutional neural network constructed in step 2 at the set learning rate and number of iterations.

[0096] The loss function designed in step 3 is:

[0097] Assume f(p) is the feature corresponding to a given pixel p, p = [x, y] TUsed to represent the position of a given pixel, based on the superpixel soft correlation matrix Q obtained in step 2.13, calculate the center c of any superpixel s s =(u s ,l s ), where u s is the feature vector, l s is the position vector; the calculation method is as follows:

[0098]

[0099] Among them, N p is the set of superpixels around p, q s (p) is the predicted probability of association between p and the superpixel; the reconstructed feature f'(p) and position p' of any pixel are calculated as follows:

[0100]

[0101] In the formula, u s is the feature vector of the pixel, l s is the position vector of the pixel; finally, the loss function consists of two terms. The first term encourages the trained superpixel segmentation model of polarimetric SAR images based on fully convolutional neural networks to group pixels with similar features; the second term enforces the spatial compactness of superpixels, selects the one-hot encoding vector of the ground truth label, and uses cross entropy as the distance metric. The loss function is calculated as follows:

[0102]

[0103] where lm is a configurable weight, ΔS is the superpixel sampling interval and is related to the size of a regular grid of configurable size, and E(·,·) is the cross entropy loss function.

[0104] Step 4, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3 to obtain superpixel segmentation results, and processing the superpixel segmentation results as input to the polarimetric SAR image classification model to train the polarimetric SAR image classification model;

[0105] Reference Figure 3 , the specific method of step 4 includes:

[0106] The polarimetric SAR image classification model adopts a simple three-layer convolutional neural network, whose convolution kernel sizes are 3×3, 2×2 and 2×2, and the number of output channels are 16, 32, and 64 respectively;

[0107] Step 4.1, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3, obtaining a superpixel segmentation result with irregular superpixel blocks, and calculating the average value of the features of each irregular superpixel block;

[0108] Step 4.2, expand the superpixel block into a rectangle, and fill the blank part with the average value of the features obtained in step 4.1;

[0109] Step 4.3, adjust the filled rectangle in step 4.2 to the set size (here set to 16×16) and use it as the input of the polarimetric SAR image classification model;

[0110] In step 4.4, the polarimetric SAR image classification model is trained using the Adam optimizer and cross entropy loss at the set learning rate and number of iterations to obtain a trained polarimetric SAR image classification model.

[0111] Step 5: Classify the polarimetric SAR image using the polarimetric SAR image classification model trained in step 4 to obtain a polarimetric SAR image classification result combined with superpixel segmentation.

[0112] The technical problems to be solved by the present invention are:

[0113] (1) Most of the existing polarimetric SAR image superpixel segmentation methods rely on traditional clustering methods, which cannot be closely integrated with deep learning to achieve end-to-end training. While reducing the dimension, they cannot effectively guarantee the accuracy of ground object classification of the subsequent polarimetric SAR deep learning model. The present invention, by constructing a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network with an encoder-decoder structure with jump connections, can use deep learning methods instead of traditional clustering methods to achieve end-to-end superpixel segmentation, and combine feature extraction and segmentation processes to efficiently and accurately obtain superpixel segmentation results.

[0114] (2) Currently, most polarimetric SAR image superpixel segmentation methods can only accept pseudo-color images as input, while ignoring the polarimetric features of the polarimetric SAR images themselves. The quality of the generated superpixels cannot guarantee the accuracy of subsequent ground object classification tasks. However, the present invention selects the polarimetric coherence matrix as the polarimetric feature of each pixel in the polarimetric SAR image and associates the position information of each pixel with the polarimetric feature, thereby solving the performance degradation problem caused by using pseudo-color images as input.

[0115] Experimental analysis

[0116] 1. Simulation conditions

[0117] The hardware platform of the simulation experiment of the present invention is: Lianzhong workstation, the CPU model is Intel Xeon Silver 4214R, the frequency is 2.40 GHZ, the GPU model is NVIDIA GeForce RTX 3090, and the video memory is 24 GB.

[0118] The software platform of the simulation experiment of the present invention is: Ubuntu 22.10 system, Python 3.9, Torch 1.13.

[0119] The input images used in the simulation experiment of the present invention are: a polarimetric SAR dataset of the San Francisco area in the United States, a polarimetric SAR dataset of the Oberpfaffenhofen area in Germany, and a polarimetric SAR dataset of the Weihe River area in Xi'an.

[0120] 2. Simulation content

[0121] The simulation experiment of the present invention adopts the present invention and the prior art SLIC and PolSAR-SSN to perform superpixel segmentation and image classification tests on the annotated samples in the polarimetric SAR images of the San Francisco area in the United States, the polarimetric SAR images of the Oberpfaffenhofen area in Germany, and the polarimetric SAR images of the Weihe area in Xi'an.

[0122] The prior art SLIC refers to: a traditional clustering method for solving superpixel segmentation of natural images proposed by Y. Liu et al. in “Manifold slic: a fast method to compute content-sensitive superpixels, CVPR, Las Vegas, NV, USA, 27–30 June 2016; pp. 651–659.”, referred to as SLIC.

[0123] The prior art PolSAR-SSN refers to: L. Wang et al. proposed a deep learning-based method for solving superpixel segmentation of polarization SAR images in “PolSAR-SSN: An end-to-end superpixel sampling network for PolSAR image Classification, IEEE Geosci. Remote Sens. Lett., 2022, vol. 19, pp. 1-5, Art no. 4505305, doi: 10.1109 / LGRS.2022.3152794”, referred to as PolSAR-SSN.

[0124] Simulation experiment 1: The present invention and the prior art SLIC and PolSAR-SSN are used to conduct simulation experiments on polarimetric SAR image data of the San Francisco area in the United States. The polarimetric SAR image of the San Francisco area in the United States was collected by NASA / Jet Propulsion Laboratory AIRSAR in 2008. The band is C band and the image size is 1895×1419. The real annotation map is as follows: Figure 4 (a) shown. Figure 4 There are five types of objects in (a), namely, category 1, category 2, category 3, category 4, and category 5, and the total number of labeled samples is 1,886,740. Simulation experiment 1 selects 30% of the labeled samples of each type of object samples in the image to form a training set, and the remaining labeled samples form a validation set, and then performs simulation to obtain the superpixel segmentation results of each sample in the validation set by the present invention and the prior art SLIC and PolSAR-SSN. The superpixel segmentation results obtained by the present invention and the prior art SLIC and PolSAR-SSN are plotted as Figure 4 (b) Figure 4 (c) and Figure 4 (d) shows the superpixel segmentation result.

[0125] Depend on Figure 4 (b) Figure 4 (c) and Figure 4 The superpixel segmentation results shown in (d) are respectively Figure 4 By comparing the real annotation image shown in (a), it can be seen that the superpixel segmentation result image obtained by the method of the present invention is consistent with the real boundary well, and most superpixel blocks contain only one category of pixels, which is very close overall. Figure 4 (a) shows the real annotation image, while in the superpixel segmentation result image obtained by the existing technologies SLIC and PolSAR-SSN, the boundary fitting is poor.

[0126] In order to verify the effect of the simulation experiment 1 of the present invention, six indicators, namely boundary recall rate, under-segmentation error, compactness, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the existing technologies SLIC and PolSAR-SSN.

[0127] The calculation formulas of the following six indicators are used to quantify the superpixel segmentation results and various types of ground objects in the simulation experiment 1. The results are shown in Table 1.

[0128]

[0129] Where TP represents the number of boundary pixels in the superpixel S that exist in the real annotation G in the specified neighborhood, and FN represents the number of boundary pixels in the superpixel S that do not exist in the real annotation G in the specified neighborhood.

[0130]

[0131] The under-segmentation error represents the ratio of the number of pixels outside the intersection of the superpixel S and the true segmentation G to the number of pixels in the true segmentation.

[0132]

[0133] Where A(.) represents the area of ​​the superpixel, and P(.) represents the area of ​​a circle with the same circumference as the superpixel.

[0134]

[0135] Table 1 Quantization results of simulation experiment 1

[0136]

[0137]

[0138] Table 1 lists the indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. It can be seen from Table 1 that the boundary recall rate of the method of the present invention is improved by 17.05% compared with the prior art, the under-segmentation error is reduced by 17.80%, the overall accuracy is improved by 2.94% compared with the prior art, and the average accuracy is improved by 1.22% compared with the prior art, and the optimal classification accuracy is obtained in all categories of ground object classification accuracy.

[0139] Simulation experiment 2: The present invention and the prior art SLIC and PolSAR-SSN are used to conduct simulation experiments on the polarimetric SAR image data of the Oberpfaffenhofen area in Germany. The polarimetric SAR image of the Oberpfaffenhofen area in Germany was collected by the ESAR airborne platform in 1999. The band is L band and the image size is 1200×1300. The real annotation map is as follows: Figure 5 (a) shown. Figure 5 In (a), there are three types of objects, namely, category 1, category 2, and category 3, and the total number of labeled samples is 1385269. Simulation experiment 2 selects 30% of the labeled samples of each type of object samples in the image to form a training set, and the remaining labeled samples form a validation set, and then performs simulation to obtain the superpixel segmentation results of each sample in the validation set by the present invention and the prior art SLIC and PolSAR-SSN. The superpixel segmentation results obtained by the present invention and the prior art SLIC and PolSAR-SSN are plotted as Figure 5 (b) Figure 5 (c) and Figure 5 (d) shows the superpixel segmentation result.

[0140] Depend on Figure 5 (b) Figure 5 (c) and Figure 5 The superpixel segmentation results shown in (d) are respectively Figure 5 By comparing the real annotation image shown in (a), it can be seen that the superpixel segmentation result image obtained by the method of the present invention is consistent with the real boundary well, and most superpixel blocks contain only one category of pixels, which is very close overall. Figure 5 (a) shows the real annotation image, while in the superpixel segmentation result image obtained by the existing technologies SLIC and PolSAR-SSN, the boundary fitting is poor.

[0141] In order to verify the effect of simulation experiment 2 of the present invention, six indicators, namely boundary recall rate, under-segmentation error, compactness, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the existing technologies SLIC and PolSAR-SSN.

[0142] Using the same calculation formulas of the six indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 2 are quantified, and the results are shown in Table 2.

[0143] Table 2 Quantization results of simulation experiment 2

[0144]

[0145] Table 2 lists the indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. As can be seen from Table 3, the boundary recall rate of the method of the present invention is improved by 0.25% compared with the prior art, the under-segmentation error is reduced by 3.05%, the compactness is improved by 10.23%, the overall accuracy is improved by 1.14% compared with the prior art, and the average accuracy is improved by 1.20% compared with the prior art, and the optimal classification accuracy is obtained in all categories of ground object classification accuracy.

[0146] Simulation experiment 3: The present invention and the prior art SLIC and PolSAR-SSN are used to simulate the polarimetric SAR image data of the Weihe River area in Xi'an. The polarimetric SAR image of the Weihe River area in Xi'an is collected by the RADARSAT-2 satellite. The band is C band and the image size is 512×512. The real annotation map is as follows: Figure 6 (a) shown. Figure 6In (a), there are three types of objects, namely, category 1, category 2, and category 3, and the total number of labeled samples is 201728. Simulation experiment 2 selects 30% of the labeled samples of each type of object samples in the image to form a training set, and the remaining labeled samples form a validation set, and then performs simulation to obtain the superpixel segmentation results of each sample in the validation set by the present invention and the prior art SLIC and PolSAR-SSN. The superpixel segmentation results obtained by the present invention and the prior art SLIC and PolSAR-SSN are plotted as Figure 6 (b) Figure 6 (c) and Figure 6 (d) shows the superpixel segmentation result.

[0147] Depend on Figure 6 (b) Figure 6 (c) and Figure 6 The superpixel segmentation results shown in (d) are respectively Figure 6 By comparing the real annotation image shown in (a), it can be seen that the superpixel segmentation result image obtained by the method of the present invention is consistent with the real boundary well, and most superpixel blocks contain only one category of pixels, which is very close overall. Figure 6 (a) shows the real annotation image, while in the superpixel segmentation result image obtained by the existing technologies SLIC and PolSAR-SSN, the boundary fitting is poor.

[0148] In order to verify the effect of simulation experiment 3 of the present invention, six indicators, namely boundary recall rate, under-segmentation error, compactness, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the existing technologies SLIC and PolSAR-SSN.

[0149] Using the same calculation formulas of the six indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 3 are quantified, and the results are shown in Table 3.

[0150] Table 3 Quantization results of simulation experiment 3

[0151]

[0152]

[0153] Table 3 lists the indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. It can be seen from Table 3 that the boundary recall rate of the method of the present invention is improved by 0.11% compared with the prior art, the under-segmentation error is reduced by 6.97%, the compactness is improved by 21.70%, the overall accuracy is improved by 1.26% compared with the prior art, and the average accuracy is improved by 1.25% compared with the prior art, and the optimal classification accuracy is obtained in all categories of ground object classification accuracy.

[0154] Simulation experiment 4: The polarization SAR image data of San Francisco area in the United States was simulated by using the present invention and the prior art SLIC and PolSAR-SSN. In simulation experiment 4, 30% of the superpixel segmentation results of the image were selected to form a training set, and the remaining superpixel blocks were used to form a validation set for simulation. A convolutional neural network was used to perform pixel-level image classification for comparison. The image classification results of each sample in the validation set were obtained by the present invention and the pixel-level classification method, the prior art SLIC, and PolSAR-SSN. The superpixel segmentation results obtained by the pixel-level classification method, the present invention, the prior art SLIC, and PolSAR-SSN were plotted as Figure 7 (a) Figure 7 (b) Figure 7 (c) and Figure 7 (d) shows the image classification result.

[0155] Depend on Figure 7 (b) Figure 7 (c) and Figure 7 The superpixel segmentation results shown in (d) are respectively Figure 7 By comparing the pixel-level classification result graph shown in (a), it can be seen that the classification result graph obtained by the method of the present invention has fewer and more concentrated misclassified samples, which can effectively assist subsequent image classification tasks, while the superpixel segmentation results obtained by the prior art SLIC and PolSAR-SSN have limited improvement on the classification task.

[0156] In order to verify the effect of simulation experiment 4 of the present invention, three indicators, namely, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the pixel-level classification method, the prior art SLIC, and PolSAR-SSN.

[0157] Using the same calculation formulas of the three indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 4 are quantified, and the results are shown in Table 4.

[0158] Table 4 Quantization results of simulation experiment 4

[0159]

[0160] Table 4 lists the indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. It can be seen from Table 4 that the overall accuracy of the method of the present invention is improved by 0.65% compared with the prior art, and the average accuracy is improved by 1.33% compared with the prior art, and the optimal classification accuracy is obtained in all categories of ground object classification accuracy.

[0161] Simulation experiment 5: The present invention and the prior art SLIC and PolSAR-SSN were used to conduct simulation experiments on the polarimetric SAR image data of the Oberpfaffenhofen area in Germany. In simulation experiment 5, 30% of the superpixel segmentation results of the image were selected to form a training set, and the remaining superpixel blocks were used to form a validation set for simulation. A convolutional neural network was used to perform pixel-level image classification for comparison. The image classification results of each sample in the validation set obtained by the present invention, the pixel-level classification method, the prior art SLIC, and PolSAR-SSN were obtained. The superpixel segmentation results obtained by the pixel-level classification method, the present invention, the prior art SLIC, and PolSAR-SSN were plotted as Figure 8 (a) Figure 8 (b) Figure 8 (c) and Figure 8 (d) shows the image classification result.

[0162] Depend on Figure 8 (b) Figure 8 (c) and Figure 8 The superpixel segmentation results shown in (d) are respectively Figure 8 By comparing the pixel-level classification result graph shown in (a), it can be seen that the classification result graph obtained by the method of the present invention has fewer and more concentrated misclassified samples, which can effectively assist subsequent image classification tasks, while the superpixel segmentation results obtained by the prior art SLIC and PolSAR-SSN have limited improvement on the classification task.

[0163] In order to verify the effect of simulation experiment 5 of the present invention, three indicators, namely, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the pixel-level classification method, the prior art SLIC, and PolSAR-SSN.

[0164] Using the same calculation formulas of the three indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 5 are quantified, and the results are shown in Table 5.

[0165] Table 5 Quantization results of simulation experiment 5

[0166]

[0167] Table 5 lists the indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. It can be seen from Table 5 that the overall accuracy of the method of the present invention is improved by 0.74% compared with the prior art, and the average accuracy is improved by 1.79% compared with the prior art, and the best classification accuracy is obtained in the classification accuracy of category 1 and category 3 objects.

[0168] Simulation experiment 6: The present invention and the prior art SLIC and PolSAR-SSN were used to conduct simulation experiments on the polarimetric SAR image data of the Weihe River area in Xi'an. In simulation experiment 6, 30% of the superpixel segmentation results of the image were selected to form a training set, and the remaining superpixel blocks were used to form a validation set for simulation. A convolutional neural network was used to perform pixel-level image classification for comparison. The image classification results of each sample in the validation set were obtained by the present invention, the pixel-level classification method, the prior art SLIC, and PolSAR-SSN. The superpixel segmentation results obtained by the pixel-level classification method, the present invention, the prior art SLIC, and PolSAR-SSN were plotted as Fig. 9 (a) Fig. 9 (b) Fig. 9 (c) and Fig. 9 (d) shows the image classification result.

[0169] Depend on Fig. 9 (b) Fig. 9 (c) and Fig. 9 The superpixel segmentation results shown in (d) are respectively Fig. 9 By comparing the pixel-level classification result graph shown in (a), it can be seen that the classification result graph obtained by the method of the present invention has fewer and more concentrated misclassified samples, which can effectively assist subsequent image classification tasks, while the superpixel segmentation results obtained by the prior art SLIC and PolSAR-SSN have limited improvement on the classification task.

[0170] In order to verify the effect of simulation experiment 6 of the present invention, three indicators, namely, classification accuracy of each type of ground object, overall classification accuracy and average classification accuracy, are used to quantify the classification results of the method of the present invention and the pixel-level classification method, the prior art SLIC, and PolSAR-SSN.

[0171] Using the same calculation formulas of the three indicators as those in simulation experiment 1, various types of ground objects in the results of simulation experiment 6 are quantified, and the results are shown in Table 6.

[0172] Table 6 Quantization results of simulation experiment 6

[0173]

[0174] Table 6 lists the various indicators of the present invention and the prior art SLIC and PolSAR-SSN respectively. It can be seen from Table 6 that the overall accuracy of the method of the present invention is improved by 1.48% compared with the prior art, and the average accuracy is improved by 2.14% compared with the prior art, and the optimal classification accuracy is obtained in the classification accuracy of category 1 and category 2 objects.

[0175] 3. Simulation results

[0176] The above simulation experiments show that the method of the present invention uses a fully convolutional neural network to realize end-to-end polarimetric SAR image superpixel segmentation, and uses polarimetric features as network input to make full use of polarimetric information. By combining polarimetric feature extraction with the superpixel segmentation process, the polarimetric SAR image superpixel segmentation result can be efficiently obtained, and can be easily combined with the polarimetric SAR image classification network to improve the image classification performance.

[0177] The present invention also provides an end-to-end polarimetric SAR image superpixel segmentation system based on a fully convolutional neural network, comprising:

[0178] A polarization feature input module is used to select the polarization coherence matrix as the polarization feature of each pixel in the polarization SAR image in step 1, and after processing the polarization coherence matrix, associate the position information of each pixel with the polarization feature to obtain the processed polarization feature associated with the position information;

[0179] The segmentation module of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network is used to implement the construction of a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through a fully convolutional neural network with an encoder-decoder structure with jump connections in step 2, use the polarization features processed in step 1 and associated with position information as input of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network, adopt a superpixel segmentation strategy on a regular network, and output a superpixel segmentation result;

[0180] The superpixel segmentation model training module of polarimetric SAR images based on a fully convolutional neural network is used to implement the superpixel segmentation task for polarimetric SAR images in step 3, design a loss function, and train the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network constructed in step 2;

[0181] The polarimetric SAR image classification model training module is used to implement step 4, inputting the processed polarimetric features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3, obtaining superpixel segmentation results, and processing the superpixel segmentation results as input of the polarimetric SAR image classification model to train the polarimetric SAR image classification model;

[0182] The polarimetric SAR image classification module is used to implement the classification of polarimetric SAR images in step 5 using the polarimetric SAR image classification model trained in step 4, and obtain polarimetric SAR image classification results combined with superpixel segmentation.

[0183] The present invention also provides an end-to-end polarimetric SAR image superpixel segmentation device based on a fully convolutional neural network, comprising:

[0184] Memory: a computer program storing the above-mentioned end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network, which is a computer-readable device;

[0185] Processor: used to implement the end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network when executing the computer program.

[0186] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the end-to-end polarization SAR image superpixel segmentation method based on a fully convolutional neural network.

Claims

1. An end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network, characterized in that: The following steps are involved: Step 1: Select a polarization coherence matrix as the polarization feature of each pixel in the polarimetric SAR image, process the polarization coherence matrix, extract the elements of the upper triangular part of the matrix, decompose the real part and the imaginary part, associate the position information of each pixel with the polarization feature, and obtain the processed polarization feature associated with the position information; Step 2: construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through an encoder-decoder structure with skip connections, use the polarization features processed in step 1 and associated with position information as the input of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network, adopt a superpixel segmentation strategy on a regular network, and output a superpixel segmentation result; Step 3, for the super-pixel segmentation task of polarimetric SAR images, a loss function is designed to train the super-pixel segmentation model of polarimetric SAR images based on a fully convolutional neural network constructed in step 2; Step 4, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3 to obtain superpixel segmentation results, and processing the superpixel segmentation results as input to the polarimetric SAR image classification model to train the polarimetric SAR image classification model; Step 5: Classify the polarimetric SAR image using the polarimetric SAR image classification model trained in step 4 to obtain a polarimetric SAR image classification result combined with superpixel segmentation.

2. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The polarization characteristics of the position information after processing in step 1 are as follows: t p =[T 11 ,T 22 ,T 33 ,Re[T 12 ],Im[T 12 ],Re[T 13 ],Im[T 13 ],Re[T 23 ],Im[T 23 ],row,col] Where, T ij Represents the elements in the polarization coherence matrix, i, j = 1, 2, 3, Re[.], Im[.] represent the real part and imaginary part respectively, row, col represent the position coordinates of the pixel.

3. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The structure of the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network in step 2 includes an encoder, a decoder and a prediction layer; The encoder consists of five layers of convolutional neural networks, with the kernel size set to 3×3 and the number of channels set to 16, 32, 64, 128 and 256 respectively. During the convolution process, the input of the first layer of the convolutional neural network of the encoder is padded according to the size of the feature map; The decoder consists of four deconvolution layers, with a kernel size of 4×4 and the number of channels set to 16, 32, 64, and 128, respectively. The feature maps are upsampled step by step, and the output features of each deconvolution layer in the decoder are associated with the output features of the convolution layer with the same number of channels in the encoder to obtain the final superpixel prediction. The activation function LeakyReLU is used for all convolution layers in the encoder and all deconvolution layers in the decoder. The prediction layer is a convolutional layer, the kernel size is set to 3×3, the number of channels is set to 9, the output result of the decoder is input into the prediction layer, and the output result of the prediction layer generates a superpixel association matrix through softmax.

4. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The specific process of the superpixel segmentation strategy on the rule network in step 2 includes: Step 2.11, segment the input image using a regular grid with a configurable size, and consider each grid as an initial superpixel; Step 2.12, assign each pixel to one of the superpixels, and record this mapping relationship in the pixel-superpixel association matrix H, calculated as follows: In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, D is the distance metric, when a pixel belongs to a superpixel, the corresponding value in H is 1, otherwise it is 0; only the mapping relationship between the pixel and its surrounding nine superpixels is considered; Step 2.13, the pixel-superpixel correlation matrix H of step 2.12 is processed by soft correlation mapping, so that the final superpixel soft correlation matrix Q is differentiable, and the calculation formula is as follows: Q=exp(-D(I P ,S i )) In the formula, I P is the feature corresponding to pixel p, S i is the center of superpixel i, m is the number of superpixels, and D is the distance metric.

5. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The specific method of step 2 includes: Step 2.21, using the polarization features processed in step 1 and associated with the position information as the input of the superpixel segmentation model of the polarization SAR image based on the fully convolutional neural network, extracting the polarization features of the polarization SAR image layer by layer through the encoder to obtain a feature map; Step 2.22, upsampling the feature map obtained in step 2.21 through the decoder, and associating it with the output features of the convolutional layer in the encoder with the same number of channels to obtain the output result of the encoder; In step 2.23, the output result of the encoder in step 2.22 is input to the prediction layer, and the output result of the prediction layer is used to generate a superpixel association matrix through softmax to obtain the final superpixel prediction.

6. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The loss function designed in step 3 is: Assume f(p) is the feature corresponding to a given pixel p, p = [x, y] T Used to represent the position of a given pixel, based on the superpixel soft correlation matrix Q obtained in step 2.13, calculate the center c of any superpixel s s =(u s ,l s ), where u s is the feature vector, l s is the position vector; the calculation method is as follows: Among them, N p is the set of superpixels around p, q s (p) is the predicted probability of association between p and the superpixel; the reconstructed feature f'(p) and position p' of any pixel are calculated as follows: In the formula, u s is the feature vector of the pixel, l s is the position vector of the pixel; finally, the loss function consists of two terms. The first term encourages the trained superpixel segmentation model of polarimetric SAR images based on fully convolutional neural networks to group pixels with similar features; the second term enforces the spatial compactness of superpixels, selects the one-hot encoding vector of the ground truth label, and uses cross entropy as the distance metric. The loss function is calculated as follows: where lm is a configurable weight, ΔS is the superpixel sampling interval and is related to the size of a regular grid of configurable size, and E(·,·) is the cross entropy loss function.

7. The end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network according to claim 1, characterized in that: The specific method of step 4 includes: The polarimetric SAR image classification model adopts a three-layer convolutional neural network, whose convolution kernel sizes are 3×3, 2×2 and 2×2, and the number of output channels are 16, 32, and 64 respectively; Step 4.1, inputting the processed polarization features with associated position information obtained in step 1 into the superpixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network trained in step 3, obtaining a superpixel segmentation result with irregular superpixel blocks, and calculating the average value of the features of each irregular superpixel block; Step 4.2, expand the superpixel block into a rectangle, and fill the blank part with the average value of the features obtained in step 4.1; Step 4.3, adjust the filled rectangle in step 4.2 to a set size and use it as the input of the polarimetric SAR image classification model; In step 4.4, the polarimetric SAR image classification model is trained using the Adam optimizer and cross entropy loss at the set learning rate and number of iterations to obtain a trained polarimetric SAR image classification model.

8. An end-to-end polarimetric SAR image superpixel segmentation system based on a fully convolutional neural network based on the method according to any one of claims 1 to 7, characterized in that: include: The polarization feature input module is used to select the polarization coherence matrix as the polarization feature of each pixel in the polarization SAR image, and after processing the polarization coherence matrix, associate the position information of each pixel with the polarization feature to obtain the processed polarization feature associated with the position information; The segmentation module of the superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network is used to construct a superpixel segmentation model of polarimetric SAR images based on a fully convolutional neural network through a fully convolutional neural network with an encoder-decoder structure with skip connections, use the processed polarization features with associated position information as the input of the superpixel segmentation model of polarimetric SAR images based on the fully convolutional neural network, adopt the superpixel segmentation strategy on the regular network, and output the superpixel segmentation result; The super-pixel segmentation model training module of polarimetric SAR images based on a fully convolutional neural network is used to design a loss function for the super-pixel segmentation task of polarimetric SAR images and train the super-pixel segmentation model of polarimetric SAR images based on a fully convolutional neural network. The polarimetric SAR image classification model training module is used to input the processed polarimetric features associated with the position information into the trained super-pixel segmentation model of the polarimetric SAR image based on the fully convolutional neural network to obtain the super-pixel segmentation results, and use the processed super-pixel segmentation results as the input of the polarimetric SAR image classification model to train the polarimetric SAR image classification model. The polarimetric SAR image classification module is used to classify polarimetric SAR images using the trained polarimetric SAR image classification model to obtain polarimetric SAR image classification results combined with superpixel segmentation.

9. An end-to-end polarimetric SAR image superpixel segmentation device based on a fully convolutional neural network, characterized in that: include: Memory: a computer program storing the end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network as described in any one of claims 1 to 7, which is a computer-readable device; Processor: used to implement the end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the end-to-end polarimetric SAR image superpixel segmentation method based on a fully convolutional neural network as described in any one of claims 1 to 7.

Images