A self-supervised semantic segmentation method for remote sensing images based on position encoding

Abstract

The present invention discloses a method for self-supervised semantic segmentation of remote sensing images based on position coding, comprising the following steps: using unlabeled data to perform self-supervised pre-training on an encoder to obtain a pre-trained encoder; using labeled data to fine-tune the pre-trained encoder to a downstream semantic segmentation task, and constructing a remote sensing image semantic segmentation model with a corresponding decoder; inputting the remote sensing image to be semantically segmented into the remote sensing image semantic segmentation model to obtain a segmentation result; the method of the present invention can utilize a large amount of unlabeled remote sensing data to learn rich semantic features of land objects, thereby reducing the dependence of remote sensing image semantic segmentation on labeled data and improving the efficiency of intelligent interpretation of remote sensing images.

Description

Technical Field

[0001] The invention discloses a remote sensing image self-supervised image semantic segmentation method based on position coding, belonging to the technical field of remote sensing image intelligent interpretation. Background Art

[0002] In recent years, semantic segmentation methods represented by deep learning have made rapid progress in the field of intelligent interpretation of remote sensing images due to their powerful feature learning and multi-level expression capabilities. However, semantic segmentation methods based on deep learning rely on massive pixel-by-pixel annotated data, and manual annotation requires huge manpower and material resources, which seriously restricts the application of remote sensing image semantic segmentation methods in large areas and complex scenes. How to reduce the dependence on annotated data is currently a key issue in the semantic segmentation of remote sensing images.

[0003] The main idea of ​​self-supervised learning is to use artificially designed self-supervised learning signals to mine the supervisory information in the data from massive unlabeled data to form pseudo-labels, thereby replacing the manually labeled data to drive the model for feature learning. It can extract rich image semantic features without relying on any labeled samples, and then complete specific target tasks through feature migration or domain adaptation.

[0004] At present, semantic segmentation networks based on deep learning mainly adopt encoder-decoder structure. Fully Convolution Neural Network (FCN) first applied convolutional neural network to the field of semantic segmentation. Later, SegNet changed the encoder to an improved VGG-16 network, and used unpooling to restore the output feature map size to make it consistent with the input in the decoding stage to improve the FCN network. U-Net improves segmentation accuracy through a U-shaped network with a contraction network and an expansion network. The contraction network extracts features and downsamples, and the expansion network upsamples to restore the feature map size. The two are connected by jump connections. The DeepLab series of models use dilated convolution to increase the receptive field of the convolution kernel, while using the ASPP structure to extract multi-scale information and improve the training speed through deep separable convolution.

[0005] Current self-supervised learning methods are mainly based on instance discrimination tasks, which potentially require that the targets obtained by performing multiple random croppings on the same image all correspond to the same object. However, remote sensing images usually have the characteristics of multi-target and multi-scale, resulting in different cropping and image transformations that may correspond to different objects. Therefore, current self-supervised learning methods are not suitable for direct application to remote sensing image semantic segmentation tasks. Summary of the invention

[0006] Purpose of the invention: In order to reduce the dependence of remote sensing image semantic segmentation tasks on manually annotated data and to solve the problem that existing self-supervised learning methods are not suitable for direct application to remote sensing image semantic segmentation tasks, the present invention proposes a remote sensing image self-supervised semantic segmentation method based on position coding.

[0007] Technical solution: A remote sensing image self-supervised semantic segmentation method based on position coding, comprising the following steps:

[0008] Step 1: Use unlabeled data to perform self-supervised pre-training on the encoder to obtain a pre-trained encoder;

[0009] Step 2: Use the labeled data to fine-tune the pre-trained encoder to the downstream semantic segmentation task, and build a remote sensing image semantic segmentation model with the corresponding decoder;

[0010] Step 3: Input the remote sensing image to be semantically segmented into the remote sensing image semantic segmentation model to obtain the segmentation result;

[0011] Wherein, step 1 includes the following sub-steps:

[0012] The remote sensing image x is transformed into a 7×7 matrix, the element values ​​in the 7×7 matrix are 0-48 respectively, and is upsampled to 224×224 size by the nearest neighbor interpolation method and the position code i is obtained;

[0013] Apply two data transformations to the remote sensing image x and the position code i in succession to obtain two different views x1, x2 of the same remote sensing image and the corresponding position codes i1, i2;

[0014] Input the view x1 and view x2 to the encoder and momentum encoder respectively to obtain the feature map f1 and feature map f2;

[0015] The feature maps f1 and f2 are globally pooled and projected through MLP to obtain the feature vector p q and the eigenvector z k , calculate the eigenvector p q and the eigenvector z k The cosine similarity L I ;

[0016] According to the position codes i1, i2, pixels with the same position codes are selected in the feature maps f1 and f2, and the similarity score L between the two pixels is calculated. P ;

[0017] By minimizing the cost function L = L p +L I Learn the encoder parameters.

[0018] Furthermore, according to the position codes i1, i2, pixels with the same position code are selected in the feature map f1 and the feature map f2, and the similarity score L between the two pixels is calculated. P , specifically including the following sub-steps:

[0019] Input the feature map f1 and feature map f2 into the corresponding ConvMLP respectively to obtain the feature map p * and z * ; For feature map p * and z * Perform l2 regularization and pass p * and z * Multiply them together to get the similarity score between the pixels of the two feature maps;

[0020] The position codes i1 and i2 corresponding to the views x1 and x2 are downsampled by 32 times using the nearest neighbor interpolation method to obtain position codes of the same size as the feature map; then a binary mask map is generated by comparing whether the values ​​in the two position codes are the same;

[0021] The similarity score is Boolean selected through the binary mask image to obtain the similarity score L between the pixels corresponding to the two views. P .

[0022] Furthermore, the ConvMLP is a multilayer perceptron in which the linear layer in the MLP is replaced by a convolutional layer with a convolution kernel size of 1; and the MLP is a multilayer perceptron with one hidden layer.

[0023] Furthermore, the encoder adopts a ResNet-50 network, and the network structure of the momentum encoder is the same as that of the encoder, but the momentum encoder performs parameter update through sliding exponential averaging.

[0024] Furthermore, the data transformation includes any one of random cropping, scaling, horizontal flipping, rotation, random noise, and random color distortion.

[0025] Furthermore, the remote sensing image semantic segmentation model adopts the DeepLabV3+ segmentation model.

[0026] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0027] (1) Aiming at the multi-target characteristics of remote sensing images, the present invention learns global common semantic features by maximizing the cosine similarity between different views of the same image, and uses position encoding to track the relative position changes between objects, thereby learning more refined semantic features and allowing the model to maintain the ability to distinguish different objects;

[0028] (2) The method of the present invention uses a large amount of unlabeled remote sensing data to learn rich semantic features of ground objects, reduce the dependence of remote sensing image semantic segmentation on labeled data, improve the efficiency of intelligent interpretation of remote sensing images, and enhance the generalization ability of the model;

[0029] (3) After self-supervised pre-training, the encoder of the present invention can achieve high-precision semantic segmentation by fine-tuning with a small amount of labeled samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flowchart of the self-supervised pre-training method;

[0031] Figure 2 Methods for generating and selecting positional codes;

[0032] Figure 3 This is an example of prediction results. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0034] The main steps of the remote sensing image self-supervised semantic segmentation method based on position coding of the present invention are described as follows.

[0035] Step 1: Use a large amount of unlabeled data to perform self-supervised pre-training on the encoder in the semantic segmentation network. The self-supervised pre-training proposed in the present invention can learn rich image semantic features without relying on any labeled data; the data used in this step can be multi-source, multi-phase, and multi-region remote sensing data, so that the trained encoder can be applied to the semantic segmentation of remote sensing images in different scenarios, and has strong generalization ability.

[0036] Now combined Figure 1 , the self-supervised pre-training process of the present invention is described as follows.

[0037] S100: The remote sensing image x is resized to 224×224 by bicubic convolution interpolation, and a 7×7 matrix is ​​created for it. The element values ​​in the 7×7 matrix are 0-48 respectively. The image is upsampled to 224×224 by the nearest neighbor interpolation method and the position code i is obtained. Figure 2 (a)

[0038] S200: Apply data transformation twice continuously to the remote sensing image x and the position code i in S100, thereby obtaining two different views (x1, x2) of the same image and their corresponding position codes (i1, i2); the data transformation in this step includes but is not limited to random cropping, scaling, flipping, rotation, random noise, and random color distortion. The specific method and parameters of the data transformation can be found in Table 1.

[0039] Table 1

[0040]

[0041]

[0042] S300: Input view x1 and view x2 to the encoder and momentum encoder respectively to obtain feature map f1 and feature map f2. In some embodiments, the encoder may adopt a ResNet-50 network, and the network structure of the momentum encoder is the same as that of the encoder, except that the momentum encoder is not updated by gradient backpropagation, but by sliding exponential averaging. By introducing a momentum encoder, the asymmetry of the overall network can be increased, thereby accelerating the convergence speed of the network. After self-supervised pre-training, only the encoder is retained to participate in the downstream segmentation task.

[0043] S400: Figure 1 As shown in Figure 2, a global branch of self-supervised learning is constructed, and the feature maps f1 and f2 are globally pooled and projected through a multi-layer perceptron (MLP) to obtain the feature vector p q and the eigenvector z k , calculate the cosine similarity L between the two I ; Among them, the multi-layer perceptron (MLP) is a multi-layer perceptron with one hidden layer, L I The calculation formula is as follows:

[0044]

[0045] like Figure 1 As shown in the figure, a local branch of self-supervised learning is constructed, and the feature map f1 and the feature map f2 are respectively input into the corresponding multi-layer perceptron (ConvMLP) composed of 1×1 convolutional layers to obtain the feature map p * and z * ; For feature map p * and z * Perform l2 regularization and pass p * and z * Multiply (mul) to get the similarity score S between each pixel of the two feature maps; then downsample the position codes i1 and i2 corresponding to view x1 and view x2 by 32 times through the nearest neighbor interpolation method to get the position code of the same size as the feature map; then generate the binary mask map M by comparing whether the values ​​in the two position codes are the same (eq); finally, perform Boolean selection on the similarity score through the binary mask map to get the similarity score L between the pixels corresponding to the two views P :

[0046] L P=2-2·S.mask_select(M) (2)

[0047] Since the spatial dimension of the feature map needs to be retained in the local branch, the linear layer in the multi-layer perceptron (MLP) is replaced by a convolutional layer with a convolution kernel size of 1, and finally a multi-layer perceptron (ConvMLP) composed of 1×1 convolutional layers is obtained. Therefore, ConvMLP is used in the local branch of self-supervised learning.

[0048] By minimizing the cost function L = L p +L I Conduct encoder parameter learning. In this step, the self-supervised learning method is used to extract the intrinsic information of the data from the massive unlabeled remote sensing data, and learn the global semantic features with time and space invariance; at the same time, the multi-target characteristics of the remote sensing image are fully considered, and the local branch is constructed in S400 and the position encoding is used to learn the refined semantic features while maintaining the encoder's ability to distinguish different targets.

[0049] Step 2: After self-supervised pre-training, the present invention only retains the encoder to participate in the downstream segmentation task. Therefore, when the pre-trained encoder is fine-tuned to the downstream semantic segmentation task using a small amount of labeled data, different decoders can be used to flexibly adjust the semantic segmentation network framework; in some embodiments, DeepLabV3+ is selected as the segmentation model for semantic segmentation of remote sensing images, and the encoder after self-supervised pre-training is used as the encoder of DeepLabV3+, and fine-tuned with the help of a small amount of target area labeled data, and finally a semantic segmentation network that meets the requirements is obtained.

[0050] Step 3: Use a semantic segmentation network that meets the requirements to perform semantic segmentation on the input remote sensing image to obtain the resolution results of different targets.

[0051] Example:

[0052] This embodiment uses the Potsdam dataset of ISPRS to evaluate the effectiveness of the present invention. The Potsdam dataset contains 38 images of 6000×6000 in size, and each image contains three bands: R, G, and B.

[0053] In this evaluation, 38 images were cropped into non-overlapping images of 256×256 size, resulting in a total of 25,000+ images. During the pre-training process, all 25,000+ images were used for model pre-training. After pre-training, 13,000+ images were used for fine-tuning.

[0054] The experimental settings are as follows:

[0055] During pre-training, the momentum SGD optimizer is used, and the weight decay is 1.5×10 -6, the learning rate is set to 1×10 -2 , the momentum parameter is set to 0.9, and a total of 400 epochs are trained.

[0056] During fine-tuning, DeepLabV3+ semantic segmentation network was used, and the pre-trained model was used as its encoder. The momentum SGD optimizer was used in training, and the weight decay was 1×10 -4 , the encoder learning rate is 0.001, the learning rate of the rest of the model is 0.01, and a total of 100 epochs are trained.

[0057] After fine-tuning training, the results are evaluated on the test set (7000+ images). The prediction results of the model on the test set are as follows: Figure 3 shown.

Claims

1. A method for self-supervised semantic segmentation of remote sensing images based on position coding, characterized in that: The following steps are involved: Step 1: Use unlabeled data to perform self-supervised pre-training on the encoder to obtain a pre-trained encoder; Step 2: Use the labeled data to fine-tune the pre-trained encoder to the downstream semantic segmentation task, and build a remote sensing image semantic segmentation model with the corresponding decoder; Step 3: Input the remote sensing image to be semantically segmented into the remote sensing image semantic segmentation model to obtain the segmentation result; Wherein, step 1 includes the following sub-steps: The remote sensing image x is resized to 224×224 by bicubic convolution interpolation, and a 7×7 matrix is ​​created for it. The element values ​​in the 7×7 matrix are 0-48 respectively. The image is upsampled to 224×224 by the nearest neighbor interpolation method and the position code i is obtained. Apply two data transformations to the remote sensing image x and the position code i in succession to obtain two different views x1, x2 of the same remote sensing image and the corresponding position codes i1, i2; Input the view x1 and view x2 to the encoder and momentum encoder respectively to obtain the feature map f1 and feature map f2; The feature maps f1 and f2 are globally pooled and projected through MLP to obtain the feature vector p q and the eigenvector z k , calculate the eigenvector p q and the eigenvector z k The cosine similarity L I ; According to the position codes i1, i2, pixels with the same position codes are selected in the feature maps f1 and f2, and the similarity score L between the two pixels is calculated. P ; By minimizing the cost function L = L p +L I Learn encoder parameters; According to the position codes i1, i2, pixels with the same position code are selected in the feature map f1 and the feature map f2, and the similarity score L between the two pixels is calculated. P , specifically including the following sub-steps: Input the feature map f1 and feature map f2 into the corresponding ConvMLP respectively to obtain the feature map p * and z * ; For feature map p * and z * Perform l2 regularization and pass p * and z * Multiply them together to get the similarity score between the pixels of the two feature maps; The position codes i1 and i2 corresponding to views x1 and x2 are downsampled by 32 times through the nearest neighbor interpolation method to obtain position codes of the same size as the feature map; Then, a binary mask image is generated by comparing the values ​​in the two position codes to see if they are the same; The similarity score is Boolean selected through the binary mask image to obtain the similarity score L between the pixels corresponding to the two views. P .

2. The method for self-supervised semantic segmentation of remote sensing images based on position coding according to claim 1, characterized in that: The ConvMLP is a multilayer perceptron in which the linear layer in the MLP is replaced by a convolutional layer with a convolution kernel size of 1; the MLP is a multilayer perceptron with one hidden layer.

3. The method for self-supervised semantic segmentation of remote sensing images based on position coding according to claim 1, characterized in that: The encoder adopts a ResNet-50 network, and the network structure of the momentum encoder is the same as that of the encoder, but the momentum encoder performs parameter update through sliding exponential averaging.

4. The method for self-supervised semantic segmentation of remote sensing images based on position coding according to claim 1, characterized in that: The data transformation includes any one of random cropping, scaling, horizontal flipping, rotation, random noise, and random color distortion.

5. The method for self-supervised semantic segmentation of remote sensing images based on position coding according to claim 1, characterized in that: The remote sensing image semantic segmentation model adopts the DeepLabV3+ segmentation model.

Images