A semantic segmentation method for remote sensing images based on supervised long-range correlation

Abstract

The present invention relates to a method for semantic segmentation of remote sensing images based on supervised long-range correlation. A sample library is constructed using remote sensing images and manually annotated pixel-level semantic category truth maps, and a remote sensing image semantic segmentation network based on supervised long-range correlation is trained to learn the characteristics of each category of ground object elements on the remote sensing image. Using the trained network model to perform semantic segmentation on new remote sensing images can achieve high-precision pixel-level classification of remote sensing images. In the method of the present invention, the long-range correlation learning process explicitly uses the semantic category truth map to generate category consistency constraints and supervise the calculated correlation scores. This allows the network model to have better consistency in the features extracted from pixels belonging to the same type of ground objects, and more obvious differences in the features extracted from pixels belonging to different categories of ground objects, thereby effectively improving the accuracy of semantic segmentation of remote sensing images.

Description

Technical Field

[0001] The present invention relates to a supervised long-range correlation remote sensing image semantic segmentation method, which realizes pixel-level image classification of optical remote sensing images and can be used for geographic information mapping, urban planning and management, environmental monitoring, and urban change monitoring. Background Art

[0002] Semantic segmentation is a fundamental task in computer vision and remote sensing, playing a crucial role in urban environmental monitoring, land use planning, crop yield estimation, and urban change detection. Currently, the mainstream network used for semantic segmentation is based on an encoder-decoder style fully convolutional network. However, unlike natural images, remote sensing images typically cover a larger scene area. This results in large intra-class variance among objects of the same type, while small inter-class variance among objects of different types. This phenomenon becomes more pronounced as the distance between pixels / regions in the image increases. Previous studies have explored and utilized attention mechanisms and Transformer models to address the challenge of large intra-class variance and small inter-class variance among objects in large-scale remote sensing imagery. These studies enhance the pixel-level segmentation accuracy of convolutional neural network segmentation models by leveraging information exchange between features across different image regions. However, these methods rely solely on matrix operations to correlate features across pixels / regions. The unsupervised learning of correlations during training may limit further improvements in semantic segmentation accuracy.

[0003] On the other hand, in order to reliably determine the classification of objects in remote sensing images, the comprehensive use of multi-scale information for pixel classification has also received widespread attention. However, previous studies have mostly extracted multi-scale information at the feature level, and fewer studies have captured multi-scale information at the level of the original input image. Some studies use images of two different scales as input, but the differences in the sizes of different objects in each scale are not taken into account during the information extraction process. Large-scale objects should use convolutional layers with larger receptive fields to capture their features, while small-scale objects should use convolutional layers with smaller receptive fields for feature extraction. Researching and implementing adaptive and effective feature capture of objects of different sizes from multi-scale input images is also very meaningful for improving the accuracy of pixel-level segmentation of remote sensing images. Summary of the Invention

[0004] This paper addresses the shortcomings of existing remote sensing image semantic segmentation methods, which rely solely on unsupervised learning of long-range correlations and inflexible multi-scale feature extraction designs. By proposing a supervised long-range correlation-based semantic segmentation method for remote sensing images, this method reliably and accurately performs pixel-level semantic segmentation on remote sensing images, accurately predicting the correct category for easily confused image pixels.

[0005] The technical solution used to achieve the objectives of the present invention is as follows: Step 1: Constructing a supervised long-range correlation semantic segmentation network (SLCNet). This network is used to learn the characteristics of different types of ground features in remote sensing images and establish a mapping relationship between the learned characteristics and pixel-level category maps. Step 2: Using optical remote sensing image data and pixel-level semantic category ground truth map data to build a sample library, the sample library and a deep learning framework are used to train the network model. By loading the trained network model weights and predicting new remote sensing images, pixel-level category prediction of remote sensing images can be achieved.

[0006] Furthermore, the supervised long-range correlation semantic segmentation network described in step 1 includes five components: a features extraction backbone, a long-range correlation supervision module, an adaptive receptive field feature extraction module, a multi-scale side-output module, and a final output layer. The feature extraction backbone uses the currently popular Residual Network (ResNet), which uses the original image as input and extracts features from shallow to deep layers. During this process, the size of the feature map is gradually reduced (by 2x, 4x, 8x, 16x, and 32x downsampling relative to the original image). The feature maps extracted by the feature extraction backbone are downsampled by 2x, 4x, 8x, and 16x, respectively, and are fed into the long-range correlation supervision module for long-range information interaction. The original images, downsampled by 2x, 4x, 8x, and 16x, are fed into the adaptive receptive field feature extraction module to extract features for objects of different sizes. The 32x downsampled features are processed through two stacked 3×3 convolution layers, batch normalization layers, and rectified linear units (ReLUs), followed by top-down upsampling and feature fusion. At each scale, the features after long-range information interaction and the features that are adaptive to object size are concatenated and then fused with the top-down features after upsampling. The fused 2x, 4x, 8x, and 16x features are fed into the side output module to predict the corresponding scale map (i.e., pixel-level semantic category probability map). The 2x features are further upsampled and processed by the final output layer to output the final pixel-level semantic category probability map consistent with the input image resolution. The index of the maximum probability value of the pixel-level semantic category probability map along the channel direction is calculated pixel by pixel. The category to which each pixel in the image belongs can be obtained based on the correspondence between the index and the pre-defined category.

[0007] Furthermore, the long-range correlation supervision module is implemented using a self-attention model based on supervised learning. It decomposes the correlation calculation of any two position features on the feature map into correlation calculations in the vertical direction (i.e., correlation calculations are performed on each column separately) and the horizontal direction (i.e., correlation calculations are performed on each row separately); that is, the long-range correlation supervision module first performs a supervised self-attention model on each column of the feature map, and then performs a supervised self-attention model on each row of the feature map; by decomposing the 2D correlation into 2 1D correlations, long-range correlation can still be performed while reducing the memory usage of the operation process. In each supervised self-attention model, given the input feature map F in ∈R H×W×C (H, W, C represent the length, width and number of channels of the feature map), three linear projection layers map the input features into three different features, namely K∈R H×W×C , Q∈R H×W×C and V∈R H×W×C ; Then for the feature Q∈R H×W×C and K∈R H×W×C Perform dot multiplication on columns / rows, and then use the softmax function to normalize the output value to between 0 and 1 to obtain the correlation score; use the correlation score as the weight to perform feature V∈R H×W×C The corresponding column / row elements are weighted to obtain the output feature F of the self-attention model out ∈R H×W×C In order to speed up the convergence of the model, after the self-attention model is executed, the input feature F of the self-attention model in ∈R H×W×C and output features F out ∈R H×W×CThey are added pixel by pixel and then input into a convolutional layer for internal feature fusion; this convolutional layer consists of a 3×3 convolution operation (Convolution), a batch normalization layer (BatchNormalization layer), and a rectified linear unit (ReLU). Since each pixel in the semantic segmentation dataset has clear category information, it can be used to explicitly optimize the correlation score. If two pixels belong to the same category, the expected value of the correlation score between them is 1, otherwise it is 0. Based on the manually annotated semantic segmentation ground truth map, a category consistency map between different pixels is generated. The learning target value of the correlation score between features of pixels belonging to the same category is set to 1, and the learning target value of the correlation score between features of pixels of different categories is set to 0. The correlation score is then supervised using the L1 loss function. The supervised self-attention module can help the network extract features with higher consistency for pixels belonging to the same category, while increasing the difference in features extracted from pixels belonging to different categories.

[0008] Furthermore, the adaptive receptive field feature extraction module takes the downsampled original image as input and outputs a feature map with the same resolution as the input image. In the adaptive receptive field feature extraction module, the input image I is processed by a convolution layer and three standard residual modules. in Mapping is performed to obtain the scale-insensitive feature map F insen Among them, the first convolution layer consists of a 3×3 convolution operation (Convolution), a batch normalization layer (Batch Normalization layer) and a rectified linear unit (Rectified Linear Unit, ReLU); the standard residual module consists of two sets of sequentially stacked 3×3 convolution operations (Convolution), batch normalization layers (Batch Normalization layer) and rectified linear units (Rectified Linear Unit, ReLU); in the standard residual module, the input features are added pixel by pixel with the features output by the second batch normalization layer, and then input into the second rectified linear unit for processing to obtain the output of the standard residual module. After obtaining the scale-insensitive feature map F insen Finally, the adaptive receptive field feature extraction module is divided into three branches here. The middle branch uses a global average pooling layer, a 1×1 convolution, a batch normalization layer and a Sigmoid activation function to generate the adaptive receptive field weight factor S∈R 1×1×CThe top branch uses a 3×3 convolution with a dilation rate of 1, a batch normalization layer, and a rectified linear unit (ReLU) to generate a small receptive field feature map F. small The bottom branch uses a 3×3 convolution with a dilation rate of 3, a batch normalization layer, and a rectified linear unit (ReLU) to generate a large receptive field feature map F. large ; Adaptive receptive field weight factor S∈R 1×1×C Responsible for weighting the small receptive field feature map F small , weight factor 1-S∈R 1×1×C It is responsible for weighting the large receptive field feature map F large . Weight factor S∈R 1×1×C Channel by channel small receptive field feature map F small Weighted, weight factor 1-S∈R 1×1×C Channel by channel large receptive field feature map F large The weighted features are obtained by adding them pixel by pixel, and then input into a 3×3 convolution layer + a batch normalization layer + a rectified linear unit (ReLU) to generate a feature F that is adaptive to the object scale. out Since different channels of the feature map can well characterize and distinguish objects belonging to different categories, and objects belonging to the same category have similar scales, using channel-by-channel weight factors can well help the network model generate features that are adaptive to the object scale.

[0009] Furthermore, the side output module includes a 3×3 convolution, a batch normalization layer, a rectified linear unit (ReLU) and a 3×3 convolution; the side output module is responsible for inputting the feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the side output branch is supervised using the cross-entropy loss function.

[0010] Furthermore, the final output layer contains a 1×1 convolution, which is responsible for transforming the input feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the final output layer is supervised by the Lovász softmax loss function.

[0011] Furthermore, at each scale, the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated and then combined with the top-down feature F top The calculation process of fusion after upsampling is as follows: the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated to generate concatenated features F cat ; Top-down feature F top Use a 2×2 transposed convolution layer to upsample by 2 times to obtain the concatenated feature F cat Features F with the same length and width top-resample ; Upsampled features F top-resample Then concatenate the feature F cat Perform concatenation to generate a new concatenation feature F cat-new ; Then, two sets of sequentially stacked 3×3 convolution + a batch normalization layer + a rectified linear unit (ReLU) are used to further fuse the internal information and output the fused features.

[0012] Furthermore, the 2x features are further upsampled and then processed by the final output layer. The upsampling operation in the final pixel-level semantic category probability map whose output is consistent with the input image resolution consists of a 2×2 transposed convolution layer and two sets of sequentially stacked 3×3 convolutions + batch normalization layers + rectified linear units (ReLU).

[0013] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0014] In step 2.1, each pixel in the remote sensing image is labeled with its category, generating a semantic category ground truth grid map. The original large-scale remote sensing image and the corresponding semantic category ground truth grid map are cropped to a standard size supported by the network model, i.e., 512×512 pixels, to generate the training sample set.

[0015] Step 2.2: Train the supervised long-range correlation semantic segmentation network. Use the cropped image patches generated in step 2.1 and the corresponding cropped semantic category ground truth grid maps as training data, and iteratively train the supervised long-range correlation semantic segmentation network until the model converges.

[0016] In step 2.3, given a new remote sensing image to be predicted, the original large-scale remote sensing image is cropped with a certain step size and input into the network model with the trained weights loaded, obtaining pixel-level prediction results for each cropped image. The pixel-level prediction results of all cropped images are projected back onto the original remote sensing image to obtain the pixel-level category prediction results for the entire large-scale remote sensing image.

[0017] The present invention has the following advantages: 1) It does not require a complex process of manually designing features, and can complete pixel-level category prediction of remote sensing images through a simple, end-to-end trainable model. 2) It is highly reusable, and the model training can continue on additional data, and can be used for continuous online learning and sustainable iterative optimization. 3) It is scalable, and the trained neural network model can be adjusted and applied to other purposes, such as ground object change detection based on remote sensing images, and ground object target extraction of interest based on remote sensing images. 4) It is highly robust, and the network model is embedded with a supervised long-range correlation learning model and an adaptive receptive field feature extraction module, which can obtain good pixel-level prediction results for various ground object elements in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the structure of the supervised long-range correlation semantic segmentation network proposed in the present invention.

[0019] Figure 2 It is a schematic diagram of the structure of the long-range correlation supervision module proposed in the present invention.

[0020] Figure 3 It is a structural diagram of the adaptive receptive field feature extraction module proposed in the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be further described in detail below with reference to embodiments and in conjunction with the accompanying drawings.

[0022] This paper proposes a semantic segmentation algorithm based on convolutional neural networks suitable for pixel-level category prediction tasks of remote sensing images, which includes the following steps:

[0023] Step 1: A supervised long-range correlation semantic segmentation network (SLCNet) is constructed. This network learns the features of different types of ground features in remote sensing images and establishes a mapping relationship between the learned features and pixel-level category maps. Step 2: A sample library is constructed using optical remote sensing image data and pixel-level semantic category ground truth maps. This sample library and a deep learning framework are used to train the network model. By loading the trained network model weights and applying them to new remote sensing images, pixel-level category prediction of remote sensing images can be achieved.

[0024] Furthermore, the supervised long-range correlation semantic segmentation network described in step 1 includes five components: a features extraction backbone, a long-range correlation supervision module, an adaptive receptive field feature extraction module, a multi-scale side-output module, and a final output layer. The feature extraction backbone uses the currently popular Residual Network (ResNet), which uses the original image as input and extracts features from shallow to deep layers. During this process, the size of the feature map is gradually reduced (by 2x, 4x, 8x, 16x, and 32x downsampling relative to the original image). The feature maps extracted by the feature extraction backbone are downsampled by 2x, 4x, 8x, and 16x, respectively, and are fed into the long-range correlation supervision module for long-range information interaction. The original images, downsampled by 2x, 4x, 8x, and 16x, are fed into the adaptive receptive field feature extraction module to extract features for objects of different sizes. The 32x downsampled features are processed through two stacked 3×3 convolution layers, batch normalization layers, and rectified linear units (ReLUs), followed by top-down upsampling and feature fusion. At each scale, the features after long-range information interaction and the features that are adaptive to object size are concatenated and then fused with the top-down features after upsampling. The fused 2x, 4x, 8x, and 16x features are fed into the side output module to predict the corresponding scale map (i.e., pixel-level semantic category probability map). The 2x features are further upsampled and processed by the final output layer to output the final pixel-level semantic category probability map consistent with the input image resolution. The index of the maximum probability value of the pixel-level semantic category probability map along the channel direction is calculated pixel by pixel. The category to which each pixel in the image belongs can be obtained based on the correspondence between the index and the pre-defined category.

[0025] Furthermore, the long-range correlation supervision module is implemented using a self-attention model based on supervised learning. It decomposes the correlation calculation of any two position features on the feature map into correlation calculations in the vertical direction (i.e., correlation calculations are performed on each column separately) and the horizontal direction (i.e., correlation calculations are performed on each row separately); that is, the long-range correlation supervision module first performs a supervised self-attention model on each column of the feature map, and then performs a supervised self-attention model on each row of the feature map; by decomposing the 2D correlation into 2 1D correlations, long-range correlation can still be performed while reducing the memory usage of the operation process. In each supervised learning self-attention model, given the input feature map F in ∈R H×W×C (H, W, C represent the length, width and number of channels of the feature map), three linear projection layers map the input features into three different features, namely K∈R H×W×C , Q∈R H×W×C and V∈R H×W×C ; Then for the feature Q∈R H×W×C and K∈R H ×W×C Perform dot multiplication on columns / rows, and then use the softmax function to normalize the output value to between 0 and 1 to obtain the correlation score; use the correlation score as the weight to perform feature V∈R H×W×C The corresponding column / row elements are weighted to obtain the output feature F of the self-attention model out ∈R H×W×C In order to speed up the convergence of the model, after the self-attention model is executed, the input feature F of the self-attention model in ∈R H×W×C and output features F out ∈R H×W×CThey are added pixel by pixel and then input into a convolutional layer for internal feature fusion; this convolutional layer consists of a 3×3 convolution operation (Convolution), a batch normalization layer (BatchNormalization layer) and a rectified linear unit (ReLU). Since each pixel in the semantic segmentation dataset has clear category information, it can be used to explicitly optimize the correlation score. If two pixels belong to the same category, the expected value of the correlation score between them is 1, otherwise it is 0. Based on the manually annotated semantic segmentation ground truth map, a category consistency map between different pixels is generated. The learning target value of the correlation score between features of pixels belonging to the same category is set to 1, and the learning target value of the correlation score between features of pixels of different categories is set to 0. The correlation score is then supervised using the L1 loss function. The supervised self-attention module can help the network extract features with higher consistency for pixels belonging to the same category, while increasing the difference between features of pixels belonging to different categories.

[0026] Furthermore, the adaptive receptive field feature extraction module takes the downsampled original image as input and outputs a feature map with the same resolution as the input image. In the adaptive receptive field feature extraction module, the input image I is processed by a convolution layer and three standard residual modules. in Mapping is performed to obtain the scale-insensitive feature map F insen Among them, the first convolution layer consists of a 3×3 convolution operation (Convolution), a batch normalization layer (Batch Normalization layer) and a rectified linear unit (Rectified Linear Unit, ReLU); the standard residual module consists of two sets of sequentially stacked 3×3 convolution operations (Convolution), batch normalization layers (Batch Normalization layer) and rectified linear units (Rectified Linear Unit, ReLU); in the standard residual module, the input features are added pixel by pixel with the features output by the second batch normalization layer, and then input into the second rectified linear unit for processing to obtain the output of the standard residual module. After obtaining the scale-insensitive feature map F insen Finally, the adaptive receptive field feature extraction module is divided into three branches here. The middle branch uses a global average pooling layer, a 1×1 convolution, a batch normalization layer and a Sigmoid activation function to generate the adaptive receptive field weight factor S∈R 1×1×CThe top branch uses a 3×3 convolution with a dilation rate of 1, a batch normalization layer, and a rectified linear unit (ReLU) to generate a small receptive field feature map F. small The bottom branch uses a 3×3 convolution with a dilation rate of 3, a batch normalization layer, and a rectified linear unit (ReLU) to generate a large receptive field feature map F. large ; Adaptive receptive field weight factor S∈R 1×1×C Responsible for weighting the small receptive field feature map F small , weight factor 1-S∈R 1×1×C It is responsible for weighting the large receptive field feature map F large . Weight factor S∈R 1×1×C Channel by channel small receptive field feature map F small Weighted, weight factor 1-S∈R 1×1×C Channel by channel large receptive field feature map F large The weighted features are obtained by adding them pixel by pixel, and then input into a 3×3 convolution layer + a batch normalization layer + a rectified linear unit (ReLU) to generate a feature F that is adaptive to the object scale. out Since different channels of the feature map can well characterize and distinguish objects belonging to different categories, and objects belonging to the same category have similar scales, using channel-by-channel weight factors can well help the network model generate features that are adaptive to the object scale.

[0027] Furthermore, the side output module includes a 3×3 convolution, a batch normalization layer, a rectified linear unit (ReLU) and a 3×3 convolution; the side output module is responsible for converting the feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the side output branch is supervised using the cross-entropy loss function.

[0028] Furthermore, the final output layer contains a 1×1 convolution, which is responsible for transforming the feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the final output layer is supervised by the Lovász softmax loss function.

[0029] Furthermore, at each scale, the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated and then combined with the top-down feature F top The calculation process of fusion after upsampling is as follows: the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated to generate concatenated features F cat ; Top-down feature F top Use a 2×2 transposed convolution layer to upsample by 2 times to obtain the concatenated feature F cat Features F with the same length and width top-resample ; Upsampled features F top-resample Then concatenate the features F cat Perform concatenation to generate a new concatenation feature F cat-new ; Then, two sets of sequentially stacked 3×3 convolution + a batch normalization layer + a rectified linear unit (ReLU) are used to further fuse the internal information and output the fused features.

[0030] Furthermore, the 2x features are further upsampled and then processed by the final output layer. The upsampling operation in the final pixel-level semantic category probability map whose output is consistent with the input image resolution consists of a 2×2 transposed convolution layer and two sets of sequentially stacked 3×3 convolutions + batch normalization layers + rectified linear units (ReLU).

[0031] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0032] In step 2.1, each pixel in the remote sensing image is labeled with its category, generating a semantic category ground truth grid map. The original large-scale remote sensing image and the corresponding semantic category ground truth grid map are cropped to a standard size supported by the network model, i.e., 512×512 pixels, to generate the training sample set.

[0033] Step 2.2: Train the supervised long-range correlation semantic segmentation network. Use the cropped image patches generated in step 2.1 and the corresponding cropped semantic category ground truth grid as training data, and iterate the supervised long-range correlation semantic segmentation network until the model converges.

[0034] In step 2.3, given a new remote sensing image to be predicted, the original large-scale remote sensing image is cropped with a certain step size and input into the network model with the trained weights loaded, obtaining pixel-level prediction results for each cropped image. The pixel-level prediction results of all cropped images are projected back onto the original remote sensing image to obtain the pixel-level category prediction results for the entire large-scale remote sensing image.

[0035] The effect of the method of the present invention is illustrated below by an example: First, a supervised long-range correlation semantic segmentation network (Supervised Long-range Correlation semantic segmentation Network, SLCNet) is constructed according to the method of the present invention. Then, training sample data is obtained, and the network model is trained using the sample data. The sample data used in the embodiment is ISPRS Vaihingen 2D semantic segmentation data, which contains 11 training images and 5 test images, including impervious surface (Imp.surf), building (Building), low vegetation (Low veg), tree (Tree) vehicle (car) and background class (Background, which accounts for a very small proportion and is ignored in the accuracy assessment), a total of 6 categories; the image size ranges from 1996×1995 pixels to 3816×2550 pixels; the pixel category of each image is generated by manual annotation. We cropped the original images and their corresponding annotated images from the training set into 512×512 pixel blocks with a step size of 128 pixels. These blocks were then fed into the network model for iterative training until the model converged and obtained the optimal weight file. After model training, the trained network model fed the test remote sensing image to be predicted for semantic segmentation. The predicted category for each pixel was then compared with the manually annotated ground truth.

[0036] To verify the effectiveness and advancement of our method, we compared it with other state-of-the-art semantic segmentation algorithms. These included U-Net, PSPNet, RefineNet, DeepLabv3+, DANet, Swin-UNet, Trans-UNet, and ST-UNet, which have demonstrated outstanding performance in various semantic segmentation tasks. All methods were trained using the same training data on the same hardware (a personal computer equipped with an NVIDIA TITAN RTX 24GB GPU, an Intel Core i9-9900K CPU, and a Windows operating system). The prediction results of all methods were quantitatively evaluated for each category using the pixel-level evaluation metrics Intersection over Union (IoU) and F1 score, respectively, and are reported in Table 1. The mean IoU (mIoU) and average F1 (Ave F1) scores across all categories were used as primary metrics for accuracy evaluation. As shown in Table 1, the mean IoU and average F1 scores demonstrate that our method outperforms several other state-of-the-art semantic segmentation methods. Our method outperforms existing methods by at least 1.5% in terms of mIoU and at least 1.1% in terms of AveF1. This demonstrates that our method is more robust and can produce more accurate pixel-level semantic category predictions. Therefore, our method possesses considerable engineering value.

[0037] Table 1 Comparison of the accuracy of the proposed method and other advanced semantic segmentation methods

[0038]

[0039] The specific implementations described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

Claims

1. A remote sensing image semantic segmentation method based on supervised long-range correlation, characterized in that: The steps include: Step 1: Build a supervised long-range correlation semantic segmentation network, which is used to learn the features of different types of objects in remote sensing images; The supervised long-range correlation semantic segmentation network includes a feature extraction backbone, a long-range correlation supervision module, an adaptive receptive field feature extraction module, a multi-scale side output module, and a final output layer; The long-range correlation supervision module is implemented using a self-attention model based on supervised learning. It decomposes the correlation calculation of any two position features on the feature map into correlation calculations performed in the vertical and horizontal directions respectively. That is, the long-range correlation supervision module first performs a supervised self-attention model on each column of the feature map, and then performs a supervised self-attention model on each row of the feature map. By decomposing the two-dimensional correlation into two one-dimensional correlations, long-range correlation can still be performed while reducing the memory usage of the calculation process. In the self-attention model based on supervised learning, based on the semantic category truth map, a category consistency map between different pixels is generated, and the learning target value of the correlation score between features of pixels belonging to the same category is set to 1, and the learning target value of the correlation score between features of pixels of different categories is set to 0. Then, the L1 loss function is used to supervise the correlation score. Step 2: Use optical remote sensing image data and manually annotated pixel-level semantic category truth maps to build a sample library, use the sample library and deep learning framework to train the network model, and predict new remote sensing images by loading the trained network model weights to achieve pixel-level category prediction of new remote sensing images.

2. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: The feature extraction backbone adopts the residual network ResNet, which takes the original image as input and performs feature extraction from shallow to deep. In this process, the size of the feature map decreases step by step, and is downsampled by 2 times, 4 times, 8 times, 16 times, and 32 times relative to the original image; the feature maps downsampled by 2 times, 4 times, 8 times, and 16 times by the feature extraction backbone are respectively input into the long-range correlation supervision module for long-range information interaction; the original images downsampled by 2 times, 4 times, 8 times, and 16 times are respectively input into the adaptive receptive field feature extraction module to extract features of objects of different sizes; the 32-fold downsampled features are subjected to two sets of sequentially stacked 3×3 convolution + batch normalization layers + rectified linear units. After the element is extracted, top-down upsampling and feature fusion are performed; at each scale, the features after long-range information interaction and the features that are adaptive to object size are concatenated, and then fused with the top-down features after upsampling; the fused 2x, 4x, 8x, and 16x features are input to the side output module to predict the prediction map of the corresponding scale; the 2x features are further upsampled and then processed by the final output layer to output the final pixel-level semantic category probability map that is consistent with the input image resolution; the index of the maximum probability value of the pixel-level semantic category probability map along the channel direction is calculated pixel by pixel, and the category to which each pixel in the image belongs can be obtained according to the correspondence between the index and the pre-defined category.

3. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: In each supervised self-attention model, given the input feature map F in ∈R H×W×C , where H, W, C represent the length, width and number of channels of the feature map, and the three linear projection layers map the input features into three different features, namely K∈R H×W×C , Q∈R H×W×C and V∈R H×W×C ; Then for the feature Q∈R H×W×C and K∈R H×W×C Perform dot multiplication by column / row, and then use the softmax function to normalize the output value to between 0 and 1 to obtain the correlation score; use the correlation score as the weight to perform feature V∈R H×W×C The corresponding column / row elements are weighted to obtain the output feature F of the self-attention model out ∈R H×W×C ; In order to speed up the convergence of the model, after the self-attention model is executed, the input feature F of the self-attention model in ∈R H×W×C and output features F out ∈R H ×W×C They are added pixel by pixel and then fed into a convolutional layer for internal feature fusion; This convolutional layer consists of a 3×3 convolution operation, a batch normalization layer, and a rectified linear unit.

4. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: The adaptive receptive field feature extraction module takes the downsampled original image as input and outputs a feature map with the same resolution as the input image; in the adaptive receptive field feature extraction module, the input image I is processed by a convolution layer and three standard residual modules. in Mapping is performed to obtain the scale-insensitive feature map F insen ; After obtaining the scale-insensitive feature map F insen Finally, the adaptive receptive field feature extraction module is further divided into three branches. The middle branch uses a global average pooling layer, a 1×1 convolution, a batch normalization layer and a Sigmoid activation function to generate the adaptive receptive field weight factor S∈R 1 ×1×C ; The top branch uses a 3×3 convolution with a dilation rate of 1, a batch normalization layer, and a rectified linear unit to generate a small receptive field feature map F. small The bottom branch uses a 3×3 convolution with a dilation rate of 3, a batch normalization layer, and a rectified linear unit to generate a large receptive field feature map F. large ; Adaptive receptive field weight factor S∈R 1×1×C Responsible for weighting the small receptive field feature map F small , weight factor 1-S∈R 1×1×C It is responsible for weighting the large receptive field feature map F large ; The weight factor S∈R 1 ×1×C Channel by channel small receptive field feature map F small Weighted, weight factor 1-S∈R 1×1×C Channel by channel large receptive field feature map F large After obtaining two weighted feature maps, they are added pixel by pixel and finally input into a 3×3 convolution + a batch normalization layer + a rectified linear unit to generate a feature map F that is adaptive to the object scale. out .

5. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: The side output module includes a 3×3 convolution, a batch normalization layer, a rectified linear unit and a 3×3 convolution; the side output module is responsible for converting the input feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the side output branch is supervised using the cross entropy loss function.

6. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: The final output layer contains a 1×1 convolution, which is responsible for transforming the input feature map F H×W×C Mapped to the semantic category probability map R that is consistent with the length and width of the feature map H×W×C , C is the number of categories of the semantic segmentation dataset currently used; the final output layer is supervised using the Lovász softmax loss function.

7. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 2, characterized in that: At each scale, the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated and then combined with the top-down feature F top The calculation process of fusion after upsampling is as follows: the feature F after long-range information interaction lr and the feature F that is adaptive to object size ad are concatenated to generate concatenated features F cat ; Top-down feature F top Use a 2×2 transposed convolution layer to upsample by 2 times to obtain the concatenated feature F cat Features F with the same length and width top-resample ; Upsampled features F top-resample Then concatenate the features F cat Perform concatenation to generate a new concatenation feature F cat-new ; Then, two sets of sequentially stacked 3×3 convolution + a batch normalization layer + a rectified linear unit are used to further fuse the internal information and output the fused features.

8. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 2, characterized in that: The 2x features are further upsampled and then processed by the final output layer to obtain the final pixel-level semantic category probability map that is consistent with the input image resolution. The upsampling operation consists of a 2×2 transposed convolution layer and two sets of sequentially stacked 3×3 convolutions + a batch of normalization layers + rectified linear units.

9. The method for semantic segmentation of remote sensing images based on supervised long-range correlation according to claim 1, characterized in that: The specific implementation of step 2 includes the following sub-steps: Step 2.1: Label the category of each pixel in the remote sensing image and generate a semantic category ground truth grid map. Crop the original large-scale remote sensing image and the corresponding semantic category ground truth grid map to the standard size supported by the network model, i.e., 512×512 pixels, to generate a training sample set. Step 2.2: Train a supervised long-range correlation semantic segmentation network. Use the cropped image patches generated in step 2.1 and the corresponding cropped semantic category ground truth grid as training data, and iteratively train the semantic segmentation network until the model converges. In step 2.3, given a new remote sensing image to be predicted, the original large-scale remote sensing image is cropped with a certain step size and input into the model with loaded training weights to obtain the pixel-level prediction results on each cropped image; the pixel-level prediction results of all cropped images are mapped back to the original remote sensing image to obtain the pixel-level category prediction result map of the entire large-scale remote sensing image.

Images