An edge feature enhancement-based remote sensing image semantic segmentation method

Abstract

The application discloses an edge feature enhancement-based remote sensing image semantic segmentation method, and belongs to the technical field of remote sensing semantic segmentation, and comprises the following steps: constructing a data set and dividing a training set, a test set and a verification set; constructing a segmentation network; the segmentation network comprises a backbone network, an edge feature extraction module, a semantic feature extraction module and a feature fusion prediction module; the backbone network is used for acquiring an intermediate feature map; the edge feature extraction module is used for acquiring an edge feature map; the semantic feature extraction module is used for acquiring a semantic feature map; the feature fusion prediction module is used for acquiring a mixed feature map by using the edge feature map and the semantic feature map and predicting a final segmentation result; and training a model; in the training stage, an edge detection head is added to the edge feature extraction module, a semantic segmentation head is added to the semantic feature extraction module, and deep supervision is performed on the edge detection head and the semantic segmentation head. The application has strong generalization ability, the segmentation result is more accurate, and a post-processing method designed by a person is not needed.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing semantic segmentation technology, specifically relating to a remote sensing image semantic segmentation method based on edge feature enhancement. Background Technology

[0002] Remote sensing semantic segmentation classifies each pixel in a remote sensing image into different features or objects present in the image, such as buildings, roads, and water bodies. It has wide applications in fields such as Geographic Information Systems (GIS), urban planning, resource management, and environmental protection, providing decision-makers with more accurate and comprehensive information, thus contributing to more scientific policy-making and planning.

[0003] Existing remote sensing semantic segmentation methods are typically based on neural networks, achieving target category region segmentation through sample labeling and model training. Many methods introduce edge detection to improve semantic segmentation accuracy, with two common approaches: The first approach designs a separate model for edge detection and fuses the edge detection results with the contour results extracted from semantic segmentation through morphological post-processing to obtain the final target category contour prediction. This method increases the overall computational load of the algorithm by designing a separate edge detection model and heavily relies on manually designed post-processing, resulting in poor generalization ability. The second approach adds an edge detection branch to the segmentation model and designs a hybrid loss function for supervised training during the edge detection branch. This method, after fusion prediction, may result in discontinuities between the internal and edge segments of the same category of target regions. Therefore, providing a remote sensing semantic segmentation algorithm with strong generalization ability that fuses edge features has become a pressing technical problem for those skilled in the art.

[0004] Chinese patent CN113408350A discloses a "Remote Sensing Image Saliency Detection Method Based on Innovative Edge Feature Extraction." This method first uses a ResNet34-based encoder to extract features from the remote sensing image. Then, the input features are decoded in decoders with edge feature branches and saliency feature branches. Finally, continuous convolutions are used to fuse and predict the decoded features, resulting in the final output saliency prediction map. However, this method does not effectively supervise the features resulting from the fusion of edge and semantic information.

[0005] Chinese patent CN110570427A discloses a "method and apparatus for semantic segmentation of remote sensing images with fused edge detection." This method first creates semantic segmentation labels and edge detection labels based on feature outlines. Then, it trains a Heated Edge Detection (HED) edge detection model and a semantic segmentation model separately. Finally, during the testing phase, the semantic segmentation result obtained from the semantic segmentation model is overlaid and fused with the edge detection result obtained from the edge detection model based on geographic location to obtain the final segmentation result. The purpose of edge detection is to improve the accuracy of target segmentation near boundaries. This method does not consider the reuse of core features and designs and trains a separate model for the edge detection task, increasing the overall computational load.

[0006] In summary, existing remote sensing semantic segmentation algorithms that incorporate edge detection have the following drawbacks:

[0007] 1) Weak generalization ability: The receptive field of the CNN backbone network is small, resulting in insufficient feature extraction ability.

[0008] 2) Inaccurate boundary segmentation results: The supervision of edge feature and semantic feature fusion is insufficient, and there are cases where the edge and the interior are separated in the prediction results.

[0009] 3) Post-processing of contour fusion relies on prior human knowledge. Summary of the Invention

[0010] To address the shortcomings of existing remote sensing semantic segmentation algorithms that incorporate edge detection, such as weak generalization ability, inaccurate boundary segmentation results, and reliance on prior human knowledge for contour fusion post-processing, this invention provides a remote sensing image semantic segmentation method based on edge feature enhancement. First, a transformer-based backbone network is used to extract feature information of different sizes. Then, an edge feature extraction module and a semantic feature extraction module are designed to decode the features extracted by the backbone network, respectively, to obtain edge features and semantic features. Finally, the semantic features and edge features are fused to obtain the final prediction result. During training, loss functions are designed for the edge feature extraction module, semantic feature extraction module, and feature fusion prediction module for supervised training, improving the accuracy of ground object segmentation results.

[0011] The technical solution adopted by this invention to solve the technical problem is as follows:

[0012] The present invention provides a semantic segmentation method for remote sensing images based on edge feature enhancement, comprising the following steps:

[0013] Step S1: Construct the dataset and divide it into training, test, and validation sets;

[0014] Step S2: Construct the segmentation network;

[0015] The segmentation network includes: a backbone network, an edge feature extraction module, a semantic feature extraction module, and a feature fusion prediction module; the backbone network is used to obtain intermediate feature maps; the edge feature extraction module is used to obtain edge feature maps; the semantic feature extraction module is used to obtain semantic feature maps; and the feature fusion prediction module is used to obtain a hybrid feature map using the edge feature maps and semantic feature maps and predict the final segmentation result.

[0016] Step S3: Train the model;

[0017] During the training phase, an edge detection head is added to the edge feature extraction module, and a semantic segmentation head is added to the semantic feature extraction module. Deep supervision is then applied to both the edge detection head and the semantic segmentation head.

[0018] Furthermore, the specific operation procedure for step S1 is as follows:

[0019] S1.1: Obtain initial remote sensing images and semantic annotation information;

[0020] S1.2: Extract edge region map information based on semantic annotation information;

[0021] S1.3: Divide the obtained data into training set, test set and validation set according to the ratio of data volume 8:1:1.

[0022] Furthermore, in step S1.2, the Canny operator is first used to extract edge map information from the semantic annotation information; then, an image morphology-based dilation operation is used to obtain edge region map information, with the foreground being the region near the edge boundary and the background being the internal region of different types of targets.

[0023] Furthermore, the backbone network includes four transformer blocks and one multilayer perceptual layer; each transformer block includes one convolutional block and N combination modules, each combination module consisting of a multi-head attention module and a hybrid feedforward neural network; the input feature map first passes through the convolutional block, and then through the multi-head attention module and the hybrid feedforward neural network to model the global information.

[0024] Furthermore, the edge feature extraction module includes a bottom-up feature aggregation path and a top-down feature aggregation path. For intermediate feature maps of different resolutions extracted by the backbone network, the feature maps are first restored to the same resolution and the same number of channels by a deconvolution layer. Then, they are processed by the top-down feature aggregation path and the bottom-up feature aggregation path to obtain feature maps of different levels. Finally, the multiple feature maps are concatenated and processed by convolution to obtain the final edge feature map.

[0025] Furthermore, the semantic feature extraction module performs convolution operations on the intermediate feature maps of different resolutions extracted by the backbone network, in order from low to high resolution, and then upsamples them to restore the resolution of the adjacent feature maps. Finally, it concatenates the intermediate feature maps with the adjacent feature maps along the channel dimension to obtain the fused semantic feature map, thereby achieving the fusion of high-level semantic information and low-level detailed information.

[0026] Furthermore, the semantic feature map extracted by the semantic feature extraction module and the edge feature map extracted by the edge feature extraction module are respectively convolved, then added together and convolved again to obtain a new feature map; the new feature map and the semantic feature map are concatenated along the channel dimension and then convolved to obtain a mixed feature map; finally, softmax prediction is performed to obtain the final result.

[0027] Furthermore, in step S3, during the training phase, loss functions are designed for the edge feature extraction module, semantic feature extraction module, and feature fusion prediction module respectively; the loss function used for deep supervision of the edge feature extraction module is as follows:

[0028]

[0029] Where i represents a data sample; This represents the label corresponding to sample i in the edge map information; This represents the prediction result of the edge feature extraction module for sample i; ,in This indicates the number of positive samples in the edge map information; This indicates the number of negative samples in the edge map information;

[0030] The loss function used for deep supervision of the semantic feature extraction module is as follows:

[0031]

[0032] Where M represents the number of categories; X represents the segmentation result; and Y represents semantic annotation information. This indicates the number of samples corresponding to category c in the segmentation result; This indicates the number of samples corresponding to category c in the semantic annotation information; This represents the number of correctly predicted samples in category c based on the segmentation result.

[0033] The loss function used for supervised training of the feature fusion prediction module is as follows:

[0034]

[0035]

[0036] Where i represents a data sample; N represents the total number of data samples; and M represents the number of categories; The label representing sample i; This represents the model's prediction result for sample i; E represents the edge weight of sample i; E represents edge region map information.

[0037] Furthermore, the loss function during the training phase is:

[0038]

[0039] in, This is the weighting balance coefficient.

[0040] The beneficial effects of this invention are:

[0041] Compared with existing remote sensing semantic segmentation algorithms that incorporate edge detection, the remote sensing image semantic segmentation method proposed in this invention, based on edge feature enhancement, improves the performance of remote sensing semantic segmentation algorithms in the following aspects:

[0042] 1. Strong generalization ability: The backbone network based on transformer is selected to encode the global information of remote sensing images, which improves the feature extraction ability of the network.

[0043] 2. More accurate segmentation results: An edge feature extraction model and a feature fusion prediction module were designed to effectively utilize edge features. With the supervision of the loss function, the accuracy of segmentation results near the edges of different categories of targets was improved.

[0044] 3. The end-to-end segmentation process does not rely on manually designed post-processing methods. Attached Figure Description

[0045] Figure 1 This is a schematic diagram for extracting edge region map information based on semantic annotation information.

[0046] Figure 2 This is a flowchart of a remote sensing image semantic segmentation method based on edge feature enhancement according to the present invention.

[0047] Figure 3 This is a diagram of the backbone network structure.

[0048] Figure 4 This is a structural diagram of the edge feature extraction module.

[0049] Figure 5 This is a structural diagram of the semantic feature extraction module.

[0050] Figure 6 This is a structural diagram of the feature fusion prediction module. Detailed Implementation

[0051] This invention provides a semantic segmentation method for remote sensing images based on edge feature enhancement. First, a transformer-based backbone network is used to extract feature information of different sizes. Then, an edge feature extraction module and a semantic feature extraction module are designed to decode the features extracted by the backbone network, respectively, to obtain edge features and semantic features. Finally, the semantic features and edge features are fused to obtain the final prediction result. During training, loss functions are designed for the edge feature extraction module, the semantic feature extraction module, and the feature fusion prediction module for supervised training, which improves the accuracy of the ground object segmentation results.

[0052] This invention provides a semantic segmentation method for remote sensing images based on edge feature enhancement. The method includes the following steps: 1) constructing a dataset; 2) constructing a model; 3) training the model.

[0053] The present invention provides a semantic segmentation method for remote sensing images based on edge feature enhancement, the specific operation process of which is as follows:

[0054] Step S1: Construct the dataset and divide it into training, test, and validation sets;

[0055] S1.1: Obtain the initial remote sensing image and semantic annotation information, and cut it into a 512×512 image patch;

[0056] S1.2: Extract edge region map information based on semantic annotation information;

[0057] Specifically, such as Figure 1 As shown, the Canny operator is first used to extract edge map information from semantic annotation information; then, the dilation operation based on image morphology is used to obtain edge region map information, where the label value in the edge region map information is 0 or 1; the foreground is the region near the edge boundary, and the background is the internal region of different types of targets.

[0058] S1.3: Divide the obtained data into training set, test set and validation set according to the ratio of data volume 8:1:1.

[0059] Step S2: Construct the segmentation network;

[0060] like Figure 2 As shown, the segmentation network mainly includes: a backbone network, an edge feature extraction module, a semantic feature extraction module, and a feature fusion prediction module.

[0061] Specifically, the backbone network of the segmentation network consists of four transformer blocks and one multilayer perceptron (MLPLayer). The structure of the transformer block is as follows: Figure 3As shown, the transformer block contains one convolutional block and N ensemble modules. Each ensemble module consists of a multi-head self-attention module and a mixed feedforward neural network (Mix-FFN). The input feature map first passes through the convolutional block, and then through the multi-head attention module and the mixed feedforward neural network to model the global information. This process can be represented as follows:

[0062]

[0063] in, This represents the input features of the transformer block. This represents the features extracted by the transformer block. This represents a convolutional block; the kernel size of the convolutional block in the first group of transformer blocks is 7, the stride is 4, and the padding is 3; the kernel size of the convolutional blocks in the second to fourth groups of transformer blocks is 3, the stride is 2, and the padding is 1; the total number of combination modules in the first to fourth groups of transformer blocks is 1, 2, 5, and 8, respectively.

[0064] The input image undergoes feature extraction from each transformer block, followed by transformation through multiple perceptron layers (MLPLayer), resulting in four resolutions. , , , The intermediate feature map, where H and W represent the height and width of the feature map.

[0065] The structure of the edge feature extraction module is as follows: Figure 4 As shown, the edge feature extraction module includes a bottom-up feature aggregation path and a top-down feature aggregation path. For intermediate feature maps of different resolutions extracted by the backbone network, a deconvolution layer is first applied to restore the feature maps to the same resolution and number of channels. Then, feature maps at different levels are obtained through both the top-down and bottom-up feature aggregation paths. Finally, the multiple feature maps are concatenated and subjected to convolution to obtain the final edge feature map. This process can be represented as follows:

[0066]

[0067]

[0068]

[0069] in, Feature maps representing different resolutions. This represents the result of the bottom-up feature aggregation path. This represents the result of the top-down feature aggregation path. This represents the extracted edge feature map; concat represents the operation of concatenating the feature maps along the channel dimension; add represents the operation of adding elements at corresponding positions of the features; conv represents a convolution operation with a kernel size of 3, a stride of 1, and padding of 1; deconv represents a deconvolution operation.

[0070] The semantic feature extraction module has the following structure: Figure 5 As shown, the intermediate feature maps of different resolutions extracted by the backbone network are first subjected to convolution operations in order of increasing resolution, then upsampled to restore the resolution of adjacent feature maps, and finally concatenated with adjacent feature maps along the channel dimension to obtain the fused semantic feature map. This achieves the fusion of high-level semantic information and low-level detail information. The process can be represented as follows:

[0071]

[0072] in, Feature maps representing different resolutions. This represents the fused semantic feature map. concat means concatenating the feature maps along the channel dimension. upsample means a 2x upsampling operation. conv means a convolution operation with a kernel size of 3, a stride of 1, and padding of 1.

[0073] The structure of the feature fusion prediction module is as follows: Figure 6 As shown, firstly, the semantic feature map extracted by the semantic feature extraction module and the edge feature map extracted by the edge feature extraction module are convolved and then added together to obtain a new feature map; then, the new feature map and the semantic feature map are concatenated along the channel dimension and convolved to obtain a mixed feature map; finally, softmax prediction is performed to obtain the final result. The process can be represented as follows:

[0074]

[0075]

[0076] in, This represents the fused semantic feature map; This represents the extracted edge feature map; This represents the resulting mixed feature map; predict represents the model prediction result; conv represents a convolution operation with a kernel size of 3, a stride of 1, and padding of 1. The convolution operation has a kernel size of 1; concat concatenates feature maps along the channel dimension; add adds corresponding elements of the features; and upsample performs a 2x upsampling operation.

[0077] Step S3: Train the model;

[0078] like Figure 2 As shown, during the training phase, an edge detection head is added to the edge feature extraction module, and a semantic segmentation head is added to the semantic feature extraction module. Deep supervision is then applied to both the edge detection head and the semantic segmentation head. This process can be represented as follows:

[0079]

[0080]

[0081] in, This represents the prediction results of the deep-supervised semantic feature extraction module (i.e. Figure 2 (semantic segmentation results in the middle) This represents the prediction result of the depth supervision of the edge feature extraction module (i.e. Figure 2 (edge ​​prediction results in the middle) This represents the fused semantic feature map; This represents the extracted edge feature map; This indicates a convolution operation with a kernel size of 1; upsample indicates a 2x upsampling operation.

[0082] Loss functions are designed for the edge feature extraction module, semantic feature extraction module, and feature fusion prediction module, respectively. The loss function used for deep supervision of the edge feature extraction module is as follows:

[0083]

[0084] Where i represents a data sample; This represents the label corresponding to sample i in the edge map information; This represents the prediction result of the edge feature extraction module for sample i; ,in This indicates the number of positive samples in the edge map information; This indicates the number of negative samples in the edge map information.

[0085] The edge detection loss is calculated using the loss function described above for depth supervision of the edge feature extraction module.

[0086] The loss function used for deep supervision of the semantic feature extraction module is as follows:

[0087]

[0088] Where M represents the number of categories; X represents the segmentation result; and Y represents semantic annotation information. This indicates the number of samples corresponding to category c in the segmentation result; This indicates the number of samples corresponding to category c in the semantic annotation information; This represents the number of correctly predicted samples in category c based on the segmentation result.

[0089] The semantic segmentation loss is calculated using the loss function described above for deep supervision of the semantic feature extraction module.

[0090] The loss function used for supervised training of the feature fusion prediction module is as follows:

[0091]

[0092]

[0093] Where i represents a data sample; N represents the total number of data samples; and M represents the number of categories; The label representing sample i; This represents the model's prediction result for sample i; E represents the edge weight of sample i; E represents edge region map information.

[0094] The fusion prediction loss is calculated using the loss function described above for deep supervision of the semantic feature extraction module.

[0095] In the supervised training of the feature fusion prediction module, a loss function based on prior edge information was designed to enhance supervision, thereby improving the supervision of pixels near the target edge and effectively alleviating the problem of inconsistent prediction results between the target's interior and edges.

[0096] The loss function during the training phase is:

[0097]

[0098] in, This is the weighting balance coefficient.

[0099] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on edge feature enhancement, characterized in that, Includes the following steps: Step S1: Construct the dataset and divide it into training, test, and validation sets; Step S2: Construct the segmentation network; The segmentation network includes: a backbone network, an edge feature extraction module, a semantic feature extraction module, and a feature fusion prediction module; the backbone network is used to obtain intermediate feature maps; the edge feature extraction module is used to obtain edge feature maps; the semantic feature extraction module is used to obtain semantic feature maps; and the feature fusion prediction module is used to obtain a hybrid feature map using the edge feature maps and semantic feature maps and predict the final segmentation result. The semantic feature map extracted by the semantic feature extraction module and the edge feature map extracted by the edge feature extraction module are respectively convolved, then added together and convolved again to obtain a new feature map; the new feature map and the semantic feature map are concatenated along the channel dimension and then convolved to obtain a mixed feature map; finally, softmax prediction is performed to obtain the final result. Step S3: Train the model; During the training phase, an edge detection head is added to the edge feature extraction module, and a semantic segmentation head is added to the semantic feature extraction module. Deep supervision is then applied to both the edge detection head and the semantic segmentation head. During the training phase, loss functions are designed for the edge feature extraction module, semantic feature extraction module, and feature fusion prediction module respectively; the loss function used for deep supervision of the edge feature extraction module is as follows: ； Where i represents a data sample; This represents the label corresponding to sample i in the edge map information; This represents the prediction result of the edge feature extraction module for sample i; ,in This indicates the number of positive samples in the edge map information; This indicates the number of negative samples in the edge map information; The loss function used for deep supervision of the semantic feature extraction module is as follows: ； Where M represents the number of categories; X represents the segmentation result; and Y represents semantic annotation information. This indicates the number of samples corresponding to category c in the segmentation result; This indicates the number of samples corresponding to category c in the semantic annotation information; This represents the number of correctly predicted samples in category c based on the segmentation result. The loss function used for supervised training of the feature fusion prediction module is as follows: ； ； Where i represents a data sample; N represents the total number of data samples; and M represents the number of categories; The label representing sample i; This represents the model's prediction result for sample i; E represents the edge weight of sample i; E represents edge region map information. The loss function during the training phase is: ； in, This is the weighting balance coefficient.

2. The remote sensing image semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, The specific operation procedure for step S1 is as follows: S1.1: Obtain initial remote sensing images and semantic annotation information; S1.2: Extract edge region map information based on semantic annotation information; S1.3: Divide the obtained data into training set, test set and validation set according to the ratio of data volume 8:1:

1.

3. The remote sensing image semantic segmentation method based on edge feature enhancement according to claim 2, characterized in that, In step S1.2, the Canny operator is first used to extract edge map information from the semantic annotation information; then, the dilation operation based on image morphology is used to obtain edge region map information, with the foreground being the region near the edge boundary and the background being the internal region of different types of targets.

4. The remote sensing image semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, The backbone network comprises four transformer blocks and one multilayer perceptual layer. Each transformer block includes one convolutional block and N ensemble modules, each of which consists of a multi-head attention module and a hybrid feedforward neural network. The input feature map first passes through the convolutional block, and then through the multi-head attention module and the hybrid feedforward neural network to model the global information.

5. The remote sensing image semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, The edge feature extraction module includes a bottom-up feature aggregation path and a top-down feature aggregation path. For intermediate feature maps of different resolutions extracted by the backbone network, the feature maps are first restored to the same resolution and the same number of channels by a deconvolution layer. Then, they are processed by the top-down and bottom-up feature aggregation paths to obtain feature maps at different levels. Finally, the multiple feature maps are concatenated and convolutional to obtain the final edge feature map.

6. The semantic segmentation method for remote sensing images based on edge feature enhancement according to claim 1, characterized in that, The semantic feature extraction module takes intermediate feature maps of different resolutions extracted by the backbone network, and performs convolution operations on them in order of increasing resolution. Then, it upsamples them to restore the resolution of the adjacent feature maps. Finally, it concatenates the intermediate feature maps with the adjacent feature maps along the channel dimension to obtain the fused semantic feature map, thereby achieving the fusion of high-level semantic information and low-level detailed information.

Images