An edge-enhanced remote sensing image segmentation method and system integrating attention and spatial state models

Abstract

Invention Name: A method and system for edge-enhanced remote sensing image segmentation that integrates attention and spatial state models Abstract: The present application discloses a method and system for edge-enhanced remote sensing image segmentation that integrates attention and spatial state models. The implementation steps are: constructing an edge texture feature enhancement structure; introducing the edge texture enhancement structure into the SegNext semantic segmentation model; dividing the remote sensing image segmentation dataset to generate a training sample set, a verification sample set, and a test sample set; preprocessing the dataset; using a neural network to preliminarily extract fine features of the optical remote sensing image, and then training the model with an edge texture enhancement decoder of the channel attention and spatial state model; finally, sending the test sample data to the edge texture enhancement model of the trained attention and spatial state model to obtain the test results. The patent of this invention utilizes the constructed edge texture feature enhancement module and the SegNext semantic segmentation model for collaborative training, which enhances the edge texture features while ensuring the features of the ground objects, thereby improving the accuracy of segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of semantic segmentation of remote sensing images, and specifically to an edge-enhanced remote sensing image segmentation method and system that integrates attention and spatial state models. The present invention can be used to improve the accuracy of remote sensing image segmentation. Background Art

[0002] Semantic segmentation uses neural networks to perform end-to-end segmentation. It involves assigning each pixel in an image to a specific category label, achieving precise object boundary detection. Over the past decade, deep learning's tremendous success in semantic segmentation has been fueled by numerous researchers' research and improvements to deep learning networks. Fully convolutional networks (FCNs) laid the foundation for deep learning models in semantic segmentation. SegNet proposed a unique encoder-decoder architecture. UNet effectively improved image segmentation accuracy by combining an encoder-decoder architecture with skip connections. The Pyramid Scene Parsing Network (PSPNet) and the DeepLab series recognized the need for networks to extract contextual information. Later, the Transformer, which has performed well in natural language processing, demonstrated impressive performance in computer vision using its self-attention mechanism. Various Transformer-based network frameworks have also rapidly developed, surpassing traditional convolutional neural network architectures. SegNeXt proposed a convolutional architecture that is more efficient at encoding contextual information than the Transformer's self-attention mechanism. Remote sensing semantic segmentation technology is being applied to disaster warning, enabling the identification of objects such as buildings, farmland, and disaster zones. By comparing with recent remote sensing image data, people can assess potential disasters in the area, such as floods and landslides, which can help with agricultural safety, flood management, and disaster relief planning. However, in remote sensing semantic segmentation, due to the large number of targets contained in satellite images, the spatial and edge features of different targets vary greatly, while the same targets vary in scale, orientation, and other aspects. In addition, the features and edges of small targets at different resolutions are difficult to identify. As a result, deep learning networks often face edge prediction errors and edge discontinuities when faced with discontinuous edges and small targets (such as water bodies, trees, and roads). Therefore, it is necessary to design a structure that can improve the accuracy of object edges in remote sensing image segmentation and enhance the applicability of the network. Summary of the Invention

[0003] A method and system for edge-enhanced remote sensing image segmentation that integrates attention and spatial state models is characterized by changing the structure of the original model, using skip connections for splicing, using a convolutional neural network to extract fine feature maps of optical remote sensing images, using a multi-stage encoder-decoder structure to fuse the feature maps, and using an edge enhancement module in the decoder that combines a spatial state model with channel attention for edge enhancement. The specific steps of the method include the following:

[0004] Step 1: Design an edge enhancement module that combines channel attention and spatial state model:

[0005] Add a channel attention mechanism to the VSS block of VMamba. The structure of the VSS block is: after normalization, the input is divided into two branches. In the first branch, the input passes through a linear layer and calculates the channel attention, and then passes through an activation function. In the second branch, the input is processed by a linear layer, a depth-separable convolution, and an activation function, and then input into a 2D selective scan (SS2D) module for further feature extraction. Subsequently, the features are normalized using layer normalization, and then element-wise generation is performed using the output of the first branch to merge the two paths. Finally, the features are mixed using the product of the linear layer and the channel attention, and this result is combined with a residual connection to form the output of the VSS block. Among them, an edge enhancement module is constructed after adding the channel attention mechanism to the linear layer of the first branch;

[0006] Step 2: Add the constructed edge enhancement module to the SegNext model:

[0007] The encoder of the SegNext model has four stages of output. The constructed edge enhancement module is placed after the last stage and added after the four corresponding upsampling stages of the decoder.

[0008] Step 3: Establish a skip connection structure:

[0009] Add the outputs of the four stages of the SegNext model encoder and decoder to establish a skip connection;

[0010] Step 4: Divide the ground feature segmentation dataset into training sample set, validation sample set, and test sample set:

[0011] Select some labeled segmented remote sensing images, and select at least 500 remote sensing image segmentation data of no size requirement, where the training set, validation set, and test set are divided in a ratio of 8:1:1;

[0012] Step 5: Preprocess the remote sensing segmentation dataset:

[0013] First, the images in each batch are preprocessed by random cropping, random flipping, etc., and the images are cropped to a size of 512*512;

[0014] Step 6: Use the modified edge-enhanced structure SegNext network model to train the dataset:

[0015] In the first step, the preprocessed training samples are input into the SegNext backbone network (MSCAN) for feature extraction, generating four layers of different features.

[0016] In the second step, the feature maps of these four different levels are upsampled. During the upsampling process, four different edge texture features are obtained through the constructed edge enhancement module. These four layers of features are fused using skip connections to obtain four fused feature maps. Finally, these four fused feature maps are sent to the segmentation head for category prediction, and then the cross entropy loss is calculated with the true label.

[0017] Step 7, get the segmentation result:

[0018] Send the test sample data into the trained SegNext edge feature enhancement model to obtain the test results;

[0019] Step 8, performance evaluation:

[0020] Use the predicted result map in step 7 and the label map of the remote sensing image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance;

[0021] The categories of evaluation indicators include:

[0022] Intersection over Union (IoU):

[0023] ;

[0024] In the formula, i represents a positive example; j represents a negative example; IoU represents the intersection-over-union ratio, that is, the ratio of the intersection and union of each category's prediction results and the true value; pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP; pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP; pji represents the total number of pixels whose true category is i and is identified as category j, that is, false negative examples FN;

[0025] Average pixel accuracy MPA:

[0026] ;

[0027] In the formula, k represents the total number of categories, i represents the positive example, j represents the negative example, pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP, and pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the flow chart of each module of the edge enhancement method.

[0029] Figure 2 It is a test result diagram of the present invention.

[0030] Figure 3 It is a structural diagram of the module of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be described in further detail below with reference to the accompanying drawings.

[0032] Refer to the attached Figure 1 , the steps of the present invention are further described in detail.

[0033] 1. A method and system for edge-enhanced remote sensing image segmentation that integrates attention and spatial state models, characterized by modifying the structure of the original model and using skip connections for splicing. A convolutional neural network is used to extract fine feature maps from optical remote sensing images, which are then fused using a multi-stage encoder-decoder structure. An edge enhancement module in the decoder that incorporates a spatial state model with channel attention is used for edge enhancement. The method comprises the following steps:

[0034] Step 1: Design an edge enhancement module that combines channel attention and spatial state model:

[0035] Add a channel attention mechanism to the VSS block of VMamba. The structure of the VSS block is: after normalization, the input is divided into two branches. In the first branch, the input passes through a linear layer and calculates the channel attention, and then passes through an activation function. In the second branch, the input is processed by a linear layer, a depth-separable convolution, and an activation function, and then input into a 2D selective scan (SS2D) module for further feature extraction. Subsequently, the features are normalized using layer normalization, and then element-wise generation is performed using the output of the first branch to merge the two paths. Finally, the features are mixed using the product of the linear layer and the channel attention, and this result is combined with a residual connection to form the output of the VSS block. Among them, an edge enhancement module is constructed after adding the channel attention mechanism to the linear layer of the first branch;

[0036] Step 2: Add the constructed edge enhancement module to the SegNext model:

[0037] The encoder of the SegNext model has four stages of output. The constructed edge enhancement module is placed after the last stage and added after the four corresponding upsampling stages of the decoder.

[0038] Step 3: Establish a skip connection structure:

[0039] Add the outputs of the four stages of the SegNext model encoder and decoder to establish a skip connection;

[0040] Step 4: Divide the ground feature segmentation dataset into training sample set, validation sample set, and test sample set:

[0041] Select some labeled segmented remote sensing images, and select at least 500 remote sensing image segmentation data of no size requirement, where the training set, validation set, and test set are divided in a ratio of 8:1:1;

[0042] Step 5: Preprocess the remote sensing segmentation dataset:

[0043] First, the images in each batch are preprocessed by random cropping, random flipping, etc., and the images are cropped to a size of 512*512;

[0044] Step 6: Use the modified edge-enhanced structure SegNext network model to train the dataset:

[0045] In the first step, the preprocessed training samples are input into the SegNext backbone network (MSCAN) for feature extraction, generating four layers of different features.

[0046] In the second step, the feature maps of these four different levels are upsampled. During the upsampling process, four different edge texture features are obtained through the constructed edge enhancement module. These four layers of features are fused using skip connections to obtain four fused feature maps. Finally, these four fused feature maps are sent to the segmentation head for category prediction, and then the cross entropy loss is calculated with the true label.

[0047] Step 7, get the segmentation result:

[0048] Send the test sample data into the trained SegNext edge feature enhancement model to obtain the test results;

[0049] Step 8, performance evaluation:

[0050] Use the predicted result map in step 7 and the label map of the remote sensing image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance;

[0051] The categories of evaluation indicators include:

[0052] Intersection over Union (IoU):

[0053] ;

[0054] In the formula, i represents a positive example; j represents a negative example; IoU represents the intersection-over-union ratio, that is, the ratio of the intersection and union of each category's prediction results and the true value; pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP; pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP; pji represents the total number of pixels whose true category is i and is identified as category j, that is, false negative examples FN;

[0055] Average pixel accuracy MPA:

[0056] ;

[0057] In the formula, k represents the total number of categories, i represents the positive example, j represents the negative example, pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP, and pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP.

[0058] Simulation experiment conditions:

[0059] 1. Simulation experiment conditions of the present invention: Server GPU: GeForce RTX3090 Ti, video memory 24G.

[0060] 2. The simulation experiment platform of the patent of this invention is: Ubuntu 20.04.4 system, Python 3.10.13, and Pytroch-gpu 2.1.1.

[0061] Simulation experiment content and analysis of experimental results:

[0062] This paper presents a simulation experiment that combines this invention with three existing techniques (SegNext segmentation, CBAM channel attention module, VMamba segmentation, and spatial state model). The paper then designs an edge enhancement module in conjunction with the channel attention module. This module is then introduced into SegNext to segment remote sensing images and obtain structural results. The simulations use the LRBS and WHDLD datasets.

[0063] The existing technologies used in the simulation experiment are:

[0064] The SegNext image segmentation method refers to the image segmentation method proposed by Guo Meng-Hao et al. in "Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems 35 (2022): 1140-1156.", referred to as the SegNext segmentation method.

[0065] The VMamba image segmentation method refers to the Vision mamba: Efficient visual representation learning with bidirectional state space model proposed by Zhu, Lianghui, et al., and is referred to as the VMamba segmentation method. The spatial state model refers to the core module VSS block and its structure in VMamba.

[0066] The PSPNet image segmentation method refers to the image segmentation method proposed by Zhao, Hengshuang et al. in "Pyramid scene parsing network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. It is referred to as the PSPNet segmentation method.

[0067] The prior art CBAM attention mechanism refers to the simple and effective feedforward convolutional neural network attention mechanism proposed by Woo, Sanghyun et al. in "Cbam: Convolutionalblock attention module. Proceedings of the European conference on computervision (ECCV). 2018.", referred to as the CBAM attention mechanism.

[0068] The input image used in the simulation experiment of this invention is the public WHDLD remote sensing dataset

[0069] The WHDLD dataset, released by Wuhan University, is a densely labeled dataset primarily used for semantic segmentation of remote sensing images. WHDLD contains 4,940 256 × 256 RGB images captured by the Gaofen-1 and Changyun-3 satellites over the urban area of ​​Wuhan. Through image fusion and resampling, the image resolution reaches 2 m / pixel. The images in WHDLD are labeled into six categories.

[0070] Simulation experiment 1 is the experimental result of introducing edge feature enhancement structure into SegNext.

[0071] Simulation Experiment 2 is the experimental result of the SegNext-1 method in the prior art under the above simulation conditions.

[0072] Simulation Experiment 3 is the experimental result of the prior art PSPNet method under the above simulation conditions.

[0073] Table 1. Comparison of results of simulation experiments of the present invention

[0074]

[0075] From Table 1, we can see that compared with the two existing methods, PSPNet and SegNext, the average intersection-over-union (IoU) of the present invention is 65.16, the average mAcc is 76.25, and the most important one is 82.53 in the vegetation class. All three indicators are higher than the two existing technical methods, especially the vegetation class, which is improved by 1.3 percentage points. This proves that the vegetation segmentation accuracy obtained by the present invention is higher.

[0076] Figure 2 The present invention obtains the prediction result graph of the LRBS remote sensing data set under the above experimental conditions. It can be seen from the prediction result graph and the true value graph that the boundary area in the prediction result graph is close to the boundary area in the true value graph, and the prediction result accurately shows the boundary.

[0077] The above simulation experiments demonstrate that this invention, employing an edge feature enhancement architecture combining SegNext, a spatial state model, and an attention mechanism, can extract more precise edge details of objects. This architecture addresses the issue of illogical predictions for small objects and fuzzy boundaries to a certain extent. By improving the network's accuracy and generalization for complex boundary segmentation, this invention provides a highly accurate method for remote sensing imagery object segmentation.

Claims

1. An edge-enhanced remote sensing image segmentation method integrating attention and spatial state models, characterized in that: The structure of the original model is changed, skip connections are used for splicing, convolutional neural networks are used to extract fine feature maps of optical remote sensing images, feature maps are fused using a multi-stage encoder-decoder structure, and an edge enhancement module combined with a spatial state model of channel attention is used in the decoder for edge enhancement. The specific steps of this method include the following: Step 1: Design an edge enhancement module that combines channel attention and spatial state model: Add a channel attention mechanism to the VSS block of VMamba; the structure of the VSS block is: after normalization, the input is divided into two branches; in the first branch, the input passes through a linear layer and calculates channel attention, and then passes through an activation function; in the second branch, the input is processed by a linear layer, a depth-separable convolution and an activation function, and then input into a 2D selective scan (SS2D) module for further feature extraction; Subsequently, the features are normalized using layer normalization, and then element-wise generation is performed using the output of the first branch to merge the two paths; Finally, the features are mixed using the product of the linear layer and the channel attention, and this result is combined with a residual connection to form the output of the VSS block; Among them, the edge enhancement module is constructed after the channel attention mechanism is added to the linear layer of the first branch; Step 2: Add the constructed edge enhancement module to the SegNext model: The encoder of the SegNext model has four stages of output. The constructed edge enhancement module is placed after the last stage and added after the four corresponding upsampling stages of the decoder. Step 3: Establish a skip connection structure: Add the outputs of the four stages of the SegNext model encoder and decoder to establish a skip connection; Step 4: Divide the ground feature segmentation dataset into training sample set, validation sample set, and test sample set: Select some labeled segmented remote sensing images, and select at least 500 remote sensing image segmentation data of no size requirement, where the training set, validation set, and test set are divided in a ratio of 8:1:1; Step 5: Preprocess the remote sensing segmentation dataset: First, the images in each batch are preprocessed by random cropping, random flipping, etc., and the images are cropped to a size of 512*512; Step 6: Use the modified edge-enhanced structure SegNext network model to train the dataset: In the first step, the preprocessed training samples are input into the SegNext backbone network (MSCAN) for feature extraction, generating four layers of different features. In the second step, the feature maps of these four different levels are upsampled. During the upsampling process, four different edge texture features are obtained through the constructed edge enhancement module. These four layers of features are fused using skip connections to obtain four fused feature maps. Finally, these four fused feature maps are sent to the segmentation head for category prediction, and then the cross entropy loss is calculated with the true label. Step 7, get the segmentation result: Send the test sample data into the trained SegNext edge feature enhancement model to obtain the test results; Step 8, performance evaluation: Use the predicted result map in step 7 and the label map of the remote sensing image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance; The categories of evaluation indicators include: Intersection over Union (IoU): In the formula, i represents a positive example; j represents a negative example; IoU represents the intersection-over-union ratio, that is, the ratio of the intersection and union of each category's prediction results and the true value; pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP; pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP; pji represents the total number of pixels whose true category is i and is identified as category j, that is, false negative examples FN; Average pixel accuracy MPA: In the formula, k represents the total number of categories, i represents the positive example, j represents the negative example, pii represents the total number of pixels whose true category is i and is identified as category i, that is, true positive examples TP, and pij represents the total number of pixels whose true category is j and is identified as category i, that is, false positive examples FP.

2. The edge-enhanced remote sensing image segmentation method integrating attention and spatial state models according to claim 1, characterized in that: The construction described in step 1, step 2, step 3 and step 6 is based on the edge texture feature enhancement structure training of SegNext, and the corresponding edge enhancement structure is designed for the edge refinement segmentation of large objects in remote sensing images to enhance the accuracy of the network for object segmentation and improve the accuracy of segmentation.

Images