Remote sensing image segmentation method and equipment based on dual-path multi-scale feature aggregation

Abstract

The invention belongs to the field of remote sensing image processing, and provides a remote sensing image segmentation method based on dual-path multi-scale feature aggregation, which comprises the following steps: S1, acquiring a remote sensing image, and dividing the remote sensing image into a training set and a verification set; s2, constructing a dual-path image segmentation model; s3, using a training set and a verification set to supervise and train the dual-path image segmentation model; and S4, segmenting a remote sensing image to be segmented by using the trained dual-path image segmentation model. According to the dual-path image segmentation model, firstly, a context feature extraction network with a transformer structure is used for modeling long-distance spatial dependency, then, a short-distance spatial feature extraction network is used for extracting static context and dynamic context features of a feature map, and finally, channel space aggregation modules CSM1 and CSM2 are used for performing feature extraction on the feature map. According to the method, adaptive aggregation is carried out on feature information from two paths and different levels, mutually dependent feature maps are selectively emphasized, and the feature interaction capability of the network is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing technology, relates to remote sensing image processing, and specifically relates to a remote sensing image segmentation method based on dual-path multi-scale feature aggregation and corresponding electronic equipment. Background Art

[0002] With the advancement of science and technology and the development of remote sensing technology, the resolution of remote sensing images we obtain is constantly improving, and the information contained is richer. Further classification of remote sensing images through semantic segmentation is a crucial task.

[0003] Traditional remote sensing image semantic segmentation mainly extracts low-level features from color, shape and texture, and then clusters and classifies these features in high-level space to perform overall image segmentation, such as logistic regression, watershed algorithm, edge operator algorithm and clustering method. These manually extracted features are difficult to optimize and require special design. Even so, most segmentation results are still not ideal. Compared with traditional methods, deep learning methods do not require manual design of features, but learn to extract features from training samples, which is more convenient and efficient. The convolutional neural network (CNN) method can learn more detailed and abstract features from a large number of sample images, avoiding the shortcomings of traditional algorithms that manually design and extract shallow features, which is very important for semantic segmentation. Long et al. proposed a fully convolutional network (FCN). Compared with traditional algorithms, it realizes a pixel-level segmentation algorithm. The FCN network can accept inputs of any size and generate outputs of corresponding sizes through effective reasoning and learning. However, due to the oversimplification of the design of the FCN decoder, 8 upsampling cannot accurately restore the precise boundaries, and the segmentation results are relatively rough. A series of works reported later used an encoder-decoder structure to capture the details of fine objects. However, this skip connection method cannot effectively connect the shallow and deep information contained in the feature maps. The low-level and fine-grained detailed feature maps generated by the encoder are connected with the high-level and coarse-grained semantic information generated by the decoder without further refinement, resulting in insufficient utilization and recognition of features.

[0004] In addition, the scales of objects contained in remote sensing images vary greatly. Objects of the same type may have different shapes and sizes, and objects of different types may have similar features. In other words, compared with images in other fields, geographic entities in remote sensing images have a higher degree of interdependence. Segmenting remote sensing images with high intra-class variance and low inter-class variance requires more effective acquisition of image context information. Summary of the invention

[0005] The purpose of the present invention is to provide a dual-path multi-scale feature aggregation network combining Transformer and CNN, and also provides corresponding equipment.

[0006] Based on the above purpose, the present invention adopts the following technical solutions:

[0007] First,

[0008] A remote sensing image segmentation method based on dual-path multi-scale feature aggregation, the steps comprising:

[0009] S1. Obtain remote sensing images and divide them into training set and validation set;

[0010] S2. Constructing a dual-path image segmentation model: The dual-path image segmentation model includes a context feature extraction network CSwin transformer and a short-range spatial feature extraction network, and the feature maps obtained by different paths are fused;

[0011] S3, using the training set and the validation set to perform supervised training on the dual-path image segmentation model;

[0012] S4, using the trained dual-path image segmentation model to segment the remote sensing image to be segmented;

[0013] Among them, the context feature extraction network CSwin transformer in S2 uses 4 stages to generate different hierarchical expressions; the image is input into CSwin transformer and processed in 4 stages in sequence to obtain feature maps CF1, CF2, CF3, and CF4 respectively; the feature map CF4 is processed by the atrous spatial pyramid pooling module ASPP to obtain the feature map F1;

[0014] The short-range spatial feature extraction network consists of four C2 blocks with convolution and improved self-attention modules. After the image is processed by different C2 blocks in turn, feature maps SF1, SF2, SF3, and SF4 are obtained respectively;

[0015] The feature maps CF1 and SF2 are processed by the channel space aggregation module to obtain the feature map F2;

[0016] The feature maps CF2 and SF3 are processed by the channel space aggregation module to obtain the feature map F3;

[0017] The feature maps CF3 and SF4 are processed by the channel space aggregation module to obtain the feature map F4;

[0018] Finally, the feature fusion module FFM is used to fuse the features of different levels obtained from the feature maps F1, F2, F3, and F4 to obtain the feature map F5. Finally, the feature map F5 outputs the segmentation result through the Find_cov module.

[0019] The specific structure of the C2 block with convolution and improved self-attention module is:

[0020] First, a CBR module with a convolution kernel size of 1×1 is used (Cov, BN and Relu operations are performed in sequence) to adjust the number of channels of the input feature map to obtain feature map C2F1; then C2F1 is passed through a CBR module with a convolution kernel size of 3×3 to obtain static context information to obtain feature map C2F1; then feature maps C2F1 and C2F2 are concatenated to obtain feature map C2F3; C2F3 then uses two continuous convolutions with a convolution kernel size of 1×1 to perform Cov, BN, Relu and Cov operations in sequence to obtain a dynamic attention matrix and obtain feature map C2F4;

[0021] After passing the feature map C2F1 through a CBR module with a convolution kernel size of 3×3 and a Softmax operation, the static context and local spatial information are obtained to obtain the feature map C2F5; finally, C2F5 is multiplied with the attention matrix C2F4 to obtain the dynamic context information C2F6 of the input feature map, and finally the dynamic context information C2F6 and the static context information C2F1 are fused and output to obtain the final result.

[0022] Furthermore, the feature maps CF1 and SF2 are processed by the channel space aggregation module CSM1 to obtain the feature map F2;

[0023] In the CSM1 module, two feature maps from different paths are first concatenated, and then the concatenated feature maps are redistributed through a two-dimensional convolution to obtain the feature map Fa.

[0024] Then, the feature map Fa is passed through the spatial attention module and the channel attention module respectively to obtain the spatial attention matrix SPA and the channel attention matrix CHA with different weights respectively; the spatial attention matrix SPA and the channel attention matrix CHA are fused and then connected with the feature map Fa with a residual structure to obtain the output feature map.

[0025] The feature maps CF2 and SF3, and the feature maps CF3 and SF4 are processed by the channel space aggregation module CSM2 for feature fusion;

[0026] In the CSM2 module, two feature maps from different paths are first made to unify the channel dimensions through a two-dimensional convolution to obtain feature maps FC1 and FC2 respectively;

[0027] Then, the feature maps FC1 and FC2 are redistributed with information weights through the spatial attention module and the channel attention module respectively, and then the feature maps with redistributed information weights are connected with the feature maps FC1 and FC2 respectively with a residual structure to obtain feature maps FC3 and FC4;

[0028] Then, the feature maps FC3 and FC4 are concatenated to fully fuse the feature information of different paths, and then further undergo a two-dimensional convolution to recalibrate the channel dimension and output the result.

[0029] The channel attention module first obtains the features of the feature map through maximum pooling and average pooling in parallel, and then passes it through a shared multi-layer perceptron model (Shared MLP) to obtain two weights. The two weights are added together and then passed through a sigmoid activation function to finally obtain the channel weight.

[0030] The spatial attention module first performs global average pooling and global maximum pooling on the input feature map in the channel dimension to obtain two feature maps, then concatenates the results of the two in the channel dimension, generates spatial attention weights through a convolutional layer and a sigmoid, and finally outputs the weighted output with the input feature map of the spatial attention.

[0031] Furthermore, in S2, each stage of the context feature extraction network includes a convolutional token embedding layer and a CSwin transformer block. The convolutional token embedding layer of the first stage uses overlapping convolutions with a convolution kernel size of 7×7 and a stride of 4 to divide the blocks. A convolutional token embedding layer with a convolution kernel size of 3×3 and a stride of 2 is inserted between each CSwin transformer block to reduce the image size and increase the number of channels.

[0032] Each CSwin transformer block consists of a cross-window self-attention module CSWin-Attention, a normalization layer LN and a multi-layer perceptron MLP. The output of the lth CSwin transformer block is in

[0033] Second,

[0034] An electronic product comprises a processor and a memory communicatively connected to the processor; the memory is used to store instructions; the processor is used to call the instructions stored in the memory to execute the remote sensing image segmentation method based on dual-path multi-scale feature aggregation described in the first aspect.

[0035] The present invention first uses a contextual feature extraction network with a transformer structure to model long-range spatial dependencies and extracts global context features at the same time. Then, a short-range spatial feature extraction network that integrates CNN and SelfAttention is used to extract static context and dynamic context features of the feature graph. Finally, channel space aggregation modules CSM1 and CSM2 composed of a spatial attention module and a channel attention module are used to adaptively aggregate feature information from two paths and different levels, selectively emphasize interdependent feature graphs, and enhance the feature interaction capability of the network.

[0036] The comparison results show that the method proposed in the present invention can obtain more coherent segmentation areas, smoother boundaries, and more accurate categories, and has achieved good results on multiple data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is an overall flow chart of the dual-path image segmentation model in a specific embodiment of the present invention;

[0038] Figure 2 It is the overall flow chart and local structure diagram of the context feature extraction network CSwin transformer in a specific embodiment of the present invention;

[0039] Figure 3 It is an overall flow chart and a local structure diagram of a short-range spatial feature extraction network in a specific embodiment of the present invention;

[0040] Figure 4 is a flow chart of a channel space aggregation module CSM1 in a specific embodiment of the present invention;

[0041] Figure 5 is a flow chart of a channel space aggregation module CSM1 in a specific embodiment of the present invention;

[0042] Figure 6 is a flow chart of a channel attention module and a spatial attention module in a specific embodiment of the present invention;

[0043] Figure 7 Comparison of visualization results of different methods. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with specific embodiments.

[0045] In order to more accurately extract complete and clear boundaries from remote sensing images, this embodiment provides a remote sensing image segmentation method based on dual-path multi-scale feature aggregation, the steps comprising:

[0046] S1. Obtain remote sensing images and divide them into training set and validation set;

[0047] S2. Constructing a dual-path image segmentation model: The dual-path image segmentation model includes a context feature extraction network CSwin transformer and a short-range spatial feature extraction network, and the feature maps obtained by different paths are fused;

[0048] S3, using the training set and the validation set to perform supervised training on the dual-path image segmentation model;

[0049] S4. Use the trained dual-path image segmentation model to segment the remote sensing image to be segmented.

[0050] Next, this embodiment will first introduce the dual-path image segmentation model constructed in this embodiment as a whole, and then introduce the context feature extraction network CSwin transformer, the short-range spatial feature extraction network, the channel space aggregation module CSM1, the channel space aggregation module CSM2 and the feature fusion module FFM in the model respectively.

[0051] 1. Overall structure of the model

[0052] The overall process of the dual-path image segmentation model constructed in this embodiment is as follows: Figure 1 As shown, in order to model the global semantic association and better establish long-distance spatial dependencies, this embodiment uses the Cswin transformer as the context feature extraction network of the proposed model, and uses self-attention to enhance the spatial correlation between pixels in the entire image, so as to more effectively obtain the global semantic context information and spatial context information of the remote sensing image.

[0053] We used C2block with convolution and improved self-attention modules for feature extraction in the short-range spatial feature extraction network. C2block is composed of a convolution layer Cov, a normalization layer BN, an excitation layer Relu, an improved self-attention module CoT (Contextual Transformerblock), and a normalization layer BN. C2block first uses convolution to extract local information, and then uses the improved self-attention module to capture static contextual information in the adjacent space, eliminating the noise of the low-resolution feature map while fully extracting the shape, boundary and other features of the image.

[0054] In addition, since the high-level abstract features extracted by the long context path and the low-level detail features extracted by the spatial path are different in spatial distribution and physical meaning, directly connecting high-level features with low-level features cannot eliminate the semantic gap between the two. Therefore, we propose two channel space aggregation modules CSM1 and CSM2 with channel attention and spatial attention to enhance the distinguishability of feature representation, reduce intra-class variance, increase inter-class variance, and adaptively aggregate the feature information extracted by the spatial path and context path. Finally, we combine the fused features Figure 1 The input feature fusion module FFM fuses features of different scales and outputs a feature map with detailed feature information and abstract feature information.

[0055] Specifically, the context feature extraction network CSwin transformer in S2 uses four stages to generate different hierarchical expressions. Each stage includes a convolutional token embedding layer and a CSwin transformer block. In the first convolutional token embedding layer, overlapping convolutions (with a kernel size of 7×7 and a stride of 4) are used to divide the patch. A convolutional token embedding layer with a kernel size of 3×3 and a stride of 2 is inserted between each CSwin transformer block. The image size is reduced and the number of channels is increased to reduce the amount of computation and expand the receptive field.

[0056] When we input an image of a given size [3, W, H], for example [3, 512, 512], the image is input into CSwin transformer and first passes through the first convolutional token embedding layer to obtain a feature map of size [48, 128, 128]. Then it is processed by four CSwin transformer blocks in sequence to obtain feature maps of sizes [64, 128, 128], [128, 64, 64], [256, 32, 32], and [512, 16, 16], respectively, marked as CF1, CF2, CF3, and CF4. The feature map CF4 is processed by the atrous spatial pyramid pooling (ASPP) module to obtain a feature map F1 of size [512, 16, 16].

[0057] The short side spatial information feature extraction network consists of four feature extraction modules (C2 blocks). Given an image of size [3,512,512], after being processed by different C2 blocks in sequence, feature maps of sizes [48,256,256], [96,128,128], [192,64,64], and [384,32,32] are obtained, which are marked as SF1, SF2, SF3, and SF4 respectively.

[0058] The feature maps CF1 and SF2 are processed by the channel space aggregation module CSM1 to obtain a feature map of size [64, 64, 64], marked as F2;

[0059] The feature maps CF2 and SF3 are processed by the channel space aggregation module CSM2 to obtain a feature map of size [128, 64, 64], marked as F3;

[0060] The feature maps CF3 and SF4 are processed by the channel space aggregation module CSM2 to obtain a feature map of size [256, 32, 32], marked as F4;

[0061] Finally, the feature fusion module FFM is used to aggregate the features of different levels obtained from the feature maps F1, F2, F3, and F4 to obtain the feature map F5. Finally, the feature map F5 outputs the segmentation result through the Find_cov module.

[0062] (II) Contextual Feature Extraction Network CSwin transformer

[0063] Convolution operations have the advantages of spatial locality and weight sharing, and can obtain local information well. However, due to the limited receptive field of convolution, CNN has difficulty in modeling long-distance dependencies and capturing global feature representations. Transformer can better capture long-distance interaction information. We use CSwintransformer as our context feature extraction network.

[0064] The overall structure of CSwintransformer is as follows Figure 2 As shown:

[0065] Convolutional Token Embedding (CVT, Swin) allows us to adjust the size of the input feature map by changing the parameters of the convolution operation. In the embodiment, we first input an image of a given size of 3×H×W (number of channels × width × height), and after convolutional token embedding (Convolutional Token Embeding) with a convolution kernel size of 7×7 and a stride of 4, we get a patch of size 4×4 and the number of H / 4×W / 4, and each patch is embedded in C1 dimensions for representation. After the first convolutional token embedding, the resulting feature map will pass through 4 CSwin transformerblocks in sequence. A convolutional token embedding with a convolution kernel size of 3×3 and a stride of 2 is used between adjacent CSwin transformers to reduce the width and height of the feature map, while increasing the number of channels of the feature map. In this way, at each stage we gradually reduce the token sequence length (the token sequence length) while increasing the token feature dimension (the token feature dimension). This allows the tokens to gradually increase the receptive field of the network like a CNN structure. After four different stages of CSwintransformer, feature maps CF1, CF2, CF3, and CF4 with sizes of [C1, H / 4, W / 4], [2C1, H / 8, W / 8], [4C1, H / 16, W / 16], and [8C1 H / 32, W / 32] are obtained respectively.

[0066] Each CSwin transformer block consists of a cross-shaped window self-attention module CSWin-Attention (Cross-Shaped Window Self-Attention, abbreviated as CSWin-Attention), a normalization layer LN (LayerNormalization, abbreviated as LN) and a multi-layer perceptron MLP according to Figure 2 The output of the lth CSwin transformer block is in

[0067] Transformer's global context modeling capability mainly comes from MultiheadedSelf-attention (MSA) [Vit, my], but MSA increases the complexity of the network while establishing long-distance spatial dependencies (the computational complexity is the quadratic of the feature map), which will face huge computational costs for remote sensing high-resolution image segmentation. To alleviate this problem, Swintransformer divides each feature map into multiple small Windows, performs Self-Attention within each Windows, and uses ShiftedWindows to expand the receptive field of the entire network, but more blocks still need to be stacked to increase the global receptive field. In order to more effectively implement global Self-Attention, we use the CSWSA module here. CSWSA uses horizontal and vertical stripes to implement Self-Attention. CSWSA divides multi-head into two parts, one for horizontal stripe self-attention and the other for vertical stripe self-attention, and then uses the concat splicing function to splice the output.

[0068] Specifically, let the input feature map be X∈R H×W×C , in horizontal self-attention, X is evenly divided into non-overlapping horizontal stripes [X 1 ,X 2 ,.....X M ], with a width of sw, and each horizontal stripe contains sw×W tokens, and M is H / sw. Assume that the dimensions of the projection Q, K, and V of the k-th head are all d k , then the horizontal stripe self-attention of the k-th head is defined as:

[0069]

[0070] Where X i ∈R sw×W×C , The projection matrix of Q, K, and V of the k-th head. Similarly, the vertical stripe self-attention of the k-th head is V_Attention k (X), and finally the two parallel groups of self-attention are connected together and output to obtain CSWSA, as shown in Formula 2:

[0071]

[0072] Among them, W O ∈RC×C is a commonly used projection matrix.

[0073] (III) Short-range spatial feature extraction network

[0074] As the network deepens, the network receptive field increases, and the local detail information of the high-level feature map will be lost. In order to extract more detailed spatial information, we use a short-range spatial feature extraction network that integrates local self-attention to extract the spatial detail information of the image. Figure 3 As shown, the short-range spatial feature extraction network consists of four C2blocks. In this embodiment, taking the input image of size [3, 512, 512] as an example, when passing through different C2blocks in sequence, feature maps SF1, SF2, SF3, and SF4 of sizes [48, 256, 256], [96, 128, 128], [192, 64, 64], and [384, 32, 32] are obtained respectively.

[0075] First, a CBR module with a convolution kernel size of 1×1 is used (Cov, BN and Relu operations are performed in sequence) to adjust the number of channels of the input feature map to obtain feature map C2F1; then C2F1 is passed through a CBR module with a convolution kernel size of 3×3 to obtain static context information to obtain feature map C2F1; then feature maps C2F1 and C2F2 are concatenated to obtain feature map C2F3; C2F3 then uses two continuous convolutions with a convolution kernel size of 1×1 to perform Cov, BN, Relu and Cov operations in sequence to obtain a dynamic attention matrix and obtain feature map C2F4;

[0076] After passing the feature map C2F1 through a CBR module with a convolution kernel size of 3×3 and a Softmax operation, the static context and local spatial information are obtained to obtain the feature map C2F5; finally, C2F5 is multiplied with the attention matrix C2F4 to obtain the dynamic context information C2F6 of the input feature map, and finally the dynamic context information C2F6 and the static context information C2F1 are fused and output to obtain the final result.

[0077] (IV) Feature Fusion Network

[0078] The feature information extracted by the context feature extraction network CSwintransformer and the short-range spatial feature extraction network at different resolutions will be different. In order to more effectively fuse these feature information, we use a feature aggregation network with an attention module to generate an attention matrix to selectively aggregate feature information of objects of different categories.

[0079] We explored the connection method of the attention module in the feature fusion module, and finally used two feature aggregation modules CSM1 and CSM2:

[0080] The specific structure of CSM1 is as follows Figure 4 As shown in the figure, in the CSM1 module, we first concatenate the two feature maps, and then redistribute the channel dimension of the concatenated feature maps through a two-dimensional convolution to obtain the feature map Fa.

[0081] Next, the feature map Fa passes through the spatial attention module and the channel attention module respectively to obtain attention matrices SPA and CHA with different weights. The spatial attention matrix SPA and the channel attention matrix CHA are fused, and finally connected with the feature map Fa with a residual structure to obtain the output feature map.

[0082] Specifically, the spatial attention module considers the relationship between pixels at different spatial locations and is able to capture the spatial dependencies between different locations of feature maps.

[0083] We assume that the input feature map is x m ∈(R C ) H×W , where m is the spatial pixel of the feature map. We use standard convolution with a convolution kernel size of 1×1 to obtain Formula 3:

[0084]

[0085] in Resh represents the reshape operation Reshape, where represents the channel embedding dimension. Parameters i and j are spatial position indices. We then use embeddedGaussian (a commonly used Gaussian function) to calculate spatial similarity, and then use Formula 4 to calculate the Spatial similarity of any spatial position. ij To rate,

[0086]

[0087] like Figure 4 As shown in the upper part, we use matrix multiplication between the feature map Fa and the Softmax function to calculate the spatial similarity score Spa ij , and all spatial similarity scores are collected to form the spatial similarity matrix SPA∈R H ×W×H×W , which can calculate the long-term interdependence of various types of objects in the spatial domain.

[0088] Channel attention can redistribute the weight information of feature maps and emphasize feature channels related to segmentation scenes.

[0089] We assume that the input feature map is x n ∈(R H×W ) C, where n is the channel of the feature map, we use the standard convolution with a convolution kernel size of 1×1, and we can get Formula 5:

[0090]

[0091] where γ(x p ),σ(x q )∈R H×W , Resh represents the Reshape operation, p and q represent different channel indices of the feature map. Then we use embedded Gaussian (a commonly used Gaussian function) to calculate the spatial similarity and channel similarity Ch pq Score as public

[0092] As shown in formula 6:

[0093]

[0094] We compute channel similarity scores and collect them to form a spatial similarity matrix CHA∈R C×C ,like Figure 4 This matrix can model the long-range interdependencies in different channels and redistribute the feature map weight information.

[0095] We assume two input feature maps {x1, x2∈R H×W×C}, where H, W, and C are the length, width, and number of channels of the input feature map, respectively.

[0096] In the CSM1 module, we obtain the feature map Fa after concatenation and standard convolution with a convolution kernel size of 1×1. The process is shown in Formula 7:

[0097] x a =Con(cat(x1, x2)) (7),

[0098] where x a ∈R H×W×C , Cov represents the convolution operation, and cat represents the concatenation operation. We input the obtained feature maps into the spatial attention and channel attention modules respectively to obtain the spatial attention matrix and channel attention matrix SPA and CHA. In order to fuse the spatial and channel attention, we use the feature map of size HWC to distribute the above two similarity matrices (SPA and CHA) to all spatial and channel positions, such as Figure 4 As shown, we perform a two-dimensional convolution on the feature map Fa and then perform a Reshape operation, which can be expressed as

[0099] f(x z )=Resh(Cov(x z )) (8),

[0100] Where f(x z )∈R H×W×C .

[0101] We will SPA, f(x z ) and CHA for matrix multiplication, and then perform residual connection with the feature map Fa to obtain the final output feature map, as shown in Formula 9:

[0102] F={(SPA×f(x z )×CHA),Fa} (9).

[0103] The specific structure of CSM2 is as follows Figure 5 As shown in the figure, in the CSM2 module, we pass the feature maps of different paths through a two-dimensional convolution to unify the channel dimension, and obtain feature maps FC1 and FC2 respectively. Then, feature maps FC1 and FC2 are passed through the spatial attention module and the channel attention module respectively, and the feature maps that finally redistribute the information weights are connected to feature maps FC1 and FC2 respectively with a residual structure to obtain feature maps FC3 and FC4. Then, the feature maps FC3 and FC4 are concatenated to fully integrate the feature information of different paths, and finally a two-dimensional convolution is performed to recalibrate the channel dimension and output the final result.

[0104] In the CSM2 module, we obtain two feature maps FC1 and FC2 after two standard convolutions with a convolution kernel size of 1×1. The process is shown in Formula 10:

[0105]

[0106] where x c1 ∈R H×W×C , x c2 ∈R H×W×C , Cov represents the convolution operation. We will get the feature map x c1 and x c2 The spatial attention module and the channel attention module are input respectively to obtain the spatial attention matrix SPA and the channel attention matrix CHA. Then, two feature maps of size H×W×C are used to redistribute the spatial attention matrix and the channel attention matrix to the input feature map. Then, the feature map of the redistributed attention matrix is ​​connected with the feature maps FC1 and FC2 with a residual structure to obtain the feature maps FC3 and FC4. The whole process can be expressed by the formula:

[0107]

[0108] Finally, the feature maps FC3 and FC4 are concatenated and then output through a two-dimensional convolution operation, which can be expressed as:

[0109] F=Cov(cat(FC3, FC4)) (12).

[0110] For the channel attention module, such as Figure 6 As shown in the figure, the feature map is first parallelized through maximum pooling and average pooling to obtain features, and then passes through a shared multi-layer perceptron model (Shared MLP) to obtain two weights. The two weights are added together and then passed through a sigmoid activation function to finally obtain the channel weight.

[0111] For the spatial branch, it aims to give different weights to each spatial position of the feature map, thereby enhancing the model's attention to important positions, such as Figure 6 As shown in the figure, the channel attention module first performs global average pooling and global maximum pooling on the input feature map in the channel dimension to obtain two feature maps, then concatenates the results of the two in the channel dimension, generates spatial attention weights through a convolutional layer and a sigmoid, and finally outputs the weighted output with the input feature map of the spatial attention.

[0112] The image is segmented using the method provided in this embodiment and several existing methods including PSPNet, DeeplabV3, CSwinTransformer, etc. The visualization results of different experimental methods are as follows: Figure 7 As shown. It can be seen that our method ( Figure 7 The CSwin transformer (marked as Ours in the figure) can obtain more coherent segmented areas, smoother boundaries, and more accurate categories. For example, in the lower left corner of the third row, our method can better distinguish between buildings and roads, while the CSwin transformer incorrectly classifies the roof as a plant. From the results in the first row, we can see that our method obtains significantly finer boundaries and has a higher degree of distinction between different types of objects.

Claims

1. A remote sensing image segmentation method based on dual-path multi-scale feature aggregation, It is characterized in that the steps include: S1. Obtain remote sensing images and divide them into training set and validation set; S2. Constructing a dual-path image segmentation model: The dual-path image segmentation model includes a context feature extraction network CSwin transformer and a short-range spatial feature extraction network, and the feature maps obtained by different paths are fused; S3, using the training set and the validation set to perform supervised training on the dual-path image segmentation model; S4, using the trained dual-path image segmentation model to segment the remote sensing image to be segmented; Among them, the context feature extraction network CSwin transformer in S2 uses four stages to generate different hierarchical expressions; the image is input into CSwin transformer and processed in four stages in turn to obtain feature maps CF 1 CF 2 CF 3 CF 4 ; Feature map CF 4 The feature map F1 is obtained by processing with the atrous spatial pyramid pooling module ASPP; The short-range spatial feature extraction network consists of four C2 blocks with convolution and improved self-attention modules. After the image is processed by different C2 blocks in turn, the feature maps SF are obtained respectively. 1 , SF 2 , SF 3 , SF 4 ; Feature Map CF 1 With SF 2 The feature map F2 is obtained after being processed by the channel space aggregation module; Feature Map CF 2 With SF 3 After being processed by the channel space aggregation module, the feature map F3 is obtained; Feature Map CF 3 With SF 4 After being processed by the channel space aggregation module, the feature map F4 is obtained; Finally, the feature fusion module FFM is used to fuse the features of different levels obtained from the feature maps F1, F2, F3, and F4 to obtain the feature map F5. Finally, the feature map F5 outputs the segmentation result through the Find_cov module.

2. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 1, It is characterized in that The specific structure of the C2 block with convolution and improved self-attention module is: First, use a CBR module with a convolution kernel size of 1×1 (perform Cov, BN, and Relu operations in sequence) to adjust the number of channels of the input feature map to obtain the feature map C2F 1 ; Then use C2F 1 The static context information is obtained through the CBR module with a convolution kernel size of 3×3 to obtain the feature map C2F 1 ; Then the feature map C2F 1 and C2F 2 Perform concatenation operation to obtain feature map C2F 3 ;C2F 3 Then use two continuous convolutions with a convolution kernel size of 1×1 to perform Cov, BN, Relu and Cov operations in sequence to obtain the dynamic attention matrix and obtain the feature map C2F 4 ; The feature map C2F 1 After a CBR module with a convolution kernel size of 3×3 and a Softmax operation, static context and local spatial information are obtained to obtain the feature map C2F 5 ; Finally, C2F 5 Same attention matrix C2F 4 Multiply the output to get the dynamic context information C2F of the input feature map 6 Finally, the dynamic context information C2F 6 and static context information C2F 1 The fusion outputs are then used to obtain the final result.

3. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 1, It is characterized in that Feature Map CF 1 With SF 2 After being processed by the channel space aggregation module CSM1, the feature map F2 is obtained; In the CSM1 module, two feature maps from different paths are first concatenated, and then the concatenated feature maps are reassigned to channel dimensions through a two-dimensional convolution to obtain the feature map Fa. Then, the feature map Fa is passed through the spatial attention module and the channel attention module respectively to obtain the spatial attention matrix SPA and the channel attention matrix CHA with different weights respectively; The spatial attention matrix SPA and the channel attention matrix CHA are fused and then connected with the feature map Fa with a residual structure to obtain the output feature map.

4. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 1, It is characterized in that Feature Map CF 2 With SF 3 , feature map CF 3 With SF 4 They are processed by the channel space aggregation module CSM2 for feature fusion; In the CSM2 module, two feature maps from different paths are first made to unify the channel dimensions through a two-dimensional convolution to obtain feature maps FC1 and FC2 respectively; Then, the feature maps FC1 and FC2 are redistributed with information weights through the spatial attention module and the channel attention module respectively, and then the feature maps with redistributed information weights are connected with the feature maps FC1 and FC2 respectively with a residual structure to obtain feature maps FC3 and FC4; Then, the feature maps FC3 and FC4 are concatenated to fully fuse the feature information of different paths, and then further undergo a two-dimensional convolution to recalibrate the channel dimension and output the result.

5. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 3 or 4, It is characterized in that The channel attention module first obtains the features of the feature map through maximum pooling and average pooling in parallel, and then passes it through a shared multi-layer perceptron model (Shared MLP) to obtain two weights. The two weights are added together and then passed through a sigmoid activation function to finally obtain the channel weight.

6. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 3 or 4, It is characterized in that The spatial attention module first performs global average pooling and global maximum pooling on the input feature map in the channel dimension to obtain two feature maps, then concatenates the results of the two in the channel dimension, generates spatial attention weights through a convolutional layer and a sigmoid, and finally outputs the weighted output with the input feature map of the spatial attention.

7. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 1, It is characterized in that In S2, each stage of the context feature extraction network includes a convolutional token embedding layer and a CSwin transformer block. The convolutional token embedding layer of the first stage uses overlapping convolutions with a convolution kernel size of 7×7 and a stride of 4 to divide the blocks. A convolutional token embedding layer with a convolution kernel size of 3×3 and a stride of 2 is inserted between each CSwin transformer block to reduce the image size and increase the number of channels.

8. The remote sensing image segmentation method based on dual-path multi-scale feature aggregation as claimed in claim 7, It is characterized in that Each CSwin transformer block consists of a cross-window self-attention module CSWin-Attention, a normalization layer LN and a multi-layer perceptron MLP. The output of the l-th CSwin transformer block is in 9. An electronic product, comprising a processor and a memory communicatively connected to the processor; the memory is used to store instructions; the processor is used to call the instructions stored in the memory to execute the remote sensing image segmentation method based on dual-path multi-scale feature aggregation as described in any one of claims 1-8.

Images