Remote sensing image segmentation method based on channel enhancement and cross-level multi-input features

Abstract

The invention discloses a remote sensing image segmentation method based on enhanced and cross-level multi-input features. The method comprises the following steps: firstly, obtaining a high-resolution remote sensing image training sample set; the method comprises the following steps: designing a Swin Transform encoder based on channel enhancement, a cross-level multi-path input decoder and a feature refining module based on fractional order differential, and constructing a pixel-level semantic segmentation model; then, based on the high-resolution remote sensing image training sample set, training the pixel-level semantic segmentation model to obtain a trained pixel-level semantic segmentation model; and finally, based on the trained pixel semantic segmentation model, performing semantic segmentation on a to-be-processed high-resolution remote sensing image to obtain a semantic segmentation result of the to-be-processed high-resolution remote sensing image. According to the method provided by the invention, the problems of fuzzy boundary, small target loss and insufficient information processing during semantic segmentation of the high-resolution remote sensing image are effectively improved, and the precision of a pixel-level semantic segmentation model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features. Background Art

[0002] In the study of semantic segmentation of high-resolution remote sensing images, the main goal is to develop methods that can effectively process ultra-high resolution images, especially under limited GPU memory, to provide high-quality segmentation results. This requires solving some problems existing in current methods, such as unclear boundaries, loss of small objects, and the complexity and time-consuming training process. In addition, these methods should make full use of the ultra-high resolution characteristics of remote sensing images while maintaining segmentation accuracy, such as effectively capturing and utilizing global context information and local detail information. At present, the mainstream semantic segmentation network usually adopts an encoder-decoder based structure, and its specific process is shown in the equation: In the formula, E(X) is the encoder, which is used to extract features from the source image X, and the extracted features are F; D(F) is the decoder, which is used to decode the extracted features F and reconstruct the segmented image corresponding to the source image.

[0003] In the encoder part, the current mainstream method uses SwimTransformer for feature extraction to obtain richer feature maps. However, in the process of processing feature maps, the calculations between the various channels of SwimTransformer are independent of each other, which means that the correlation information between channels is ignored. This independent calculation method limits its ability to extract and utilize the interactive information between channels. Therefore, although SwimTransformer performs well in feature extraction on a single channel, it still has certain limitations in overall inter-channel feature fusion and information complementarity. In the decoder part, compared with traditional machine learning methods, methods based on convolutional neural networks (CNNs) can capture more refined local context information, which provides great potential for feature representation and pattern recognition. However, the convolution operation with a fixed receptive field is mainly used to extract local features and lacks the ability to model global context information or long-distance dependencies. In semantic segmentation, modeling only local information usually leads to ambiguity in global pixel-level classification, because the semantic content of each pixel becomes more accurate with the help of global context information. In addition, the current mainstream method uses a mixed stacking convolution method of multiple features in the convolution process to extract the complex nonlinear association between features and targets as much as possible, but this undoubtedly increases the computational complexity of the network. In traditional convolutional templates, integer-order differential operators are usually suitable for processing high-frequency components in images, but have poor processing capabilities for details of discontinuous boundary points and low-frequency variation characteristics. The rich surface texture details in images belong to medium and low-frequency components, which are difficult to be effectively processed by integer-order differential operators.

[0004] Therefore, how to effectively process the semantic segmentation of ultra-high resolution remote sensing images and provide high-quality segmentation results has become an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features, so as to improve the accuracy of the pixel-level semantic segmentation model and improve the problems of blurred boundaries, loss of small targets and insufficient information processing in the semantic segmentation of high-resolution remote sensing images.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A remote sensing image segmentation method based on channel enhancement and cross-level multi-input features comprises the following steps:

[0008] Obtain high-resolution remote sensing image training sample sets;

[0009] Design a Swin Transformer encoder based on channel enhancement, a cross-level multi-path input decoder, and a feature refinement module based on fractional differentiation, and introduce a cosine annealing strategy and a composite training loss function to build a pixel-level semantic segmentation model;

[0010] Based on the high-resolution remote sensing image training sample set, the pixel-level semantic segmentation model is trained to obtain a trained pixel-level semantic segmentation model;

[0011] Based on the trained pixel semantic segmentation model, semantic segmentation is performed on the high-resolution remote sensing image to be processed to obtain a semantic segmentation result of the high-resolution remote sensing image to be processed.

[0012] As a preferred solution, the pixel-level semantic segmentation model includes four cascaded channel-enhanced SwinTransformer encoders, four cross-level multi-path input decoders, and one feature refinement module based on fractional differentiation.

[0013] The pixel-level semantic segmentation model uses the input high-resolution remote sensing image as the input of the first SwinTransformer encoder; the output of each Swin Transformer encoder is used as the input of the next one in the pixel semantic segmentation model; the output of the fourth Swin Transformer encoder is used as the input of the fourth cross-level multi-path input decoder; the input of each cross-level multi-path input decoder is the output of a corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model; the output of the first cross-level multi-path decoder is used as the input of a feature refinement module based on fractional-order differentiation, and the output of the feature refinement module is used as the overall output of the pixel-level semantic segmentation model.

[0014] As a preferred solution, the channel-enhanced Swin Transformer encoder includes a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet downsampling module, and a connected convolution unit; the parallel connection result of the average pooling layer and the maximum pooling layer is cascaded with the wavelet downsampling module, and the parallel connection result of the Swin transformer block and the wavelet downsampling module is cascaded with the connected convolution unit;

[0015] The inputs of the channel-enhanced Swin Transformer encoder are respectively used as the inputs of the Swin transformer block, the maximum pooling layer and the average pooling layer. The outputs of the maximum pooling layer and the average pooling layer are multiplied and used as the inputs of the wavelet downsampling module. The outputs of the wavelet downsampling module and the outputs of the Swin transformer block are used as the inputs of the connected convolution unit. The outputs of the connected convolution unit are used as the outputs of the channel-enhanced Swin Transformer encoder.

[0016] As a preferred solution, the processing process of the channel-enhanced Swin Transformer encoder is shown in the following formula:

[0017] Y=Concat(F SwinB ,F HWD );

[0018] F HWD =HWD(AvgPool(X)⊙MaxPool(X));

[0019] Where Y is the feature map output by the channel-enhanced Swin Transformer encoder, AvgPool(.) is the average pooling layer, MaxPool(.) is the maximum pooling layer, HWD(.) is the downsampling operation after element-wise multiplication of the average pooling layer output and the maximum pooling layer output, and F HWD is the output feature map obtained after downsampling, F SwinB is the feature map obtained by the SwinTransformer encoder after processing the input X, and Concat(.) is the connection operation.

[0020] As a preferred solution, the cross-level multi-path input decoder includes a local branch, a global branch and a multi-path feature fusion module connected in parallel, wherein the parallel connection result of the local branch and the global branch is cascaded with the multi-path feature fusion module;

[0021] Among them, the high-resolution remote sensing image is input into the Swin Transformer encoder based on channel enhancement, and the output is the underlying spatial detail feature map and the high-level semantic feature map, which are respectively input into the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced ​​with the output of the global branch as the input of the multi-path feature fusion module, and the output of the multi-path feature fusion module is used as the overall output of the cross-level multi-path input decoder.

[0022] As a preferred solution, the local branch includes a cascaded 1×1 convolution layer, a pyramid pooling layer, three parallel convolution layers, a batch normalization layer and an erosion expansion unit; the three parallel convolution layers are respectively a 1×1 convolution layer, a 3×3 convolution layer and a fractional-order convolution layer; wherein the input of the local branch is used as the input of the 1×1 convolution layer, the output of the 1×1 convolution layer is used as the input of the pyramid pooling layer, the output of the pyramid pooling layer is used as the input of the three parallel convolution layers, the outputs of the three parallel convolution layers are spliced ​​and used as the input of the batch normalization layer, the output of the batch normalization layer is used as the input of the erosion expansion unit, and the output of the erosion expansion unit is used as the output of the local branch;

[0023] The global branch adopts a self-attention mechanism, inputs the input feature map into the Patch Embedding layer to generate a Query, Key and Value matrix, performs a dot product operation on the Query and Key, and outputs an attention score; then, a bias term is added to the dot product result of the Query and Key and a Softmax function is applied to convert the attention score into a standardized probability distribution; finally, the standardized probability distribution and the Key are weighted averaged to obtain a global branch feature map as the output of the global branch;

[0024] The multi-channel feature fusion module includes a cascaded Sum layer, a depth-separable convolution layer, a batch normalization layer and a 1×1 convolution layer; wherein the input local branch and global branch output feature maps are used as inputs of the Sum layer to perform an addition operation on the input feature map in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is output; then, the combined feature map is used as input of the depth-separable convolution layer to further extract and integrate local feature information and global feature information; finally, the output of the depth-separable convolution layer is further compressed and the feature dimension is integrated through the batch normalization layer and the 1×1 convolution layer to generate the output feature map of the multi-channel feature fusion module.

[0025] As a preferred solution, the feature refinement module based on fractional differential includes a cascaded channel path, a spatial path, a convolution activation unit, a 1×1 convolution layer and an upsampling module; the result of the channel path and the spatial path in parallel is cascaded with the convolution activation unit, the channel path includes a cascaded global average pooling layer, a fully connected layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth-separable convolution layer and a Sigmoid activation function layer, the convolution activation unit includes a cascaded 3×3 convolution layer, a batch normalization layer and a ReLU activation function layer; wherein the input feature map is used as the input of the channel path and the spatial path respectively to generate the channel attention weight and the spatial attention weight, which are multiplied with the input feature map to fuse the output feature map; then, the fused feature map is processed by the convolution activation unit; then, the input feature map is reduced in dimension by a 1×1 convolution layer and added to the fused feature map to realize the residual connection; finally, the feature map after the addition operation is used as the input of the upsampling module to obtain the semantic segmentation map as the output.

[0026] As a preferred solution, in the process of training the pixel-level semantic segmentation model, a composite training loss is calculated based on the semantic segmentation results of the high-resolution remote sensing image training samples by the pixel-level semantic segmentation model and the true semantic category labels of the corresponding high-resolution remote sensing image training samples, and the parameters of the pixel-level semantic segmentation model are optimized and updated with the goal of minimizing the composite training loss function. The learning rate of the optimized update parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

[0027] As a preferred solution, the cosine annealing strategy is shown in the following formula:

[0028]

[0029] Where η t is the learning rate of the tth training batch, T is the total training batches, and η is the initial learning rate.

[0030] 10. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that the function expression of the composite training loss is as follows:

[0031]

[0032] In the formula, is the composite training loss function, is the main loss function, is the auxiliary loss function, α is the auxiliary loss weight factor, is the Dice loss function, is the cross entropy loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample types contained in the high-resolution remote sensing image training sample set, respectively. represents the true semantic category label of the kth high-resolution remote sensing image training sample of the nth sample category, Represents the semantic segmentation result of the pixel-level semantic segmentation model for the kth high-resolution remote sensing image training sample of the nth sample category.

[0033] Compared with the prior art, the present invention has the following technical effects:

[0034] (1) The present invention introduces a channel enhancement mechanism based on the Swin Transformer encoder, uses a pooling operation to integrate the information of multiple adjacent channels, and combines the global average pooling and global maximum pooling layers between channels, so that the output feature map can better retain the relationship characteristics between channels, thereby enhancing the network's ability to capture detailed information and significantly improving the overall feature representation capability.

[0035] (2) The cross-level multi-path input decoder proposed in the present invention integrates feature information at different levels to effectively process deep semantics and detailed spatial detail information. By processing global information and local information in parallel, it ensures the edge smoothness and overall consistency of the segmentation result, thereby solving the problem of processing complex structures and diverse spectral information in high-resolution images. The feature map of global information and local information is processed by combining bilinear interpolation sampling and convolution through a multi-path feature fusion module to make them consistent in spatial resolution. The local information and global information are effectively fused through adaptive weights, which enhances the network's recognition ability for objects of different shapes and sizes and improves the generalization of the decoder. In the feature fusion process, dynamic convolution is introduced. In the depthwise separable convolution, the input feature map is first deep convolved, and then the result of the deep convolution is convolved point by point using dynamic convolution to achieve adaptive fusion of features, which helps to better adapt to targets of different shapes and sizes in the feature map, and further improves the model's adaptability and segmentation accuracy in complex scenes.

[0036] (3) The present invention uses a feature refinement module based on feature decoding of fractional order differentials to capture high-level features in images using fractional order differential operators, especially when processing edge and texture information of images. Compared with traditional feature extraction techniques, fractional order differentials can more effectively reveal subtle structural changes in images, provide richer and more detailed information for the model, and thus support more accurate image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0038] Figure 1 The present invention discloses a flow chart of a remote sensing image segmentation method based on enhanced and cross-level multi-input features;

[0039] Figure 2 This is a network structure diagram of the pixel-level semantic segmentation model of the present invention;

[0040] Figure 3 The network structure diagram of the Swin Transformer encoder based on channel enhancement of the present invention;

[0041] Figure 4 This is a network structure diagram of a cross-level multi-path input decoder of the present invention;

[0042] Figure 5 Schematic diagram of the fractional-order convolution template of the present invention;

[0043] Figure 6 Schematic diagram of the fractional-order convolution calculation process. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0046] In the study of semantic segmentation of high-resolution remote sensing images, the main goal is to develop methods that can effectively process ultra-high resolution images, especially to provide high-quality segmentation results under limited GPU memory. However, existing technologies have some problems in implementation, such as unclear boundaries, loss of small objects, and complexity and time-consuming training process. In addition, how to fully utilize the ultra-high resolution features of remote sensing images while maintaining segmentation accuracy, such as how to effectively capture and utilize global context information and local detail information, etc., limits the practical application of semantic segmentation of high-resolution remote sensing images.

[0047] In order to overcome this limitation, the present invention proposes a remote sensing image segmentation method based on enhanced and cross-level multi-input features. The method process is as follows: Figure 1 As shown, the specific steps include:

[0048] Obtain high-resolution remote sensing image training sample sets;

[0049] Design a Swin Transformer encoder based on channel enhancement, a cross-level multi-path input decoder, and a feature refinement module based on fractional differentiation to build a pixel-level semantic segmentation model;

[0050] Based on the high-resolution remote sensing image training sample set, the pixel-level semantic segmentation model is trained to obtain a trained pixel-level semantic segmentation model;

[0051] Based on the trained pixel semantic segmentation model, semantic segmentation is performed on the high-resolution remote sensing image to be processed to obtain a semantic segmentation result of the high-resolution remote sensing image to be processed.

[0052] Through the above-mentioned process of the present invention, it can be seen that in the encoder part of the present invention, the current mainstream method uses SwimTransformer for feature extraction to obtain richer feature maps. However, in the process of processing feature maps, the calculations between the various channels of SwimTransformer are independent of each other, which means that the correlation information between channels is ignored. This independent calculation method limits its ability to extract and utilize the interactive information between channels. It can be seen that although SwimTransformer has excellent feature extraction performance on a single channel, it still has certain limitations in overall channel feature fusion and information complementarity. Therefore, the present invention introduces a channel enhancement mechanism on the basis of the prior art, so that the output feature map can better retain the relationship characteristics between channels, thereby enhancing the network's ability to capture detail information and significantly improving the overall feature representation ability. In the decoder part, although the current method based on convolutional neural networks can capture more refined local context information, the convolution operation of the fixed receptive field is mainly used to extract local features and lacks the ability to model global context information or long-distance dependencies. Therefore, the present invention introduces a self-attention mechanism to add global context information to local features to alleviate the above problems. In addition, the current mainstream method uses a mixed stacking convolution method of multiple features in the convolution process to increase the computational complexity of the network. The present invention proposes a simple convolution combination based on fractional-order convolution to extract nonlinear relationships as much as possible. In addition, in traditional convolution templates, integer-order differential operators are usually suitable for processing high-frequency changing components in images, but have poor processing capabilities for details of discontinuous boundary points and low-frequency changing characteristics. Therefore, the present invention uses fractional-order differential operators with orders between 0 and 1 to process images, which can better process noise or edge information, enhance texture details in images, and retain smooth area information.

[0053] In general, the present invention proposes a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features in the existing high-resolution remote sensing image semantic segmentation method. On the one hand, by introducing the channel enhancement mechanism on the basis of the Swin Transformer encoder, the extraction and utilization capabilities of channel correlation information are enhanced, thereby improving the overall feature representation capability. On the other hand, the encoder based on the Swin Transformer and the multi-head self-attention mechanism are combined with a lightweight decoder design, so that the model has significant computational efficiency while ensuring high precision, which is suitable for efficient processing requirements in actual industrial applications.

[0054] In order to better introduce the technical solution of the present invention, a more detailed description is given in the following parts.

[0055] 1. Image preprocessing

[0056] In the specific implementation application, the high-resolution remote sensing images used for training and testing the pixel semantic segmentation model can be selected using the public Potsdam dataset as a high-resolution remote sensing image training sample set for training the pixel-level semantic segmentation model. After selecting the high-resolution remote sensing image training samples from the public Potsdam dataset, the high-resolution remote sensing image training samples are cropped, and the cropped dataset is subjected to data enhancement processing. A portion of the enhanced high-resolution remote sensing image training samples are selected using the RGB color space and used as a test set, and the remaining enhanced high-resolution remote sensing image training samples are used as a training set. For example, the public Potsdam dataset contains 38 high-resolution remote sensing images, each of which is 6000x6000 in size. We use samples in the RGB color space, and select 14 images for testing, and the remaining images are used for training.

[0057] During the experiment, we cropped the original images into blocks of 1024x1024 size and enhanced the dataset by using random angle rotation or horizontal flipping to improve the training effect.

[0058] 2. Pixel-level semantic segmentation model

[0059] like Figure 2 As shown, the pixel-level semantic segmentation model proposed by the method of the present invention includes four Swin Transformer encoders based on channel enhancement, four cross-level multi-path input decoders and one feature refinement module based on fractional-order differentials that are cascaded in sequence; the pixel-level semantic segmentation model uses the input high-resolution remote sensing image as the input of the first Swin Transformer encoder; the output of each Swin Transformer encoder is used as the input of the next one connected in the pixel semantic segmentation model; the output of the fourth Swin Transformer encoder is used as the input of the fourth cross-level multi-path input decoder; the input of each cross-level multi-path input decoder is the output of a corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model; the output of the first cross-level multi-path decoder is used as the input of the feature refinement module based on fractional-order differentials, and the output of the feature refinement module is used as the overall output of the pixel-level semantic segmentation model.

[0060] Next, we will explain the various parts of the pixel-level semantic segmentation model.

[0061] 2.1. Swin Transformer encoder based on channel enhancement

[0062] like Figure 3As shown, the channel-enhanced Swin Transformer encoder includes a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet downsampling module, and a connected convolution unit; the parallel connection result of the average pooling layer and the maximum pooling layer is cascaded with the wavelet downsampling module, and the parallel connection result of the Swin transformer block and the wavelet downsampling module is cascaded with the connected convolution unit;

[0063] The inputs of the channel-enhanced Swin Transformer encoder are respectively used as the inputs of the Swin transformer block, the maximum pooling layer and the average pooling layer. The outputs of the maximum pooling layer and the average pooling layer are multiplied and used as the inputs of the wavelet downsampling module. The outputs of the wavelet downsampling module and the outputs of the Swin transformer block are used as the inputs of the connected convolution unit. The outputs of the connected convolution unit are used as the outputs of the channel-enhanced Swin Transformer encoder.

[0064] It can be seen that the present invention uses a Swin-Base-based encoder in the specific implementation application because the Transformer-based encoder performs outstandingly in the pursuit of high precision. However, SwinB only calculates windows and sliding windows at the feature map level, and naturally ignores the relationship between channels, resulting in the loss of some channel information in the SwinB-based encoder. Therefore, we perform channel enhancement on the basis of SwinB, so that the feature map output by SwinB can retain more channel relationship features, and use pooling to integrate the information of multiple adjacent channels. A channel-based global average pooling and global maximum pooling layer is used to enhance the co-occurring features while retaining the overall representation of the channel. Finally, the integrated features are passed through a wavelet downsampling layer to reduce the spatial resolution while retaining more information.

[0065] The processing of the Swin Transformer encoder based on channel enhancement is shown in the following formula:

[0066] Y=Concat(F SwinB ,F HWD );

[0067] F HWD =HWD(AvgPool(X)⊙MaxPool(X));

[0068] Where Y is the feature map output by the channel-enhanced Swin Transformer encoder, AvgPool(.) is the average pooling layer, MaxPool(.) is the maximum pooling layer, HWD(.) is the downsampling operation after element-wise multiplication of the average pooling layer output and the maximum pooling layer output, and F HWDis the output feature map obtained after downsampling, F SwinB is the feature map obtained by the SwinTransformer encoder after processing the input X, and Concat(.) is the connection operation.

[0069] 2.2 Cross-stage multi-path input decoder

[0070] like Figure 4 As shown, the cross-level multi-path input decoder includes a local branch, a global branch and a multi-path feature fusion module connected in parallel, wherein the parallel connection result of the local branch and the global branch and the multi-path feature fusion module are cascaded;

[0071] The local branch includes a cascaded 1×1 convolution layer, a pyramid pooling layer, three parallel convolution layers, a batch normalization layer and an erosion expansion unit; the three parallel convolution layers are respectively a 1×1 convolution layer, a 3×3 convolution layer and a fractional-order convolution layer; wherein the input of the local branch is used as the input of the 1×1 convolution layer, the output of the 1×1 convolution layer is used as the input of the pyramid pooling layer, the output of the pyramid pooling layer is used as the input of the three parallel convolution layers, the outputs of the three parallel convolution layers are spliced ​​and used as the input of the batch normalization layer, the output of the batch normalization layer is used as the input of the erosion expansion unit, and the output of the erosion expansion unit is used as the output of the local branch;

[0072] The global branch adopts a self-attention mechanism, inputs the input feature map into the Patch Embedding layer to generate a Query, Key and Value matrix, performs a dot product operation on the Query and Key, and outputs an attention score; then, a bias term is added to the dot product result of the Query and Key and a Softmax function is applied to convert the attention score into a standardized probability distribution; finally, the standardized probability distribution and the Key are weighted averaged to obtain a global branch feature map as the output of the global branch;

[0073] The multi-channel feature fusion module includes a cascaded Sum layer, a depth-separable convolution layer, a batch normalization layer and a 1×1 convolution layer; wherein the input local branch and global branch output feature maps are used as inputs of the Sum layer to perform an addition operation on the input feature map in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is output; then, the combined feature map is used as input of the depth-separable convolution layer to further extract and integrate local feature information and global feature information; finally, the output of the depth-separable convolution layer is further compressed and the feature dimension is integrated through the batch normalization layer and the 1×1 convolution layer to generate the output feature map of the multi-channel feature fusion module.

[0074] Among them, the high-resolution remote sensing image is input into the Swin Transformer encoder based on channel enhancement, and the output is the underlying spatial detail feature map and the high-level semantic feature map, which are respectively input into the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced ​​with the output of the global branch as the input of the multi-path feature fusion module, and the output of the multi-path feature fusion module is used as the overall output of the cross-level multi-path input decoder.

[0075] It can be seen that the present invention integrates the output of the decoder with the cross-layer connection of the encoder, and fuses the two inputs after respectively extracting their respective semantic information. By integrating feature information at different levels, the deep semantics and detail space detail information are effectively processed. By processing global information and local information in parallel, the difficulty in simultaneously capturing and utilizing the global context information and local detail information of the image during segmentation is avoided, and the edge smoothness and overall consistency of the segmentation results are ensured, thereby solving the problem of processing complex structures and diverse spectral information in high-resolution images.

[0076] Through the multi-path feature fusion module, the feature maps of global information and local information are processed by combining bilinear interpolation sampling and convolution to make them consistent in spatial resolution, and the local information and global information are effectively fused through adaptive weights, which enhances the network's recognition ability for objects of different shapes and sizes and improves the generalization of the decoder. In the feature fusion process, dynamic convolution is introduced. In the depthwise separable convolution, the input feature map is first deep convolved, and then the result of the deep convolution is convolved point by point using dynamic convolution to achieve adaptive fusion of features, which helps to better adapt to targets of different shapes and sizes in the feature map, and further improves the adaptability and segmentation accuracy of the model in complex scenes.

[0077] 2.3. Feature refinement module based on fractional differentiation

[0078] The feature refinement module based on fractional differential includes a cascaded channel path, a spatial path, a convolutional activation unit, a 1×1 convolutional layer and an upsampling module; the result of the channel path and the spatial path being connected in parallel is cascaded with the convolutional activation unit, the channel path includes a cascaded global average pooling layer, a fully connected layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth-separable convolutional layer and a Sigmoid activation function layer, and the convolutional activation unit includes a cascaded 3×3 convolutional layer, a batch normalization layer and a ReLU activation function layer; wherein the input feature map is used as the input of the channel path and the spatial path respectively to generate a channel attention weight and a spatial attention weight, and the channel attention weight and the spatial attention weight are multiplied with the input feature map to fuse the output feature map; then, the fused feature map is processed by the convolutional activation unit; then, the input feature map is reduced in dimension by a 1×1 convolutional layer and then added to the fused feature map to realize residual connection; finally, the feature map after the addition operation is used as the input of the upsampling module to obtain a semantic segmentation map as the output.

[0079] The present invention uses fractional differential operators to capture high-level features in images, extracts the first n coefficients from the approximate expression of fractional derivatives as parameters to construct a fractional convolution mask, and generates fractional convolution templates in eight directions diverging from the center. The approximate expression of the fractional derivative is as follows:

[0080]

[0081] In the formula, is the fractional derivative of function f(t) with respect to t, v is the order of the fractional derivative, v is a non-integer, Γ(.) is the gamma function, n is the upper limit of the number of terms in the approximate expression, and n! is the factorial of n;

[0082] Take the first 4 items as parameters to construct a 7×7 convolution template, give a learnable initial convolution fractional order parameter v0, and learn with the network back propagation. The constructed convolution template is as follows Figure 5 The calculation process is as shown in Figure 6 As shown, the template parameter calculation method is as follows:

[0083]

[0084] Where W 0 , W 1 , W 2 , W 3 , W 4 Both represent the convolution template weight coefficient.

[0085] The generated convolution template is used to perform sliding convolution on the feature map to obtain a feature refinement map guided by fractional features. This helps improve the image segmentation model's ability to capture details, especially when processing image edge and texture information. Compared with traditional feature extraction techniques, fractional differentials can more effectively reveal subtle structural changes in images, provide the model with richer and more detailed information, and thus support more accurate image segmentation.

[0086] 3. Training of pixel-level semantic segmentation model

[0087] In a specific implementation, the pixel-level semantic segmentation model is trained in the following manner: the divided training set is used as the input of the pixel-level semantic segmentation model, and a loss function is constructed to improve the model's sensitivity to different types of errors. In the present invention, a composite training loss is calculated based on the semantic segmentation results of the high-resolution remote sensing image training samples by the pixel-level semantic segmentation model and the true semantic category labels of the corresponding high-resolution remote sensing image training samples. The parameters of the pixel-level semantic segmentation model are optimized and updated with the goal of minimizing the composite training loss function, and the learning rate of the optimized update parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

[0088] In the application of the present invention, the cosine annealing strategy is adopted, and the AdamW optimizer is used for gradient descent and parameter optimization update. The initial learning rate is set to 6×10 -4 , the weight decay rate is set to 2.5×10 -4 , the specific formula is shown as follows:

[0089]

[0090] In the formula, in the formula, η t is the learning rate of the tth training batch, T is the total training batches, and η is the initial learning rate.

[0091] In the application of the present invention, the following composite training loss function is preferably used:

[0092]

[0093] In the formula, is the composite training loss function, is the main loss function, is the auxiliary loss function, α is the auxiliary loss weight factor, is the Dice loss function, is the cross entropy loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample types contained in the high-resolution remote sensing image training sample set, respectively. They represent the true semantic category labels of the k-th high-resolution remote sensing image training samples of the n-th sample category, represents the semantic segmentation result of the pixel-level semantic segmentation model for the kth high-resolution remote sensing image training sample of the nth sample type, n∈[1,…,N], k∈[1,…,K]; the preferred value of the auxiliary loss weight factor α can be set to 0.4.

[0094] It can be seen that the main losses is the Dice loss and Cross Entropy Loss Combination of auxiliary losses A cross entropy loss is used and applied to the auxiliary head. The auxiliary head takes the fused features of the three channel-enhanced Transformer blocks as input and constructs a 3×3 convolution layer with batch normalization and ReLU, a 1×1 convolution layer, and an upsampling operation to generate the output.

[0095] 4. Evaluation experiment of pixel-level semantic segmentation model

[0096] 4.1 Evaluation Metrics

[0097] In the specific implementation, three objective evaluation indicators, mAcc, mF1 and mIoU, are used to comprehensively judge the model effect.

[0098] Specifically, mAcc measures the average classification accuracy of the model in each category. First, the classification accuracy of the model in each class is calculated, and then the average is taken for all classes. For each class i, the accuracy is defined as:

[0099] Then the average classification accuracy is:

[0100] Where Acc i is the classification accuracy on each class, TPi is the number of true positive examples of class i, FPi is the number of false positive examples of class i, and FNi is the number of false negative examples of class i.

[0101] mF1 provides a balanced evaluation parameter for comprehensive precision and recall, which is the harmonic mean of precision and recall and is suitable for cases of category imbalance.

[0102] For each class i, the F1 score is defined as and the overall average F1 score is calculated as:

[0103]

[0104] The calculation formula for the average F1 score is:

[0105] The mIoU average intersection over union directly reflects the degree of overlap between the model's prediction and the true segmentation. The intersection over union measures the overlap between the predicted area and the true area. For each class i, the calculation formula for IoU is:

[0106]

[0107] The calculation formula of average intersection-over-union ratio is:

[0108] 4.2 Experimental Description

[0109] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0110] This experiment selects a fixed 14-image Postdam test dataset as the test dataset for comparative evaluation of the algorithms. Table 1 shows the comparison results between the model proposed by this method and several other model algorithms under three objective evaluation indicators.

[0111] Table 1 Comparison of objective evaluation criteria of different model algorithms.

[0112]

[0113] As shown in Table 1, although this method did not achieve the best performance in all categories, it reached the top level among similar models in most categories, and the overall effect in all categories reached the best state.

[0114] 5. Overview

[0115] In summary, the method of the present invention has the following technical advantages:

[0116] (1) The present invention introduces a channel enhancement mechanism based on the Swin Transformer encoder, uses a pooling operation to integrate the information of multiple adjacent channels, and combines the global average pooling and global maximum pooling layers between channels, so that the output feature map can better retain the relationship characteristics between channels, thereby enhancing the network's ability to capture detailed information and significantly improving the overall feature representation capability.

[0117] (2) The cross-level multi-path input decoder proposed in the present invention integrates feature information at different levels to effectively process deep semantics and detailed spatial detail information. By processing global information and local information in parallel, it ensures the edge smoothness and overall consistency of the segmentation result, thereby solving the problem of processing complex structures and diverse spectral information in high-resolution images. The feature map of global information and local information is processed by combining bilinear interpolation sampling and convolution through a multi-path feature fusion module to make them consistent in spatial resolution. The local information and global information are effectively fused through adaptive weights, which enhances the network's recognition ability for objects of different shapes and sizes and improves the generalization of the decoder. In the feature fusion process, dynamic convolution is introduced. In the depthwise separable convolution, the input feature map is first deep convolved, and then the result of the deep convolution is convolved point by point using dynamic convolution to achieve adaptive fusion of features, which helps to better adapt to targets of different shapes and sizes in the feature map, and further improves the model's adaptability and segmentation accuracy in complex scenes.

[0118] (3) The present invention uses a feature refinement module based on feature decoding of fractional order differentials to capture high-level features in images using fractional order differential operators, especially when processing edge and texture information of images. Compared with traditional feature extraction techniques, fractional order differentials can more effectively reveal subtle structural changes in images, provide richer and more detailed information for the model, and thus support more accurate image segmentation.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A remote sensing image segmentation method based on enhanced and cross-level multi-input features, characterized in that: The steps include: Obtain high-resolution remote sensing image training sample sets; Design a Swin Transformer encoder based on channel enhancement, a cross-level multi-path input decoder, and a feature refinement module based on fractional differentiation to build a pixel-level semantic segmentation model; Based on the high-resolution remote sensing image training sample set, the pixel-level semantic segmentation model is trained to obtain a trained pixel-level semantic segmentation model; Based on the trained pixel semantic segmentation model, semantic segmentation is performed on the high-resolution remote sensing image to be processed to obtain a semantic segmentation result of the high-resolution remote sensing image to be processed.

2. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that: The pixel-level semantic segmentation model consists of four cascaded channel-enhanced Swin Transformer encoders, four cross-level multi-path input decoders, and a feature refinement module based on fractional differentiation. The pixel-level semantic segmentation model uses the input high-resolution remote sensing image as the input of the first Swin Transformer encoder; the output of each Swin Transformer encoder is used as the input of the next one in the pixel semantic segmentation model; the output of the fourth Swin Transformer encoder is used as the input of the fourth cross-level multi-path input decoder; the input of each cross-level multi-path input decoder is the output of a corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model; the output of the first cross-level multi-path decoder is used as the input of a feature refinement module based on fractional-order differentiation, and the output of the feature refinement module is used as the overall output of the pixel-level semantic segmentation model.

3. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 2, characterized in that: The channel-enhanced Swin Transformer encoder includes a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet downsampling module, and a connected convolution unit; the parallel connection result of the average pooling layer and the maximum pooling layer is cascaded with the wavelet downsampling module, and the parallel connection result of the Swin transformer block and the wavelet downsampling module is cascaded with the connected convolution unit; The inputs of the channel-enhanced Swin Transformer encoder are respectively used as the inputs of the Swin transformer block, the maximum pooling layer and the average pooling layer. The outputs of the maximum pooling layer and the average pooling layer are multiplied and used as the inputs of the wavelet downsampling module. The outputs of the wavelet downsampling module and the outputs of the Swin transformer block are used as the inputs of the connected convolution unit. The outputs of the connected convolution unit are used as the outputs of the channel-enhanced Swin Transformer encoder.

4. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 3, characterized in that: The processing process of the channel-enhanced Swin Transformer encoder is shown in the following formula: Y=Concat(F SwinB ,F HWD ); F HWD =HWD(AvgPool(X)⊙MaxPool(X)); Where Y is the feature map output by the channel-enhanced Swin Transformer encoder, AvgPool(.) is the average pooling layer, MaxPool(.) is the maximum pooling layer, HWD(.) is the downsampling operation after element-wise multiplication of the average pooling layer output and the maximum pooling layer output, and F HWD is the output feature map obtained after downsampling, F SwinB is the feature map obtained by the SwinTransformer encoder after processing the input X, and Concat(.) is the connection operation.

5. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 2, characterized in that: The cross-level multi-path input decoder comprises a local branch, a global branch and a multi-path feature fusion module connected in parallel, wherein the parallel connection result of the local branch and the global branch is cascaded with the multi-path feature fusion module; Among them, the high-resolution remote sensing image is input into the Swin Transformer encoder based on channel enhancement, and the output is the underlying spatial detail feature map and the high-level semantic feature map, which are respectively input into the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced ​​with the output of the global branch as the input of the multi-path feature fusion module, and the output of the multi-path feature fusion module is used as the overall output of the cross-level multi-path input decoder.

6. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 5, characterized in that: The local branch includes a cascaded 1×1 convolution layer, a pyramid pooling layer, three parallel convolution layers, a batch normalization layer and an erosion expansion unit; the three parallel convolution layers are respectively a 1×1 convolution layer, a 3×3 convolution layer and a fractional-order convolution layer; wherein the input of the local branch is used as the input of the 1×1 convolution layer, the output of the 1×1 convolution layer is used as the input of the pyramid pooling layer, the output of the pyramid pooling layer is used as the input of the three parallel convolution layers, the outputs of the three parallel convolution layers are spliced ​​and used as the input of the batch normalization layer, the output of the batch normalization layer is used as the input of the erosion expansion unit, and the output of the erosion expansion unit is used as the output of the local branch; The global branch adopts a self-attention mechanism, inputs the input feature map into the Patch Embedding layer to generate a Query, Key and Value matrix, performs a dot product operation on the Query and Key, and outputs an attention score; then, a bias term is added to the dot product result of the Query and Key and a Softmax function is applied to convert the attention score into a standardized probability distribution; finally, the standardized probability distribution and the Key are weighted averaged to obtain a global branch feature map as the output of the global branch; The multi-channel feature fusion module includes a cascaded Sum layer, a depth-separable convolution layer, a batch normalization layer and a 1×1 convolution layer; wherein the input local branch and global branch output feature maps are used as inputs of the Sum layer to perform an addition operation on the input feature map in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is output; then, the combined feature map is used as input of the depth-separable convolution layer to further extract and integrate local feature information and global feature information; finally, the output of the depth-separable convolution layer is further compressed and the feature dimension is integrated through the batch normalization layer and the 1×1 convolution layer to generate the output feature map of the multi-channel feature fusion module.

7. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 2, characterized in that: The feature refinement module based on fractional differential includes a cascaded channel path, a spatial path, a convolution activation unit, a 1×1 convolution layer and an upsampling module; the result of the channel path and the spatial path being connected in parallel is cascaded with the convolution activation unit, the channel path includes a cascaded global average pooling layer, a fully connected layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth-separable convolution layer and a Sigmoid activation function layer, and the convolution activation unit includes a cascaded 3×3 convolution layer, a batch normalization layer and a ReLU activation function layer; wherein the input feature map is used as the input of the channel path and the spatial path respectively to generate a channel attention weight and a spatial attention weight, and the channel attention weight and the spatial attention weight are multiplied with the input feature map to fuse the output feature map; then, the fused feature map is processed by the convolution activation unit; then, the input feature map is reduced in dimension by a 1×1 convolution layer and then added to the fused feature map to realize residual connection; finally, the feature map after the addition operation is used as the input of the upsampling module to obtain a semantic segmentation map as the output.

8. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that: In the process of training the pixel-level semantic segmentation model, the composite training loss is calculated based on the semantic segmentation results of the high-resolution remote sensing image training samples by the pixel-level semantic segmentation model and the true semantic category labels of the corresponding high-resolution remote sensing image training samples. The parameters of the pixel-level semantic segmentation model are optimized and updated with the goal of minimizing the composite training loss function, and the learning rate of the optimized update parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

9. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 8, characterized in that: The cosine annealing strategy is shown below: Where η t is the learning rate of the tth training batch, T is the total training batches, and η is the initial learning rate.

10. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 8, characterized in that: The functional expression of the composite training loss is as follows: In the formula, is the composite training loss function, is the main loss function, is the auxiliary loss function, α is the auxiliary loss weight factor, is the Dice loss function, is the cross entropy loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample types contained in the high-resolution remote sensing image training sample set, respectively. represents the true semantic category label of the kth high-resolution remote sensing image training sample of the nth sample category, Represents the semantic segmentation result of the pixel-level semantic segmentation model for the kth high-resolution remote sensing image training sample of the nth sample category.

Images