A remote sensing image segmentation method based on channel enhancement and cross-level multi-input features

Abstract

The application discloses a remote sensing image segmentation method based on enhancement and cross-level multi-input features, first, a high-resolution remote sensing image training sample set is acquired; a Swin Transformer encoder based on channel enhancement, a cross-level multi-path input decoder and a feature refinement module based on fractional differential are designed to construct a pixel-level semantic segmentation model; then, based on the high-resolution remote sensing image training sample set, the pixel-level semantic segmentation model is trained to obtain a trained pixel-level semantic segmentation model; finally, based on the trained pixel semantic segmentation model, semantic segmentation is performed on a high-resolution remote sensing image to be processed to obtain a semantic segmentation result of the high-resolution remote sensing image to be processed. The method effectively improves the problems of boundary blur, small target loss and insufficient information processing in high-resolution remote sensing image semantic segmentation, and improves the accuracy of the pixel-level semantic segmentation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features. BACKGROUND

[0002] In the research of high-resolution remote sensing image semantic segmentation, the main goal is to develop a method that can effectively process ultra-high resolution images, especially under limited GPU memory, and provide high-quality segmentation results. This requires solving some problems existing in current methods, such as unclear boundaries, small target loss, and complexity and time-consuming of the training process. In addition, these methods should fully utilize the ultra-high resolution characteristics of remote sensing images, such as effectively capturing and utilizing global context information and local detail information, while maintaining segmentation accuracy. At present, the mainstream semantic segmentation network usually adopts an encoder-decoder based structure, and its specific process is as shown in the equation: In the formula, E(X) is an encoder for feature extraction of a source image X, and the extracted feature is F; D(F) is a decoder for decoding the extracted feature F to reconstruct a segmentation image corresponding to the source image

[0003] In the encoder part, the current mainstream method uses Swin Transformer for feature extraction to obtain richer feature maps. However, in the process of processing the feature maps, the calculation between each channel of the Swin Transformer is independent of each other, which means that the correlation information between the channels is ignored. This independent calculation method limits its ability to extract and utilize the interaction information between the channels. Therefore, although the Swin Transformer performs excellently in feature extraction on a single channel, it still has certain limitations in overall channel feature fusion and information complementation. In the decoder part, compared with traditional machine learning methods, the method based on convolutional neural network (CNN) can capture more fine-grained local context information, which provides great potential for it in feature representation and pattern recognition. However, the convolution operation with fixed receptive field is mainly used for extracting local features, and lacks the modeling ability of global context information or long-distance dependency. In semantic segmentation, only modeling local information often leads to ambiguity in global pixel-level classification, because the semantic content of each pixel becomes more accurate with the help of global context information. In addition, the current mainstream method uses a convolution method that mixes and stacks multiple features during convolution to extract as much complex nonlinear relationship between features and targets as possible, but this undoubtedly increases the computational complexity of the network. In the traditional convolution template, the integer-order differential operator is usually suitable for processing high-frequency components in the image, but it has poor detail processing ability for non-continuous boundary points and low-frequency variation characteristics. The rich surface texture details in the image belong to medium and low frequency components, and the integer-order differential operator is difficult to effectively process.

[0004] Therefore, how to effectively process the semantic segmentation of ultra-high resolution remote sensing images and provide high-quality segmentation results has become a problem that those skilled in the art need to solve. SUMMARY

[0005] In view of the above problems of the prior art, the present application provides a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features, which can improve the accuracy of the pixel-level semantic segmentation model and solve the problems of boundary blurring, small target loss and insufficient information processing in high-resolution remote sensing image semantic segmentation.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] A remote sensing image segmentation method based on channel enhancement and cross-level multi-input features, comprising the following steps:

[0008] Obtain a high-resolution remote sensing image training sample set;

[0009] A pixel-level semantic segmentation model is constructed by designing a channel-enhanced Swin Transformer encoder, a cross-level multi-path input decoder and a fractional differential-based feature refinement module, and introducing a cosine annealing strategy and a composite training loss function.

[0010] The pixel-level semantic segmentation model is trained based on the high-resolution remote sensing image training sample set to obtain a trained pixel-level semantic segmentation model.

[0011] The high-resolution remote sensing image to be processed is subjected to semantic segmentation based on the trained pixel semantic segmentation model to obtain a semantic segmentation result of the high-resolution remote sensing image to be processed.

[0012] As a preferred scheme, the pixel-level semantic segmentation model comprises four channel-enhanced Swin Transformer encoders, four cross-level multi-path input decoders and one fractional differential-based feature refinement module connected in sequence.

[0013] The pixel-level semantic segmentation model takes the input high-resolution remote sensing image as the input of the first Swin Transformer encoder; the output of each Swin Transformer encoder is taken as the input of the next Swin Transformer encoder connected in the pixel semantic segmentation model; the output of the fourth Swin Transformer encoder is taken as the input of the fourth cross-level multi-path input decoder; the input of each cross-level multi-path input decoder is the output of the corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model; the output of the first cross-level multi-path decoder is taken as the input of the fractional differential-based feature refinement module, and the output of the feature refinement module is taken as the overall output of the pixel-level semantic segmentation model.

[0014] As a preferred scheme, the channel-enhanced Swin Transformer encoder comprises a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet downsampling module and a connection convolution unit; the parallel results of the average pooling layer and the maximum pooling layer and the wavelet downsampling module are connected in cascade, the parallel results of the Swin transformer block and the wavelet downsampling module and the connection convolution unit are connected in cascade.

[0015] The input of the channel enhancement based Swin Transformer encoder is respectively input into a Swin transformer block, a maximum pooling layer and an average pooling layer, the outputs of the maximum pooling layer and the average pooling layer are multiplied, and the multiplied result is input into a wavelet downsampling module, the output of the wavelet downsampling module and the output of the Swin transformer block are input into a connection convolution unit, and the output of the connection convolution unit is the output of the channel enhancement based Swin Transformer encoder.

[0016] As a preferred solution, the processing process of the channel enhancement based Swin Transformer encoder is shown in the following formula:

[0017] Y = Concat (F SwinB , F HWD );

[0018] F HWD = HWD (AvgPool (X) O MaxPool (X));

[0019] In the formula, Y is the feature map output by the channel enhancement based Swin Transformer encoder, AvgPool(.) is an average pooling layer, MaxPool(.) is a maximum pooling layer, HWD(.) is a downsampling operation after element-wise multiplication of the average pooling layer output and the maximum pooling layer output, F HWD is the output feature map after downsampling, F SwinB is a feature map obtained by processing the input X by the Swin Transformer encoder, and Concat(.) is a connection operation.

[0020] As a preferred solution, the cross-level multi-path input decoder includes a local branch, a global branch and a multi-path feature fusion module connected in parallel, and the parallel connection result of the local branch and the global branch is cascaded with the multi-path feature fusion module.

[0021] The high-resolution remote sensing image is input into the channel enhancement based Swin Transformer encoder, and the bottom layer spatial detailed feature map and the high layer semantic feature map are obtained, which are respectively input into the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced with the output of the global branch, and the spliced result is input into the multi-path feature fusion module, and the output of the multi-path feature fusion module is the overall output of the cross-level multi-path input decoder.

[0022] As a preferred solution, the local branch comprises a cascaded 1x1 convolution layer, a pyramid pooling layer, three parallel convolution layers, a batch normalization layer and an erosion dilation unit; the three parallel convolution layers are respectively a 1x1 convolution layer, a 3x3 convolution layer and a fractional convolution layer; wherein, the input of the local branch is taken as the input of the 1x1 convolution layer, the output of the 1x1 convolution layer is taken as the input of the pyramid pooling layer, the output of the pyramid pooling layer is taken as the input of the three parallel convolution layers, the outputs of the three parallel convolution layers are spliced and taken as the input of the batch normalization layer, the output of the batch normalization layer is taken as the input of the erosion dilation unit, and the output of the erosion dilation unit is taken as the output of the local branch;

[0023] The global branch adopts a self-attention mechanism, inputs an input feature map into a Patch Embedding layer to generate Query, Key and Value matrices, performs a dot product operation on the Query and Key to obtain an attention score, adds a bias term to the dot product result of the Query and Key, and applies a Softmax function to convert the attention score into a normalized probability distribution, and finally, the normalized probability distribution is weighted and averaged with the Key to obtain a global branch feature map as the output of the global branch;

[0024] The multi-path feature fusion module comprises a cascaded Sum layer, a depth separable convolution layer, a batch normalization layer and a 1x1 convolution layer; wherein, the input local branch and global branch output feature maps are taken as the input of the Sum layer, which is used to add the input feature maps in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is output; then, the combined feature map is taken as the input of the depth separable convolution layer to further extract and integrate the local feature information and the global feature information; finally, the output of the depth separable convolution layer is further compressed and integrated in the feature dimension through the batch normalization layer and the 1x1 convolution layer to generate the output feature map of the multi-path feature fusion module.

[0025] As a preferred solution, the feature refinement module based on fractional differential includes a cascaded channel path, a spatial path, a convolution activation unit, a 1×1 convolution layer and an upsampling module; the result of the channel path and the spatial path in parallel is cascaded with the convolution activation unit, the channel path includes a cascaded global average pooling layer, a fully connected layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth-separable convolution layer and a Sigmoid activation function layer, the convolution activation unit includes a cascaded 3×3 convolution layer, a batch normalization layer and a ReLU activation function layer; wherein, the input feature map is used as the input of the channel path and the spatial path respectively to generate the channel attention weight and the spatial attention weight, which are multiplied with the input feature map to fuse the output feature map; then, the fused feature map is processed by the convolution activation unit; after that, the input feature map is reduced in dimension by a 1×1 convolution layer and added to the fused feature map to realize the residual connection; finally, the feature map after the addition operation is used as the input of the upsampling module to obtain the semantic segmentation map as the output.

[0026] As a preferred solution, in the process of training the pixel-level semantic segmentation model, a composite training loss is calculated based on the semantic segmentation results of the high-resolution remote sensing image training samples by the pixel-level semantic segmentation model and the true semantic category labels of the corresponding high-resolution remote sensing image training samples. The parameters of the pixel-level semantic segmentation model are optimized and updated with the goal of minimizing the composite training loss function, and the learning rate of the optimized update parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

[0027] As a preferred solution, the cosine annealing strategy is shown as follows:

[0028]

[0029] Where η t is the learning rate of the tth training batch, T is the total training batches, and η is the initial learning rate.

[0030] As a preferred solution, the functional expression of the composite training loss is as follows:

[0031]

[0032] Where, is the composite training loss function, is the main loss function, is the auxiliary loss function, α is the auxiliary loss weight factor, is the cross entropy loss function, Dice loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample categories contained in the high-resolution remote sensing image training sample set respectively, The true semantic class label of the kth high-resolution remote sensing image training sample of the nth sample category, The semantic segmentation result of the kth high-resolution remote sensing image training sample of the nth sample category by the pixel-level semantic segmentation model.

[0033] Compared with the prior art, the present application has the following technical effects:

[0034] (1) The present application introduces a channel enhancement mechanism based on the Swin Transformer encoder, integrates the information of multiple adjacent channels using pooling operation, and uses a combination of inter-channel global average pooling and global maximum pooling layer to make the output feature map better preserve the relationship features between channels, thereby enhancing the network's ability to capture detailed information and significantly improving the overall feature representation capability.

[0035] (2) The cross-level multi-path input decoder proposed in the present application effectively processes deep-level semantics and detailed spatial detail information by integrating feature information of different levels, ensures edge smoothness and overall consistency of the segmentation result by parallel processing of global information and local information, thereby solving the processing difficulty of complex structures and diverse spectral information in high-resolution images; the multi-path feature fusion module processes the feature maps of global information and local information through a combination of bilinear interpolation sampling and convolution, so that they are consistent in spatial resolution, and effectively fuses local information and global information through adaptive weights, enhancing the network's ability to recognize objects of different shapes and sizes and improving the generalization of the decoder; in the feature fusion process, dynamic convolution is introduced, which first performs depth convolution on the input feature map in the depth separable convolution, and then uses dynamic convolution to perform point-by-point convolution on the result of the depth convolution to achieve adaptive fusion of the features, which helps to better adapt to targets of different shapes and sizes in the feature map, further improving the model's adaptability and segmentation precision in complex scenes.

[0036] (3) The feature refinement module based on fractional differential feature decoding in the present application uses fractional differential operators to capture high-level features in images, especially when processing image edges and texture information. Compared with traditional feature extraction techniques, fractional differential can more effectively reveal the subtle structural changes of images, providing more abundant and fine information for the model, thereby supporting more accurate image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings, in which:

[0038] Figure 1 A flowchart of a remote sensing image segmentation method based on enhanced and cross-level multi-input features is disclosed in the application.

[0039] Figure 2 A pixel-level semantic segmentation model network structure diagram is disclosed in the application.

[0040] Figure 3 A Swin Transformer encoder network structure diagram based on channel enhancement is disclosed in the application.

[0041] Figure 4 A cross-level multi-path input decoder network structure diagram is disclosed in the application.

[0042] Figure 5 A fractional convolution template schematic diagram is disclosed in the application.

[0043] Figure 6 A fractional convolution calculation process schematic diagram is disclosed in the application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings, in which:

[0045] The application will be further described in detail below with reference to the drawings.

[0046] In the research of high-resolution remote sensing image semantic segmentation, the main goal is to develop a method that can effectively process ultra-high resolution images, especially under limited GPU memory, to provide high-quality segmentation results. However, there are some problems in the implementation of existing technologies, such as unclear boundaries, small target loss, and complexity and time-consuming of the training process. In addition, how to fully utilize the ultra-high resolution features of remote sensing images while maintaining segmentation accuracy, such as how to effectively capture and utilize global context information and local detail information, and a series of problems limit the practical application of high-resolution remote sensing image semantic segmentation.

[0047] To overcome this limitation, the present application proposes a remote sensing image segmentation method based on enhanced and cross-level multi-input features, the method flow is as shown in Figure 1 The specific steps include the following:

[0048] Obtain a high-resolution remote sensing image training sample set;

[0049] Design a Swin Transformer encoder based on channel enhancement, a cross-level multi-path input decoder, and a feature refinement module based on fractional order differentiation to construct a pixel-level semantic segmentation model;

[0050] Based on the high-resolution remote sensing image training sample set, the pixel-level semantic segmentation model is trained to obtain a trained pixel-level semantic segmentation model;

[0051] Based on the trained pixel semantic segmentation model, the high-resolution remote sensing image to be processed is subjected to semantic segmentation to obtain the semantic segmentation result of the high-resolution remote sensing image to be processed.

[0052] By the above process of the present application, it can be seen that in the encoder part, the current mainstream method uses SwinTransformer for feature extraction to obtain more rich feature maps. However, in the process of processing the feature maps, the calculation between each channel of SwinTransformer is independent of each other, which means that the correlation information between the channels is ignored. This independent calculation method limits its ability to extract and utilize the interaction information between the channels. As can be seen, although SwinTransformer performs excellently in feature extraction on a single channel, it still has certain limitations in overall channel feature fusion and information complementation. Therefore, the present application introduces a channel enhancement mechanism on the basis of the prior art, so that the output feature map can better retain the relationship features between the channels, thereby enhancing the network's ability to capture detailed information and significantly improving the overall feature representation capability. In the decoder part, although the current method based on convolutional neural network can capture more fine local context information, however, the fixed receptive field convolution operation is mainly used for extracting local features, and lacks the modeling ability of global context information or long-distance dependency relationship. Therefore, the present application introduces a self-attention mechanism to add global context information to local features to alleviate the above problems. In addition, the current mainstream method adds the operation complexity of the network by using a mixed stack of multiple feature convolutions in the convolution process. The present application proposes a simple convolution combination based on fractional order convolution to extract non-linear relationships as much as possible. In addition, in the traditional convolution template, the integer order differential operator is usually suitable for processing the components with high frequency changes in the image, but has poor processing ability for non-continuous boundary points and details with low frequency change characteristics. Therefore, the present application uses a fractional order differential operator with an order between 0 and 1 to process the image, which can better process noise or edge information, enhance the texture details in the image, and preserve the information of smooth areas.

[0053] In general, the present application proposes a remote sensing image segmentation method based on channel enhancement and cross-level multi-input features, which enhances the extraction and utilization of inter-channel correlation information by introducing a channel enhancement mechanism based on the Swin Transformer encoder, thereby improving the overall feature representation capability. On the other hand, the combination of the Swin Transformer encoder and the lightweight decoder based on the multi-head self-attention mechanism enables the model to ensure high precision while having significant computational efficiency, which is suitable for high-efficiency processing requirements in actual industrial applications.

[0054] In order to better introduce the technical scheme of the present application, the following several parts will be described in more detail.

[0055] 1、Image preprocessing

[0056] In a specific implementation application, the high-resolution remote sensing image used for training and testing of the pixel semantic segmentation model can be selected using the public Potsdam dataset as the training sample set of the high-resolution remote sensing image for training the pixel-level semantic segmentation model. After selecting the high-resolution remote sensing image training sample from the public Potsdam dataset, the high-resolution remote sensing image training sample is cropped, and the cropped dataset is subjected to data enhancement processing. A part of the high-resolution remote sensing image training sample subjected to enhancement processing is selected using the RGB color space and used as the test set, and the remaining high-resolution remote sensing image training sample subjected to enhancement processing is used as the training set. For example, the public Potsdam dataset contains 38 high-resolution remote sensing images, each image has a size of 6000x6000, and we use the sample of the RGB color space to fix 14 images for testing and the remaining images for training.

[0057] In the experiment, we cropped the original image into a 1024x1024 size block and enhanced the dataset by rotating at a random angle or horizontally flipping to improve the training effect.

[0058] 2、Pixel-level semantic segmentation model

[0059] As shown in Figure 2 , the pixel-level semantic segmentation model proposed by the method of the application includes four channel-enhanced Swin Transformer encoders, four cross-level multi-path input decoders, and one feature refinement module based on fractional order differentiation, which are sequentially connected. The pixel-level semantic segmentation model takes the input high-resolution remote sensing image as the input of the first Swin Transformer encoder. The output of each Swin Transformer encoder is used as the input of the next Swin Transformer encoder in the pixel semantic segmentation model. The output of the fourth Swin Transformer encoder is used as the input of the fourth cross-level multi-path input decoder. The input of each cross-level multi-path input decoder is the output of its corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model. The output of the first cross-level multi-path decoder is used as the input of the feature refinement module based on fractional order differentiation, and the output of the feature refinement module is used as the overall output of the pixel-level semantic segmentation model.

[0060] Next, each part of the pixel-level semantic segmentation model will be described.

[0061] 2.1、Channel-enhanced Swin Transformer encoder

[0062] As shown in Figure 3As shown, the channel enhancement based Swin Transformer encoder includes a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet down-sampling module and a connection convolution unit; the parallel results of the average pooling layer and the maximum pooling layer and the wavelet down-sampling module are cascaded, the parallel results of the Swin transformer block and the wavelet down-sampling module and the connection convolution unit are cascaded;

[0063] The input of the channel enhancement based Swin Transformer encoder is taken as the input of the Swin transformer block, the maximum pooling layer and the average pooling layer respectively, the output of the maximum pooling layer and the average pooling layer is multiplied and taken as the input of the wavelet down-sampling module, the output of the wavelet down-sampling module and the output of the Swin transformer block are taken as the input of the connection convolution unit, and the output of the connection convolution unit is taken as the output of the channel enhancement based Swin Transformer encoder.

[0064] It can be seen that the encoder based on the Transformer performs outstandingly in pursuing high precision, so the encoder based on Swin-Base is adopted in the specific implementation and application of the present application. However, SwinB only calculates the window and the sliding window at the feature map level, which naturally ignores the relationship between channels, resulting in the loss of some channel information of the encoder based on SwinB. Therefore, the channel enhancement is performed on the basis of SwinB, so that the feature map output by SwinB can retain more inter-channel relationship features. Pooling is used to integrate the information of multiple adjacent channels, a channel-based global average pooling layer and a global maximum pooling layer are used to enhance the common features while retaining the overall representation of the channels, and finally the integrated features pass through a wavelet down-sampling layer to reduce the spatial resolution while retaining more information.

[0065] The processing process of the channel enhancement based Swin Transformer encoder is shown in the following formula:

[0066] Y=Concat(F SwinB ,F HWD );

[0067] F HWD =HWD(AvgPool(X)⊙MaxPool(X));

[0068] In the formula, Y is the feature map output by the channel enhancement based Swin Transformer encoder, AvgPool(.) is the average pooling layer, MaxPool(.) is the maximum pooling layer, HWD(.) is the down-sampling operation after the element-wise multiplication of the average pooling layer output and the maximum pooling layer output, and F HWDF SwinB is the feature map obtained after processing the input X by the SwinTransformer encoder, and Concat(.) is a concatenation operation.

[0069] 2.2 Cross-level multi-path input decoder

[0070] As shown in Figure 4 , the cross-level multi-path input decoder includes a local branch, a global branch and a multi-path feature fusion module in parallel, wherein the parallel result of the local branch and the global branch is cascaded with the multi-path feature fusion module;

[0071] The local branch includes a 1x1 convolution layer, a pyramid pooling layer, three convolution layers in parallel, a batch normalization layer and an erosion expansion unit in cascade; the three convolution layers in parallel are a 1x1 convolution layer, a 3x3 convolution layer and a fractional convolution layer respectively; wherein the input of the local branch is taken as the input of the 1x1 convolution layer, the output of the 1x1 convolution layer is taken as the input of the pyramid pooling layer, the output of the pyramid pooling layer is taken as the input of the three convolution layers in parallel, the outputs of the three convolution layers in parallel are spliced and taken as the input of the batch normalization layer, the output of the batch normalization layer is taken as the input of the erosion expansion unit, and the output of the erosion expansion unit is taken as the output of the local branch;

[0072] The global branch adopts a self-attention mechanism, inputs the input feature map into a Patch Embedding layer to generate Query, Key and Value matrices, performs a dot product operation on Query and Key to obtain an attention score, then adds a bias term to the dot product result of Query and Key and applies a Softmax function to convert the attention score into a normalized probability distribution, and finally performs a weighted average processing on the normalized probability distribution and Key to obtain a global branch feature map as the output of the global branch;

[0073] The multi-path feature fusion module includes a Sum layer, a depth separable convolution layer, a batch normalization layer and a 1x1 convolution layer in cascade; wherein the input local branch and global branch output feature maps are taken as the input of the Sum layer, which is used to add the input feature maps in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is obtained; then the combined feature map is taken as the input of the depth separable convolution layer to further extract and integrate the local feature information and the global feature information; finally, the output of the depth separable convolution layer is further compressed and integrated in the feature dimension by the batch normalization layer and the 1x1 convolution layer to generate the output feature map of the multi-path feature fusion module.

[0074] The high-resolution remote sensing image input to the channel enhancement-based Swin Transformer encoder outputs the bottom-layer spatial detailed feature map and the high-layer semantic feature map, which are respectively input to the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced with the output of the global branch as the input of the multi-path feature fusion module, and the output of the multi-path feature fusion module is the overall output of the cross-level multi-path input decoder.

[0075] It can be seen that the output of the decoder is integrated with the cross-layer connection of the encoder, and after extracting respective semantic information, the two inputs are fused, the feature information of different levels is integrated, the deep semantic and detailed spatial detail information are effectively processed, the global information and the local information are processed in parallel, the problem that it is difficult to effectively capture and utilize the global context information and the local detail information of the image at the same time when segmenting is avoided, the edge smoothness and overall consistency of the segmentation result are ensured, and thus the processing difficulty of the complex structure and the diverse spectral information in the high-resolution image is solved.

[0076] The feature maps of the global information and the local information are processed by the multi-path feature fusion module in a manner combining bilinear interpolation sampling and convolution, so that they are consistent in spatial resolution, and the local information and the global information are effectively fused through adaptive weights, the recognition ability of the network to objects of different shapes and sizes is enhanced, and the generalization of the decoder is improved; in the feature fusion process, dynamic convolution is introduced, in the depth separable convolution, the input feature map is first subjected to depth convolution, and then the result of the depth convolution is subjected to point-by-point convolution using the dynamic convolution, so as to realize adaptive fusion of the features, which is helpful to better adapt to targets of different shapes and sizes in the feature map, and further improves the adaptability and segmentation precision of the model in complex scenes.

[0077] 2.3, feature refinement module based on fractional differential

[0078] The feature refinement module based on fractional differential includes a cascaded channel path, a spatial path, a convolution activation unit, a 1*1 convolution layer and an up-sampling module; the parallel results of the channel path and the spatial path are cascaded with the convolution activation unit, the channel path includes a cascaded global average pooling layer, a full connection layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth separable convolution layer and a Sigmoid activation function layer, and the convolution activation unit includes a cascaded 3*3 convolution layer, a batch normalization layer and a ReLU activation function layer; wherein, input feature maps are taken as inputs of the channel path and the spatial path respectively to generate channel attention weights and spatial attention weights, the channel attention weights and the spatial attention weights are multiplied with the input feature maps to fuse output feature maps; then, the fused feature maps are processed through the convolution activation unit; after that, the input feature maps are reduced in dimension through the 1*1 convolution layer and added with the fused feature maps to realize residual connection; finally, the feature maps after the addition operation are taken as inputs of the up-sampling module to obtain a semantic segmentation map as an output.

[0079] The application utilizes a fractional differential operator to capture advanced features in an image, constructs a fractional convolution mask according to a coefficient of a first n term in an approximate expression of a fractional derivative as a parameter, generates a fractional convolution template from eight directions diverging from a center, and the approximate expression of the fractional derivative is as follows:

[0080]

[0081] In the formula, is a fractional derivative of a function f(t) with respect to t, v is an order of the fractional derivative, v is a non-integer, Gamma(.) is a gamma function, n is an upper limit of a term number in the approximate expression, and n! is a factorial of n;

[0082] The first 4 terms are taken as parameters to construct a 7*7 convolution template, a learnable initial convolution fractional parameter v0 is given, and learning is performed through network back propagation, wherein the constructed convolution template is as shown in Figure 5 , a calculation process is as shown in Figure 6 , and a template parameter calculation method is as shown in the following formula:

[0083]

[0084] In the formula, W 0 , W 1 , W 2 , W 3 , and W 4 all represent convolution template weight coefficients.

[0085] The generated convolution template is used for sliding convolution on the feature map to obtain a fractional order feature-oriented feature refinement map. This helps to improve the ability of the image segmentation model to capture details, especially when dealing with image edge and texture information. Compared with traditional feature extraction techniques, fractional differential can more effectively reveal the subtle structural changes of the image, providing more rich and fine information for the model, thereby supporting more accurate image segmentation.

[0086] 3. Training of the pixel-level semantic segmentation model

[0087] In a specific implementation, the pixel-level semantic segmentation model is trained by: taking the divided training set as the input of the pixel-level semantic segmentation model, constructing a loss function for improving the sensitivity of the model to different types of errors, in the present application, a composite training loss is calculated according to the semantic segmentation result of the pixel-level semantic segmentation model on the high-resolution remote sensing image training sample and the real semantic class label of the corresponding high-resolution remote sensing image training sample, the parameters of the pixel-level semantic segmentation model are optimized and updated to minimize the composite training loss function, and the learning rate of the optimized and updated parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

[0088] In the application of the present application, the cosine annealing strategy is used, the AdamW optimizer is used for gradient descent and parameter optimization update, the initial learning rate is set to 6x10 -4 , the weight decay rate is set to 2.5x10 -4 , and the specific formula is as shown in the following formula:

[0089]

[0090] In the formula, η t is the learning rate of the tth training batch, T is the total training batch, and η is the initial learning rate.

[0091] In the application of the present application, the following composite training loss function is preferably used:

[0092]

[0093] In the formula, L is the composite training loss function, is the main loss function, is the auxiliary loss function, a is the auxiliary loss weight factor, is the cross-entropy loss function, is the Dice loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample categories contained in the high-resolution remote sensing image training sample set, respectively represent the real semantic class label of the kth high-resolution remote sensing image training sample of the nth sample category, respectively represent the semantic segmentation result of the kth high-resolution remote sensing image training sample of the nth sample category by the pixel-level semantic segmentation model, n∈[1,…,N], k∈[1,…,K]; the preferred value of the auxiliary loss weight factor a can be set as 0.4.

[0094] It can be seen that the main loss is a combination of the Dice loss and the cross-entropy loss , and the auxiliary loss is a cross-entropy loss and is applied to the auxiliary head. The auxiliary head takes three fusion features based on the channel enhanced Transformer block as input, constructs a 3×3 convolution layer containing batch normalization and ReLU, a 1×1 convolution layer, and an upsampling operation to generate the output.

[0095] 4. Evaluation experiment of the pixel-level semantic segmentation model

[0096] 4.1. Evaluation index

[0097] In specific implementation, three objective evaluation indexes of mAcc, mF1 and mIoU are used to comprehensively judge the model effect.

[0098] Specifically, mAcc measures the average classification accuracy of the model on each class, calculates the classification accuracy of the model on each class first, and then takes the average of all classes. For each class i, the accuracy is defined as:

[0099] The average classification accuracy is:

[0100] In the formula, Acc i is the classification accuracy on each class, TPi is the number of true positives of class i, FPi is the number of false positives of class i, and FNi is the number of false negatives of class i.

[0101] mF1 provides a balanced evaluation parameter of the comprehensive accuracy and recall rate, which is the harmonic mean of the accuracy (Precision) and the recall rate (Recall), and is suitable for the case of class imbalance.

[0102] For each class i, the F1 score is defined as and the calculation formula of the overall average F1 score is:

[0103]

[0104] The calculation formula of the average F1 score is:

[0105] The mIoU directly reflects the degree of overlap between the predicted and the real segmentation of the model. The IoU measures the overlap between the predicted region and the real region. For each class i, the calculation formula of the IoU is defined as:

[0106]

[0107] The calculation formula of the average IoU is:

[0108] 4.2, Experimental description

[0109] In order to better illustrate the advantages of the technical scheme of the present application, the following experiments are disclosed in this embodiment.

[0110] In this experiment, 14 fixed Postdam test data sets were selected as test data sets for comparative evaluation of the algorithm, and Table 1 shows the comparison results of the proposed model and several other model algorithms in three objective evaluation indicators.

[0111] Table 1 Comparison of objective evaluation criteria of different model algorithms.

[0112]

[0113] As shown in Table 1, although the method does not perform optimally in all categories, it reaches the top level in the same type of model in most categories, and the overall effect in all categories reaches the optimal state.

[0114] 5, Review

[0115] In summary, the method of the present application has the following technical advantages:

[0116] (1) The present application introduces a channel enhancement mechanism based on the Swin Transformer encoder, integrates the information of multiple adjacent channels using a pooling operation, and uses a combination of inter-channel global average pooling and global maximum pooling layer to enable the output feature map to better preserve the relationship features between channels, thereby enhancing the network's ability to capture detailed information and significantly improving the overall feature representation capability.

[0117] (2)The cross-level multi-path input decoder provided by the present application effectively processes deep semantic and detailed spatial detail information by integrating feature information of different levels, ensures edge smoothness and overall consistency of the segmentation result by parallel processing of global information and local information, thereby solving the processing difficulty of complex structures and diverse spectral information in high-resolution images;The multi-path feature fusion module processes the feature maps of global information and local information through a combination of bilinear interpolation sampling and convolution, so that they are consistent in spatial resolution, and effectively fuses local information and global information through adaptive weights, enhancing the network's ability to recognize objects of different shapes and sizes and improving the generalization of the decoder;In the feature fusion process, dynamic convolution is introduced, and in the depth separable convolution, the input feature map is first subjected to depth convolution, and then the results of the depth convolution are point-by-point convolved using dynamic convolution to achieve adaptive fusion of the features, which helps better adapt to targets of different shapes and sizes in the feature map and further improves the model's adaptability and segmentation accuracy in complex scenes.

[0118] (3)The feature refinement module based on fractional differential feature decoding provided by the present application uses fractional differential operators to capture high-level features in images, especially when processing image edges and texture information. Compared with traditional feature extraction techniques, fractional differential can more effectively reveal the subtle structural changes of images, providing more abundant and fine information for the model, thereby supporting more accurate image segmentation.

[0119] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described by referring to the preferred embodiments of the present application, those skilled in the art should understand that various changes can be made in form and details without departing from the spirit and scope of the present application as defined in the appended claims.

Claims

1. A remote sensing image segmentation method based on enhanced and cross-level multi-input features, characterized in that: The steps include: Obtain high-resolution remote sensing image training sample sets; Design a channel-enhanced Swin Transformer encoder, a cross-level multi-path input decoder, and a feature refinement module based on fractional differentiation to build a pixel-level semantic segmentation model. Training the pixel-level semantic segmentation model based on the high-resolution remote sensing image training sample set to obtain a trained pixel-level semantic segmentation model; Based on the trained pixel semantic segmentation model, semantic segmentation is performed on the high-resolution remote sensing image to be processed to obtain the semantic segmentation result of the high-resolution remote sensing image to be processed; The pixel-level semantic segmentation model consists of four cascaded channel-enhanced Swin Transformer encoders, four cross-level multi-path input decoders, and a feature refinement module based on fractional differentiation. The channel-enhanced Swin Transformer encoder includes a Swin transformer block, an average pooling layer, a maximum pooling layer, a wavelet downsampling module, and a connected convolution unit; the parallel connection result of the average pooling layer and the maximum pooling layer is cascaded with the wavelet downsampling module, and the parallel connection result of the Swin transformer block and the wavelet downsampling module is cascaded with the connected convolution unit; The inputs of the channel-enhanced Swin Transformer encoder are respectively used as the inputs of the Swin transformer block, the maximum pooling layer, and the average pooling layer. The outputs of the maximum pooling layer and the average pooling layer are multiplied and used as the inputs of the wavelet downsampling module. The outputs of the wavelet downsampling module and the outputs of the Swin transformer block are used as the inputs of the convolution unit. The outputs of the convolution unit are used as the outputs of the channel-enhanced Swin Transformer encoder. The fractional differential-based feature refinement module includes a cascaded channel path, a spatial path, a convolutional activation unit, a 1×1 convolutional layer and an upsampling module; the result of the parallel connection of the channel path and the spatial path is cascaded with the convolutional activation unit, the channel path includes a cascaded global average pooling layer, a fully connected layer and a Sigmoid activation function layer, the spatial path includes a cascaded depth-separable convolutional layer and a Sigmoid activation function layer, and the convolutional activation unit includes a cascaded 3×3 convolutional layer, a batch normalization layer and a ReLU activation function layer; wherein, the input feature map is used as the input of the channel path and the spatial path respectively to generate channel attention weights and spatial attention weights, and the channel attention weights and spatial attention weights are multiplied with the input feature map to fuse the output feature map; then, the fused feature map is processed by the convolutional activation unit; thereafter, the input feature map is reduced in dimension by a 1×1 convolutional layer and then added to the fused feature map to realize residual connection; finally, the feature map after the addition operation is used as the input of the upsampling module to obtain a semantic segmentation map as the output.

2. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that The pixel-level semantic segmentation model uses the input high-resolution remote sensing image as the input of the first SwinTransformer encoder; the output of each Swin Transformer encoder is used as the input of the next one in the pixel semantic segmentation model; the output of the fourth Swin Transformer encoder is used as the input of the fourth cross-level multi-path input decoder; the input of each cross-level multi-path input decoder is the output of its corresponding Swin Transformer encoder and the output of the previous layer of the cross-level multi-path input decoder in the model; the output of the first cross-level multi-path decoder is used as the input of the feature refinement module based on fractional-order differentiation, and the output of the feature refinement module is used as the overall output of the pixel-level semantic segmentation model.

3. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that: The processing process of the channel-enhanced Swin Transformer encoder is shown in the following formula: Y=Concat(F SwinB ,F HWD ); F HWD =HWD(AvgPool(X)⊙MaxPool(X)); Where Y is the feature map output by the channel-enhanced Swin Transformer encoder, AvgPool(.) is the average pooling layer, MaxPool(.) is the maximum pooling layer, HWD(.) is the downsampling operation after element-wise multiplication of the average pooling layer output and the maximum pooling layer output, and F HWD is the output feature map obtained after downsampling, F SwinB is the feature map obtained after the SwinTransformer encoder processes the input X, and Concat(.) is the connection operation.

4. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that The cross-level multi-path input decoder includes a local branch, a global branch and a multi-path feature fusion module connected in parallel, wherein the parallel connection result of the local branch and the global branch is cascaded with the multi-path feature fusion module; Among them, the high-resolution remote sensing image is input into the channel-enhanced Swin Transformer encoder, and the output is the underlying spatial detail feature map and the high-level semantic feature map, which are respectively input into the local branch and the global branch of the cross-level multi-path input decoder; the output of the local branch is spliced ​​with the output of the global branch as the input of the multi-path feature fusion module, and the output of the multi-path feature fusion module is used as the overall output of the cross-level multi-path input decoder.

5. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 4, characterized in that: The local branch includes a cascaded 1×1 convolution layer, a pyramid pooling layer, three parallel convolution layers, a batch normalization layer, and an erosion and expansion unit; the three parallel convolution layers are respectively a 1×1 convolution layer, a 3×3 convolution layer, and a fractional-order convolution layer; wherein the input of the local branch is used as the input of the 1×1 convolution layer, the output of the 1×1 convolution layer is used as the input of the pyramid pooling layer, the output of the pyramid pooling layer is used as the input of the three parallel convolution layers, the outputs of the three parallel convolution layers are spliced ​​as the input of the batch normalization layer, the output of the batch normalization layer is used as the input of the erosion and expansion unit, and the output of the erosion and expansion unit is used as the output of the local branch; The global branch uses a self-attention mechanism to input the input feature map into the Patch Embedding layer to generate the query, key, and value matrices. The query and key are dot-producted to obtain the attention score. Then, a bias term is added to the dot product of the query and key and the softmax function is applied to convert the attention score into a normalized probability distribution. Finally, the normalized probability distribution and the key are weighted averaged to obtain the global branch feature map as the output of the global branch. The multi-way feature fusion module includes a cascaded Sum layer, a depthwise separable convolution layer, a batch normalization layer, and a 1×1 convolution layer; the local branch and global branch output feature maps of the input are used as the input of the Sum layer to perform an addition operation on the input feature map in the feature dimension to ensure that the local output feature map and the global output feature map are combined in the spatial dimension, and the combined feature map is output; then, the combined feature map is used as the input of the depthwise separable convolution layer to further extract and integrate local feature information and global feature information; finally, the output of the depthwise separable convolution layer is further compressed and the feature dimension is integrated through the batch normalization layer and the 1×1 convolution layer to generate the output feature map of the multi-way feature fusion module.

6. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 1, characterized in that During the training of the pixel-level semantic segmentation model, a composite training loss is calculated based on the semantic segmentation results of the high-resolution remote sensing image training samples by the pixel-level semantic segmentation model and the true semantic category labels of the corresponding high-resolution remote sensing image training samples. The parameters of the pixel-level semantic segmentation model are optimized and updated with the goal of minimizing the composite training loss function, and the learning rate of the optimized update parameters is adjusted using a cosine annealing strategy until the pixel-level semantic segmentation model converges, thereby completing the training of the pixel-level semantic segmentation model.

7. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 6, characterized in that: The cosine annealing strategy is shown below: Where η t is the learning rate of the tth training batch, T is the total training batches, and η is the initial learning rate.

8. The remote sensing image segmentation method based on enhanced and cross-level multi-input features according to claim 6, characterized in that: The functional expression of the composite training loss is as follows: Where, is the composite training loss function, is the main loss function, is the auxiliary loss function, α is the auxiliary loss weight factor, is the cross entropy loss function, is the Dice loss function, N and K are the number of high-resolution remote sensing image training samples and the number of sample types contained in the high-resolution remote sensing image training sample set, respectively. represents the true semantic category label of the k-th high-resolution remote sensing image training sample of the n-th sample category, Represents the semantic segmentation result of the pixel-level semantic segmentation model for the k-th high-resolution remote sensing image training sample of the n-th sample category.

Images