Transform remote sensing semantic segmentation method based on space channel cross decoding

Abstract

The invention discloses a Transform remote sensing semantic segmentation method based on spatial channel cross decoding, which comprises the following steps: collecting remote sensing image sample data, and preprocessing the remote sensing image data; generating a high-dimensional feature representation of the remote sensor image through an encoder; decoding the spatial features and the channel features by using a double-current decoder network, and extracting output features of a double-current decoder, the output features including the spatial features and the channel features of the image; fusing the output features of the double-current decoder and the original image features of different scales extracted by the encoder module; the fused features are refined, and finally a segmented mask image is obtained; and obtaining a remote sensing image classification result detection result, and returning the detection result. According to the method, a deformable attention mechanism is utilized to respectively capture the context dependency relationship of space and channel features to enhance feature representation, and accurate classification of detected targets is realized.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing semantic segmentation, and more specifically, to a Transformer remote sensing semantic segmentation method based on spatial channel cross decoding. Background Art

[0002] Remote sensing images usually contain a large number of complex and diverse objects, which makes them of great value in object recognition, land cover classification and change detection. However, current remote sensing image segmentation and classification methods are difficult to detect and classify these targets efficiently and accurately. At the same time, in complex scenes, traditional segmentation methods may not be able to accurately extract object boundaries, resulting in insufficient accuracy of classification and analysis results.

[0003] Therefore, building a fast and accurate remote sensing image segmentation and classification method has become an important issue that needs to be solved urgently. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a Transformer remote sensing semantic segmentation method based on spatial channel cross decoding, which can perform image segmentation on input remote sensing image data and display the segmentation results.

[0005] To achieve the above object, the present invention provides the following technical solutions: A Transformer remote sensing semantic segmentation method based on spatial channel cross decoding comprises the following steps: collecting remote sensing image sample data and preprocessing the remote sensing image data; generating a high-dimensional feature representation of the remote sensor image through an encoder; decoding the spatial features and channel features using a dual-stream decoder network, extracting the output features of the dual-stream decoder, wherein the output features include the spatial features and channel features of the image; fusing the output features of the dual-stream decoder with the original image features of different scales extracted by the encoder module; refining the fused features to finally obtain a segmented mask image; obtaining a remote sensing image classification result detection result, and returning the detection result.

[0006] The remote sensing image sample data is collected and preprocessed, specifically including: collecting remote sensing image data containing various types of scenes and categories; scaling the remote sensing image to the model input size and scaling the pixel value to a certain range, and unifying the specifications to generate a remote sensing image data set.

[0007] In the above scheme, the high-dimensional feature representation of the remote sensor image is generated by the encoder, which specifically includes: inputting the processed remote sensing image data into the encoder module of the model, learning the dependency relationship between each pixel in the image, and generating a high-dimensional feature representation.

[0008] In the above scheme, the encoder module uses ResNet-18, and the pre-trained ResNet-18 model is loaded in the encoder module. Using the multi-layer convolution and pooling operations in the encoder, the spatial resolution of the feature map is gradually reduced, and the depth of the feature is gradually increased; the feature maps of different dimensions output by the encoder are passed to the decoder to realize multi-scale feature fusion operations.

[0009] In the above scheme, a dual-stream decoder network is used to decode the spatial features and channel features, and the output features of the dual-stream decoder are extracted. The output features include the spatial features and channel features of the image, specifically including: obtaining feature embedding through an embedding module, and adding position encoding to the feature embedding to enhance the position information in the feature embedding; for the features after adding the position encoding, they are respectively input into a multi-layer spatial decoder and a channel decoder based on a Transformer network for decoding, and the attention modules in the decoder are a self-attention module and a cross-attention module respectively; and a deformable attention mechanism is used to obtain new feature values.

[0010] The method of fusing the output features of the dual-stream decoder with the original image features of different scales extracted by the encoder module specifically includes: using a multi-scale feature fusion module to fuse the output features of the decoder and the original image features of different scales extracted by the encoder module; using the multi-scale fusion module to multiply the fused features from the decoder and the output features of the current layer encoder by a set of learnable adaptive weight coefficients respectively and concatenate and fuse them in the channel dimension to fuse features of different dimensions, and continue to use the features as input features of the lower layer decoder.

[0011] The fused features are refined to finally obtain the segmented mask image, which includes: Using the convolutional layer in the segmentation head module, the fused feature map is converted to the dimension required for segmentation to refine the details of the feature map, making it suitable for generating segmentation masks.

[0012] The membership function formula based on the improved power equipment indicator quality level is specifically as follows: the remote sensing image classification result detection result is obtained and the detection result is returned, specifically including: performing category prediction on each pixel through the segmentation head module and converting the probability value into the final category label by setting a threshold, and finally outputting and displaying the image segmentation result.

[0013] The technical effects and advantages of the Transformer remote sensing semantic segmentation method based on spatial channel cross decoding of the present invention are as follows: The remote sensing image classification method based on the convolutional neural network of the present invention realizes the rapid classification task of remote sensing images by virtue of the characteristics of the deep learning model such as fast image data processing speed, high detection accuracy and strong data analysis ability, and provides strong technical support and data support for the fields of environmental monitoring, urban planning, agricultural management, etc. It not only includes cross-decoding and feature fusion of spatial features and channel features to capture relevant semantic information. At the same time, the deformable attention mechanism is used to capture the contextual dependencies of spatial and channel features respectively to enhance the feature representation, so as to realize the accurate classification of the detected targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the Transformer remote sensing semantic segmentation method based on spatial channel cross decoding disclosed in the present invention; Figure 2 The SCCFormer model structure diagram of the present invention; Figure 3 It is the structural diagram of SCCFormer Block of the present invention; Figure 4 This is a structural diagram of the deformable attention mechanism of the present invention; Figure 5 This is a structural diagram of the feature fusion module of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1, Figure 1 The present invention provides a Transformer remote sensing semantic segmentation method based on spatial channel cross decoding, comprising the following steps: S1. Collect remote sensing image data sets containing various types of scenes and categories, and perform preprocessing operations such as data annotation and data enhancement on the remote sensing images. Among them, collect remote sensing image data containing various types of scenes and categories, scale the remote sensing images to the model input size and scale the pixel values ​​to a certain range, unify their specifications, and generate a remote sensing image data set.

[0017] S2, input the processed remote sensing image data into the encoder module of the model, learn the dependency between each pixel in the image, and generate a high-dimensional feature representation; S2-1 loads the pre-trained encoder model in the encoder module to reduce the amount of training calculation and time; S2-2 uses multi-layer convolution and pooling operations in the encoder to gradually reduce the spatial resolution of the feature map and gradually increase the depth of the feature, capturing rich image features at each layer of the original data; S2-3 passes the feature maps of different dimensions output by the encoder to the decoder to implement subsequent multi-scale feature fusion operations; In this experimental case, the system chooses to use ResNet-18 as the encoder module. By introducing residual connections to solve the gradient vanishing and gradient exploding problems in deep neural networks, the network can extract features more deeply and effectively. The encoder can extract high-level features of the image during image segmentation.

[0018] S3, using a dual-stream decoder network to decode the spatial features and channel features, and extract the spatial features and channel features of the image; S3-1 Input features are embedded through the embedding module, and position encoding is added to the feature embedding to enhance the position information in the feature embedding; S3-2 After adding position encoding, the features are input into the multi-layer spatial decoder and channel decoder based on the Transformer network for decoding; The attention modules in the S3-3 decoder are the self-attention module and the cross-attention module.

[0019] For the self-attention module, the features after adding the position encoding will be transformed to obtain the self-attention search vector and , and performs self-attention operation based on deformable attention mechanism with feature embedding without adding position encoding.

[0020] For the criss-cross attention module, its criss-cross attention search vector and comes from the output of the previous decoder. And the attention query vector from the previous layer is cross-transferred. The vector is used to calculate the channel cross attention, and It is used to calculate the spatial cross attention, and realizes the cross decoding of spatial features and channel features by cross-transmitting the spatial query vector and the channel query vector between the multi-layer decoder modules. The specific process is shown in equations (1) and (2). (1) (2) in, , , , Respectively represent spatial cross attention, spatial self-attention, channel cross attention, channel self-attention operations, Feature Embedding.

[0021] S3-4 uses a deformable attention mechanism to obtain new feature values, which is specifically reflected in the spatial deformable attention module in the spatial decoder and the channel deformable attention module in the channel decoder.

[0022] The input of the spatial deformable attention module is the spatial query vector and spatial features. The query vector is mapped through the convolutional layer to generate the offset matrix and weight matrix, and the spatial position reference coordinates of the current feature point are generated from the spatial features. The offset matrix represents the offset value of the current feature point. The actual sampling position is obtained by adding the offset value to the reference coordinates and resampling the feature map to obtain multiple offset feature points. The offset feature point is multiplied by the corresponding feature point weight to generate the new feature value of the current feature point.

[0023] The channel attention mechanism obtains the channel offset matrix and weight matrix from the channel query vector through the convolution layer operation, and generates the reference coordinates of the current channel from the channel features. Multiple offset channels of the current channel of the channel features are obtained by adding the offset matrix and the reference coordinates, and the offset channels are multiplied by the corresponding weight coefficients as the new feature values ​​of the current channel.

[0024] S4, fusion of the output features of the dual-stream decoder and the original image features of different scales extracted by the encoder module; S4-1 uses the multi-scale feature fusion module to fuse the output features of the decoder with the original image features of different scales extracted by the encoder module.

[0025] (3) (4) (5) For the spatial features output by the upper decoder And channel characteristics The feature fusion module adopts a cross-weight fusion strategy to generate a set of attention weights for spatial features and channel features respectively, and , With weight coefficient , After cross multiplication, add fusion is performed to obtain the spatial channel fusion feature .

[0026] S4-2 uses a multi-scale fusion module. The fusion features from the decoder and the output features of the current layer encoder are multiplied by a set of learnable adaptive weight coefficients and concatenated and fused in the channel dimension to fuse features of different dimensions and continue to use the features as input features of the lower layer decoder.

[0027] In this experimental case, the system chooses to use a multi-scale feature fusion module, which fuses the output features of the dual-stream Transformer decoder with the original image features of different scales extracted by the encoder module, and connects decoder modules at different levels. The feature fusion module includes a spatial and channel feature fusion module and a multi-scale feature fusion module, which captures the relevant semantic information between spatial features and channel features, thereby improving the model detection accuracy.

[0028] S5-1 uses the convolutional layer in the segmentation head module to convert the fused feature map into the dimension required for segmentation, thereby refining the details of the feature map and making it suitable for generating segmentation masks.

[0029] S6. Obtain the remote sensing image classification result detection result and return the detection result; S6-1 The segmentation head module predicts the category of each pixel and converts the probability value into the final category label by setting a threshold. Finally, the image segmentation result is output and displayed.

[0030] In this experimental case, the spatial deformable attention and channel deformable attention in the system both use multi-head attention mechanisms to fully capture the global and local information of the spatial and channel dimensions. It should be noted that since the first-layer decoder does not have input from the pre-decoder, the cross-attention module is removed in the first-layer decoder, and only self-attention operations are performed.

[0031] S4, fusion of the output features of the dual-stream decoder and the original image features of different scales extracted by the encoder module; S4-1 uses the multi-scale feature fusion module to fuse the output features of the decoder with the original image features of different scales extracted by the encoder module.

[0032] S4-2 uses a multi-scale fusion module. The fusion features from the decoder and the output features of the current layer encoder are multiplied by a set of learnable adaptive weight coefficients and concatenated and fused in the channel dimension to fuse features of different dimensions and continue to use the features as input features of the lower layer decoder.

[0033] In this experimental case, the system chooses to use a multi-scale feature fusion module, which fuses the output features of the dual-stream Transformer decoder with the original image features of different scales extracted by the encoder module, and connects decoder modules at different levels. The feature fusion module includes a spatial and channel feature fusion module and a multi-scale feature fusion module, which captures the relevant semantic information between spatial features and channel features, thereby improving the model detection accuracy.

[0034] S5, refine the fused features through the segmentation head module, and finally obtain the segmented mask image; S5-1 uses the convolutional layer in the segmentation head module to convert the fused feature map into the dimension required for segmentation, thereby refining the details of the feature map and making it suitable for generating segmentation masks.

[0035] S6. Obtain the remote sensing image classification result detection result and return the detection result; S6-1 The segmentation head module predicts the category of each pixel and converts the probability value into the final category label by setting a threshold. Finally, the image segmentation result is output and displayed.

[0036] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0037] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0038] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0039] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0040] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A Transformer remote sensing semantic segmentation method based on spatial channel cross decoding, characterized in that: The steps include: Collect remote sensing image sample data and pre-process the remote sensing image data; Generate a high-dimensional feature representation of the remote sensor image through an encoder; Decoding the spatial features and the channel features using a dual-stream decoder network, and extracting output features of the dual-stream decoder, wherein the output features include the spatial features and the channel features of the image; Fusion of the output features of the dual-stream decoder and the original image features of different scales extracted by the encoder module; The fused features are refined to finally obtain the segmented mask image; Get the remote sensing image classification result detection result and return the detection result.

2. According to claim 1, the Transformer remote sensing semantic segmentation method based on spatial channel cross decoding is characterized in that: The collecting of remote sensing image sample data and preprocessing of the remote sensing image data specifically includes: Collect remote sensing image data containing a variety of different types of scenes and categories; The remote sensing image is scaled to the model input size and the pixel values ​​are scaled to a certain range and standardized to generate a remote sensing image dataset.

3. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 2 is characterized in that: The step of generating a high-dimensional feature representation of a remote sensor image by an encoder specifically includes: The processed remote sensing image data is input into the encoder module of the model to learn the dependencies between pixels in the image and generate high-dimensional feature representations.

4. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 2 is characterized in that: The encoder module adopts ResNet-18, and a pre-trained ResNet-18 model is loaded in the encoder module.

5. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 4 is characterized in that: Using multi-layer convolution and pooling operations in the encoder, the spatial resolution of the feature map is gradually reduced and the depth of the feature is gradually increased; the feature maps of different dimensions output by the encoder are passed to the decoder to realize multi-scale feature fusion operations.

6. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 1, characterized in that: The spatial features and channel features are decoded using a dual-stream decoder network, and the output features of the dual-stream decoder are extracted. The output features include the spatial features and channel features of the image, specifically including: The feature embedding is obtained through the embedding module, and position encoding is added to the feature embedding to enhance the position information in the feature embedding; The features after adding position encoding are respectively input into the multi-layer spatial decoder and channel decoder based on the Transformer network for decoding. The attention modules in the decoder are the self-attention module and the cross-attention module respectively. Use the deformable attention mechanism to obtain new feature values.

7. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 6 is characterized in that: The output features of the fused dual-stream decoder and the original image features of different scales extracted by the encoder module specifically include: Use the multi-scale feature fusion module to fuse the output features of the decoder and the encoder module to extract original image features of different scales; Using a multi-scale fusion module, the fusion features from the decoder and the output features of the current layer encoder are multiplied by a set of learnable adaptive weight coefficients and concatenated and fused in the channel dimension to fuse features of different dimensions, and the features continue to be used as input features of the lower layer decoder.

8. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 6, characterized in that: The fused features are refined to finally obtain the segmented mask image, which includes: Using the convolutional layer in the segmentation head module, the fused feature map is converted to the dimension required for segmentation to refine the details of the feature map, making it suitable for generating segmentation masks.

9. The Transformer remote sensing semantic segmentation method based on spatial channel cross decoding according to claim 2, characterized in that: The membership function formula based on the improved power equipment indicator quality level is specifically as follows: the acquisition of remote sensing image classification result detection result and returning the detection result specifically includes: The segmentation head module predicts the category of each pixel and converts the probability value into the final category label by setting a threshold. Finally, the image segmentation result is output and displayed.

Images