Transform remote sensing semantic segmentation method based on adaptive dynamic attention mechanism

Abstract

The invention discloses a Transform remote sensing semantic segmentation method based on an adaptive dynamic attention mechanism, and relates to the technical field of the adaptive dynamic attention mechanism, and the method comprises the following steps: obtaining a remote sensing image data set, and carrying out the data marking and data enhancement; adjusting the remote sensing image into a model input size and a pixel value unified specification, and generating a standardized data set; feature embedding is generated based on a preset embedding module, and position coding is added to enhance position information for decoding; correcting the decoding process by using a self-attention and cross attention module; combining different scale features with continuously iterated updated adaptive weights to obtain a fusion feature value; constructing a multi-task joint loss function based on the to-be-solved problem; refining the fusion feature value through a segmentation module, and generating predicted values of different sub-tasks; according to the method, the problems of information loss and inconsistency possibly occurring during feature fusion in a multi-scale task in a traditional method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of adaptive dynamic attention mechanism, and more specifically, to a Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism. Background Art

[0002] Semantic segmentation of remote sensing images is an important task in the field of computer vision, and has been widely used in the fields of geographic information systems (GIS), agricultural monitoring, urban planning, environmental protection, etc. The goal of the semantic segmentation task is to classify each pixel in the image into a specific category, which is usually used to automatically analyze different types of objects in remote sensing images, such as buildings, water bodies, forests, etc. However, due to the high resolution, large scale, changing environment and complex object structure of remote sensing images, traditional image segmentation methods face many challenges, such as insufficient segmentation accuracy, difficulty in feature extraction, unclear boundaries and other problems.

[0003] At present, deep learning technology, especially convolutional neural network (CNN), has achieved remarkable results in remote sensing image segmentation. CNN can automatically learn high-level features from images, avoiding the shortcomings of traditional methods that rely on manual feature design. However, with the increasing complexity of remote sensing images, the existing single convolutional network still has limitations in multi-scale information extraction, context understanding and fine-grained feature expression. This is mainly because the convolution operation may be limited by spatial resolution when capturing long-distance dependencies, making it difficult to fully utilize global information.

[0004] In order to solve this problem, in recent years, Transformer-based models have gradually been introduced into the remote sensing image segmentation task. The Transformer model can capture long-distance dependencies globally through the self-attention mechanism, overcoming the shortcomings of traditional CNN in local feature processing. Especially in multi-scale scenarios, Transformer can effectively process complex contextual information, thereby improving the accuracy of remote sensing image segmentation. However, existing Transformer-based remote sensing image segmentation methods still face problems such as how to effectively fuse multi-scale features, how to dynamically adjust feature weights, and how to improve the accuracy of the decoding process.

[0005] In addition, existing semantic segmentation methods usually use fixed network structures and training strategies, lack flexibility and adaptability, and are difficult to optimize for different remote sensing image data and task requirements. Therefore, how to design a more adaptive network architecture, introduce more sophisticated feature fusion and processing strategies, and how to effectively combine multi-task learning and loss function optimization have become important directions for improving remote sensing image segmentation performance.

[0006] In response to the above technical problems, this paper proposes a Transformer remote sensing image semantic segmentation method based on an adaptive dynamic attention mechanism, which aims to further improve the performance of remote sensing image segmentation tasks, especially the accuracy and robustness in complex scenes and detail segmentation, by introducing technologies such as multi-scale feature fusion, adaptive dynamic weight adjustment, and multi-task joint optimization. .

[0007] The above disclosed technical solutions have at least the following technical problems: information loss and inconsistency problems may occur when the traditional method fuses features in multi-scale tasks. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0008] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a Transformer remote sensing semantic segmentation method based on an adaptive dynamic attention mechanism, which effectively fuses different scale features from the decoder and the encoder to solve the problems of information loss and inconsistency that may occur when traditional methods fuse features in multi-scale tasks.

[0009] To achieve the above object, the present invention provides the following technical solutions: A Transformer remote sensing semantic segmentation method based on an adaptive dynamic attention mechanism includes the following steps: obtaining a remote sensing image data set, and performing data annotation and data enhancement; adjusting the remote sensing image to a model input size, unifying pixel values, and generating a standardized data set; using a pre-trained ResNet-18 as an encoder, extracting deep features based on multi-layer convolution and pooling operations as multi-scale input; generating feature embedding based on a preset embedding module, and adding position encoding to enhance position information for decoding; correcting the decoding process using self-attention and cross-attention modules; extracting different scale features of the corrected features based on the encoder; combining different scale features with continuously iteratively updated adaptive weights to obtain fused feature values; constructing a multi-task joint loss function based on the problem to be solved; refining the fused feature values ​​through a segmentation module to generate prediction values ​​of different subtasks; and applying the prediction values ​​of different subtasks obtained from the adjusted decoding process to the problem to be solved.

[0010] In a preferred embodiment, the remote sensing image dataset is obtained, and data annotation and data enhancement are performed, specifically: creating pixel-level segmentation labels and target detection boxes for each remote sensing image, specifically including semantic segmentation, target detection, and instance segmentation; importing the remote sensing image into an annotation tool; annotating regional polygons and target detection boxes according to task requirements; saving the annotations in one-to-one correspondence with the image to obtain a processed remote sensing image dataset.

[0011] In a preferred embodiment, the pre-trained ResNet-18 is used as an encoder, and deep features are extracted based on multi-layer convolution and pooling operations as multi-scale input, specifically: the ImageNet pre-trained weights of ResNet-18 are used as an encoder for remote sensing image feature extraction; shallow and deep feature outputs are selected in ResNet-18 respectively: shallow output Conv2_x: captures low-level features of edges and textures; middle-layer output Conv3_x, Conv4_x: extracts target structure features; deep output Conv5_x: extracts high-level semantic features; the extracted features are saved based on a branching mechanism to form a multi-layer feature pyramid.

[0012] In a preferred embodiment, the feature embedding is generated based on a preset embedding module, and position coding is added to enhance position information for decoding, specifically: the embedding module includes a group of one-dimensional and two-dimensional convolution operation groups; a multi-layer feature pyramid is used as input features of the encoder to obtain output features; the encoder output features are mapped to a fixed embedding space, and position information is introduced based on the Fully Connected Layer for optimized embedding; the optimized feature embedding is used as a new input feature; and the new input feature is encoded based on sine and cosine function encoding.

[0013] In a preferred embodiment, the decoding process is corrected using self-attention and cross-attention modules, specifically: based on several linear transformation layers, the input features are mapped to a corresponding number of self-attention mechanisms; the attention weights are calculated and normalized using dot products; the value vectors are aggregated according to the attention weights, and the corrected features are generated.

[0014] In a preferred embodiment, the different scale features based on the features extracted and corrected by the encoder are specifically: gradually extracting feature maps of different scales through convolutional layers and pooling layers; correcting the extracted feature maps through a self-attention module; applying a cross-attention mechanism, combining feature maps from the decoder, and further correcting the features extracted by the encoder; fusing features of different scales, and merging multi-scale feature maps into a comprehensive feature representation through splicing or weighted averaging.

[0015] In a preferred embodiment, the different scale features are combined with the iteratively updated adaptive weights to obtain the fused feature value, specifically: each scale feature map is assigned an initial adaptive weight by the convolution layer and the fully connected layer; in the feature fusion stage, the different scale features are weighted and merged based on the current weights, and the feature maps of multiple scales are combined by weighted averaging and splicing to obtain the fused feature value.

[0016] In a preferred embodiment, the multi-task joint loss function is constructed based on the problem to be solved, specifically: using Cross-Entropy Loss to classify each pixel; using IoU loss to evaluate the overlap between the detection box and the true bounding box, and optimizing the accuracy of the target position; using Softmax or cross entropy loss function to classify the target to ensure the correctness of the target category; using Smooth L1 Loss to regress the bounding box of the target and optimize the accuracy of the target box; assigning weights to each subtask according to the priority and importance of the task; and weighted summing the loss functions of each subtask to obtain a total loss function.

[0017] In a preferred embodiment, the over-segmentation module refines and fuses feature values ​​to generate prediction values ​​for different subtasks, specifically: inputting the fused feature values ​​into the over-segmentation module; in the over-segmentation module, using convolution operations to refine the feature map to enhance the spatial resolution of the feature map; introducing the SE module to dynamically weight the information between channels, enhance the weights of key feature channels, and refine the feature map; adjusting the feature map in the spatial dimension based on non-local attention; applying the convolution layer to generate the classification prediction value of each pixel through the refined feature map, and outputting the probability distribution of the category to which each pixel belongs; based on the refined feature map, predicting the target detection task, the prediction includes the category, position and bounding box regression value of the target; fusing the prediction values ​​of different subtasks, performing multi-task joint output through the task-by-task output layer and the shared layer, and generating the final prediction result.

[0018] The technical effects and advantages of the Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism of the present invention are as follows: 1. The present invention designs a dual-stream decoder network to process spatial features and channel features respectively. This method introduces different decoding modules in both spatial and channel dimensions, which can simultaneously capture the spatial and channel dependencies of remote sensing images and improve the diversity and comprehensiveness of feature extraction; this dual-stream decoding method effectively avoids the bottleneck problem of traditional single decoders in complex feature fusion and enhances the model's ability to express different types of features.

[0019] 2. The present invention refines the fusion features through the segmentation module and gradually refines the spatial resolution of the feature map, thereby improving the shortcomings of traditional methods in segmentation details and target boundary extraction. This refinement method combines the spatial attention and channel attention mechanisms to make the classification of each pixel more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a structural schematic diagram of the Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Embodiment 1, Figure 1 The present invention provides a Transformer remote sensing semantic segmentation method based on an adaptive dynamic attention mechanism, comprising the following steps: S1, obtain remote sensing image datasets, and perform data annotation and data enhancement; The remote sensing image data set is obtained, and data annotation and data enhancement are performed, specifically: Create pixel-level segmentation labels and object detection boxes for each remote sensing image, including semantic segmentation, object detection, and instance segmentation; Import remote sensing images into annotation tools; Label the area polygons and target detection boxes according to task requirements; The annotations are saved in one-to-one correspondence with the images to obtain the processed remote sensing image dataset.

[0023] S2, adjusts the remote sensing image to the model input size, unifies the pixel values, and generates a standardized data set.

[0024] S3 uses the pre-trained ResNet-18 as the encoder and extracts deep features as multi-scale input based on multi-layer convolution and pooling operations; The pre-trained ResNet-18 is used as the encoder, and deep features are extracted based on multi-layer convolution and pooling operations as multi-scale input, specifically: Use the ImageNet pre-trained weights of ResNet-18 as the encoder for remote sensing image feature extraction; Select shallow and deep feature outputs in ResNet-18 respectively: Shallow output Conv2_x: captures low-level features of edges and textures; Middle layer output Conv3_x, Conv4_x: extract target structure features; Deep output Conv5_x: extracts high-level semantic features; The extracted features are saved based on the branching mechanism to form a multi-layer feature pyramid.

[0025] S4, generates feature embedding based on the preset embedding module, and adds position encoding to enhance position information for decoding; The feature embedding is generated based on the preset embedding module, and the position encoding is added to enhance the position information for decoding, specifically: The embedding module includes a set of one-dimensional and two-dimensional convolution operations; Use the multi-layer feature pyramid as the input feature of the encoder to obtain the output feature; Map the encoder output features to a fixed embedding space and introduce position information based on the Fully Connected Layer for optimized embedding; Embed the optimized features as new input features; Encode new input features based on sine-cosine function encoding.

[0026] S5, uses self-attention and cross-attention modules to modify the decoding process; The self-attention and cross-attention modules are used to modify the decoding process, specifically: Based on several linear transformation layers, the input features are mapped to the corresponding number of self-attention mechanisms; Use dot product to calculate attention weights and normalize; The value vectors are aggregated according to the attention weights and the modified features are generated.

[0027] S6, extracts features of different scales based on the corrected features from the encoder; The different scale features of the features extracted and corrected based on the encoder are specifically: Feature maps of different scales are gradually extracted through convolutional layers and pooling layers; The extracted feature map is modified through the self-attention module; Apply the cross-attention mechanism and combine the feature maps from the decoder to further modify the features extracted by the encoder; Features of different scales are fused and multi-scale feature maps are merged into a comprehensive feature representation by splicing or weighted averaging.

[0028] S7, combining different scale features and iteratively updating adaptive weights to obtain fused feature values; The fusion feature value is obtained by combining the features of different scales and updating the adaptive weights iteratively, specifically: Assign each scale feature map the initial adaptive weights from the convolutional layer and the fully connected layer; In the feature fusion stage, features of different scales are weighted and merged based on the current weights, and feature maps of multiple scales are combined by weighted averaging and splicing to obtain fused feature values.

[0029] S8, construct a multi-task joint loss function based on the problem to be solved; The multi-task joint loss function is constructed based on the problem to be solved, specifically: Use Cross-Entropy Loss to classify each pixel; Use IoU loss to evaluate the overlap between the detection box and the true bounding box to optimize the accuracy of the target location; Use Softmax or cross entropy loss function to classify the target and ensure the correctness of the target category; Use Smooth L1 Loss to regress the target’s bounding box and optimize the accuracy of the target box; Assign weights to each subtask based on its priority and importance; The loss functions of each subtask are weighted and summed to obtain the total loss function.

[0030] S9, refines the fusion feature values ​​through the segmentation module to generate prediction values ​​for different subtasks; The over-segmentation module refines the fusion feature values ​​to generate prediction values ​​for different subtasks, specifically: Input the fused feature values ​​into the over-segmentation module; In the over-segmentation module, convolution operations are used to refine the feature map and enhance the spatial resolution of the feature map; The SE module is introduced to dynamically weight the information between channels, enhance the weight of key feature channels, and refine the feature map; Adjusting feature maps in spatial dimensions based on non-local attention; Through the refined feature map, the convolution layer is applied to generate the classification prediction value of each pixel and output the probability distribution of the category to which each pixel belongs; Based on the refined feature map, the target detection task is predicted, including the target category, location, and bounding box regression value; The prediction values ​​of different subtasks are fused, and multi-task joint output is performed through the task-by-task output layer and the shared layer to generate the final prediction result.

[0031] S10, applying the predicted values ​​of different subtasks obtained according to the adjusted decoding process to the problem to be solved.

[0032] The specific calculation formula of the loss function is as follows:

[0033] in is the number of element, is the total dimension of the input vector.

[0034] The advantages of this application are as follows: Traditional attention mechanisms usually use fixed weights to process input features, while the present invention uses an adaptive dynamic attention mechanism to automatically adjust the attention weights by analyzing input features in real time. This dynamic adjustment can adaptively optimize feature representation according to the importance of different regions and features of the input image, thereby enhancing the model's adaptability to complex scenes; through this mechanism, the model can more accurately capture key areas in remote sensing images (such as object boundaries and details) and improve segmentation accuracy, especially in complex multi-scale scenes.

[0035] The present invention effectively fuses the different scale features from the decoder and encoder through a multi-scale feature fusion module, overcoming the information loss and inconsistency problems that may occur when the traditional method fuses features in multi-scale tasks. In particular, in the segmentation and classification tasks of remote sensing images, the fusion of features of different scales can effectively improve the robustness and accuracy of the model, especially in the boundary detection and detail classification of complex objects.

[0036] The present invention uses the over-segmentation module to refine the fused features, and improves the shortcomings of traditional methods in segmentation details and target boundary extraction by gradually refining the spatial resolution of the feature map. This refinement method combines the spatial attention and channel attention mechanisms to make the classification of each pixel more accurate. Through this refinement process, the model can generate more delicate and accurate segmentation results at output, improving the quality of the overall prediction.

[0037] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0038] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0039] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0040] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0041] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0042] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism, characterized by: The steps include: Obtain remote sensing image datasets and perform data annotation and data enhancement; Adjust the remote sensing image to the model input size, unify the pixel values, and generate a standardized data set; Use pre-trained ResNet-18 as the encoder, based on multi-layer convolution and pooling operations, to extract deep features as multi-scale input; Generate feature embedding based on the preset embedding module, and add position encoding to enhance position information for decoding; Use self-attention and cross-attention modules to modify the decoding process; Extracting features of different scales based on the corrected features from the encoder; Combine different scale features and iteratively update adaptive weights to obtain fused feature values; Construct a multi-task joint loss function based on the problem to be solved; The segmentation module refines the fusion feature values ​​to generate prediction values ​​for different subtasks; The predicted values ​​of different subtasks obtained according to the adjusted completed decoding process are applied to the problem to be solved.

2. The Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism according to claim 1, characterized in that: The remote sensing image data set is obtained, and data annotation and data enhancement are performed, specifically: Create pixel-level segmentation labels and object detection boxes for each remote sensing image, including semantic segmentation, object detection, and instance segmentation; Import remote sensing images into annotation tools; Label the area polygons and target detection boxes according to task requirements; The annotations are saved in one-to-one correspondence with the images to obtain the processed remote sensing image dataset.

3. The Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism according to claim 2 is characterized in that: The pre-trained ResNet-18 is used as the encoder, and deep features are extracted based on multi-layer convolution and pooling operations as multi-scale input, specifically: Use the ImageNet pre-trained weights of ResNet-18 as the encoder for remote sensing image feature extraction; Select shallow and deep feature outputs in ResNet-18 respectively: Shallow output Conv2_x: captures low-level features of edges and textures; Middle layer output Conv3_x, Conv4_x: extract target structure features; Deep output Conv5_x: extracts high-level semantic features; The extracted features are saved based on the branching mechanism to form a multi-layer feature pyramid.

4. The Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism according to claim 3 is characterized in that: The feature embedding is generated based on the preset embedding module, and the position encoding is added to enhance the position information for decoding, specifically: The embedding module includes a set of one-dimensional and two-dimensional convolution operations; Use the multi-layer feature pyramid as the input feature of the encoder to obtain the output feature; Map the encoder output features to a fixed embedding space and introduce position information based on the Fully Connected Layer for optimized embedding; Embed the optimized features as new input features; Encode new input features based on sine-cosine function encoding.

5. The Transformer remote sensing semantic segmentation method based on the adaptive dynamic attention mechanism according to claim 4 is characterized in that: The self-attention and cross-attention modules are used to modify the decoding process, specifically: Based on several linear transformation layers, the input features are mapped to the corresponding number of self-attention mechanisms; Use dot product to calculate attention weights and normalize; The value vectors are aggregated according to the attention weights and the modified features are generated.

6. The Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism according to claim 5, characterized in that: The different scale features of the features extracted and corrected based on the encoder are specifically: Feature maps of different scales are gradually extracted through convolutional layers and pooling layers; The extracted feature map is modified through the self-attention module; Apply the cross-attention mechanism and combine the feature maps from the decoder to further modify the features extracted by the encoder; Features of different scales are fused and multi-scale feature maps are merged into a comprehensive feature representation by splicing or weighted averaging.

7. The Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism according to claim 6, characterized in that: The fusion feature value is obtained by combining the features of different scales and updating the adaptive weights iteratively, specifically: Assign each scale feature map the initial adaptive weights from the convolutional layer and the fully connected layer; In the feature fusion stage, features of different scales are weighted and merged based on the current weights, and feature maps of multiple scales are combined by weighted averaging and splicing to obtain fused feature values.

8. The Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism according to claim 7, characterized in that: The multi-task joint loss function is constructed based on the problem to be solved, specifically: Use Cross-Entropy Loss to classify each pixel; Use IoU loss to evaluate the overlap between the detection box and the true bounding box to optimize the accuracy of the target location; Use Softmax or cross entropy loss function to classify the target and ensure the correctness of the target category; Use Smooth L1 Loss to regress the target’s bounding box and optimize the accuracy of the target box; Assign weights to each subtask based on its priority and importance; The loss functions of each subtask are weighted and summed to obtain the total loss function.

9. The Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism according to claim 8, characterized in that: The over-segmentation module refines the fusion feature values ​​to generate prediction values ​​for different subtasks, specifically: Input the fused feature values ​​into the over-segmentation module; In the over-segmentation module, convolution operations are used to refine the feature map and enhance the spatial resolution of the feature map; The SE module is introduced to dynamically weight the information between channels, enhance the weight of key feature channels, and refine the feature map; Adjusting feature maps in spatial dimensions based on non-local attention; Through the refined feature map, the convolution layer is applied to generate the classification prediction value of each pixel and output the probability distribution of the category to which each pixel belongs; Based on the refined feature map, the target detection task is predicted, including the target category, location, and bounding box regression value; The prediction values ​​of different subtasks are fused, and multi-task joint output is performed through the task-by-task output layer and the shared layer to generate the final prediction result.

10. The Transformer remote sensing semantic segmentation method based on adaptive dynamic attention mechanism according to claim 9, characterized in that: The specific calculation formula of the loss function is as follows: in is the number of element, is the total dimension of the input vector.

Images