A remote sensing image semantic segmentation method based on double-branch dynamic attention

Abstract

The application discloses a remote sensing image semantic segmentation method based on a double-branch dynamic attention, which comprises the following steps: obtaining a trained semantic segmentation model, inputting a remote sensing image to be semantically segmented into the semantic segmentation model, obtaining a semantic segmentation result of the remote sensing image, introducing a relationship attention module into a dynamic attention double-branch in a decoder module of the semantic segmentation model, obtaining local texture details and global context information, and additionally, designing a dynamic weight distribution mechanism branch to adaptively adjust the contribution of the two dynamic attention branches, thereby constructing more discriminative feature representation. The application not only improves the shortcomings of the traditional attention mechanism, but also realizes adaptive adjustment of the contribution of local information and global information in the remote sensing image, thereby solving the problems of multiple background noises and complex and diverse ground objects, providing a new solution for a high-resolution remote sensing image segmentation task, and improving the accuracy of remote sensing image semantic segmentation.

Description

Technical Field

[0001] This invention applies technologies from the fields of deep learning and computer vision, specifically inventing and applying a semantic segmentation method for remote sensing images based on bi-branch dynamic attention. Background Technology

[0002] Remote sensing images are typically captured by sensors from distant locations (usually aircraft or satellites). Recent advancements in sensors have enabled the capture of an increasing number of high-resolution remote sensing images globally, rich in spatial detail and semantic content. Semantic segmentation plays a crucial role in image processing by assigning semantic labels to each pixel in an image based on the category of ground objects or the regions they represent. Semantic segmentation of remote sensing images can serve a variety of applications, such as land cover classification, road extraction, and urban planning.

[0003] The rapid development of deep learning technology has driven a series of semantic segmentation methods based on Convolutional Neural Networks (CNNs) because they can automatically extract more information from image features and integrate richer contextual information, which is beneficial for feature representation and pattern recognition. However, their convolutional operations, with their limited receptive fields, are designed to extract local patterns and lack the ability to model global contextual information or long-range dependencies.

[0004] Introducing attention mechanisms is an effective way to reduce predictive category confusion without losing spatial information. Aggregating global statistics from the entire image allows for the embedding of scene information to highlight features with appropriate relevance or suppress features with erroneous relevance, achieving significant results in semantic segmentation tasks. However, the relevance of each query key pair is calculated independently, ignoring the relevance of other query key pairs. This can lead to erroneous relevance due to imperfect feature representations and interference from complex backgrounds, resulting in additional background noise and blurred attention weights. Therefore, typical attention-based methods are difficult to directly apply to semantic segmentation of remote sensing images. Summary of the Invention

[0005] The technical problem this invention aims to solve is how to improve the feature representation capability of remote sensing images by effectively fusing local texture details and global contextual information during feature extraction, while removing the noise and ambiguity introduced by independent correlation calculations in traditional attention mechanisms and considering the complex backgrounds and diverse land features of remote sensing images. This invention provides a remote sensing image semantic segmentation method based on bi-branch dynamic attention. The invention introduces a correlation attention module to seek consensus among all relevant vectors to refine the correlation and extend the general attention mechanism. Simultaneously, it acquires local and global information through local and global dynamic attention branches, and designs a dynamic weight allocation mechanism to fuse the output information of the two branches, thereby capturing more discriminative feature representations of land cover and improving the accuracy of the semantic segmentation model.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] A semantic segmentation method for remote sensing images based on bi-branch dynamic attention is described below:

[0008] A trained semantic segmentation model is obtained, and the remote sensing image to be semantically segmented is input into the semantic segmentation model to obtain the semantic segmentation result of the remote sensing image. The semantic segmentation model includes an encoder module and a decoder module.

[0009] In the encoder module, the remote sensing image to be semantically segmented is first input into the backbone network for feature extraction. The features output by the backbone network are used as coarse feature representations, and the features output by the low layer of the backbone network are used as low layer feature representations.

[0010] The decoder module includes a local dynamic attention branch, a global dynamic attention branch, and a dynamic weight allocation mechanism branch. Both the local and global dynamic attention branches contain a relational attention module. In the decoder module, the coarse feature representation is first input into the local dynamic attention branch to obtain a local feature representation. Then, the coarse feature representation is input into the global dynamic attention branch to obtain a global feature representation. Finally, the coarse feature representation is input into the dynamic weight allocation mechanism branch to obtain scalar weights for the local and global feature representations. The scalar weights of the local feature representation are multiplied element-wise with the global feature representation to obtain a global refined feature representation. The scalar weights of the local feature representation are multiplied element-wise with the local feature representation to obtain a local refined feature representation. The local refined feature representation is added element-wise with the global refined feature representation to obtain a refined feature representation. The refined feature representation and the coarse feature representation are added element-wise to obtain an enhanced feature representation. The enhanced feature representation and the low-level feature representation are concatenated along the channel direction to obtain an output feature representation. The output feature representation is upsampled to obtain the semantic segmentation result of the remote sensing image.

[0011] Preferably, in the local dynamic attention branch, the coarse feature representation is sliced ​​to obtain a patch representation. The patch representation is passed through a convolutional layer to obtain a first local feature map. The convolutional layer has a kernel size of 1, a stride of 1, and padding of 0. The patch representation is then input into the convolutional layer to obtain a second local feature map. The first local feature map and the second local feature map are multiplied element-wise to obtain initial local attention weights. These initial local attention weights are then input into the relational attention module for denoising to obtain denoised local attention weights. The denoised local attention weights and the initial local attention weights are then multiplied together. The attention weights are added element-wise to obtain updated local attention weights. The updated local attention weights are then processed through a Softmax operation to obtain enhanced local attention weights. The patch representation is passed through a convolutional layer to obtain a local value vector. The enhanced local attention weights and the local value vector are multiplied element-wise to obtain a third local feature map. The third local feature map is passed through another convolutional layer to obtain a fourth local feature map. The fourth local feature map is added element-wise to the patch representation to obtain a fifth local feature map. The fifth local feature map is then patched back to obtain the final local feature representation.

[0012] Preferably, in the global dynamic attention branch, the patch representation is sequentially passed through a convolutional layer and an average pooling layer to obtain a global descriptor. The patch representation is then passed through a convolutional layer to obtain a first global feature map. The first global feature map and the global descriptor are multiplied element-wise to obtain initial global attention weights. The initial global attention weights are input into the relational attention module for denoising to obtain denoised global attention weights. The denoised global attention weights and the initial global attention weights are added element-wise to obtain updated global attention weights. The updated global attention weights are subjected to a Softmax operation to obtain enhanced global attention weights. The enhanced local attention weights and the global descriptor are multiplied element-wise to obtain a second global feature map. The second global feature map is passed through a convolutional layer to obtain a third global feature map. The third global feature map is added element-wise to the patch representation to obtain a fourth global feature map. The fourth global feature map is then sliced ​​and restored to obtain the final global feature representation.

[0013] Preferably, in the dynamic weight allocation mechanism branch, the input feature representation is input into an average pooling layer to obtain a first intermediate feature representation. The first intermediate feature representation is then passed through two convolutional layers for dimensionality reduction to obtain a second intermediate feature representation. The second intermediate feature representation is then subjected to a Softmax operation to obtain the scalar weights of the input feature representation.

[0014] Preferably, in the relational attention module, the initial attention weight matrix A is sequentially passed through a fully connected layer to obtain a query vector Q and a key vector K. The query vector and the key vector are then passed through a normalization layer to obtain a first intermediate vector Q′ and a second intermediate vector K′. Each column of the initial attention weight matrix is ​​used as a value vector V. The value vector is then passed through the aforementioned normalization layer to obtain a third intermediate vector V′. An attention operation is performed on the first intermediate vector Q′, the second intermediate vector K′, and the third intermediate vector V′ to obtain a denoised attention weight A′. The functional form of the denoised attention weight A′ is:

[0015]

[0016] Where f represents the Softmax operation, W′ a represents the linear transformation weight, used to adjust aggregation correlation; C represents a constant; T represents matrix transpose.

[0017] Preferably, the patch is 32×32 in size.

[0018] Preferably, the dynamic weight allocation mechanism branches follow α.l +α g =1 constraint, where α l The local feature represents the scalar weight; α g The global feature represents the scalar weight.

[0019] Preferably, the backbone network is a ResNet-50 model.

[0020] Preferably, the training samples of the semantic segmentation model are labeled training data, which are obtained by processing original remote sensing images. The original remote sensing images are from the ISPRS Vaihingen dataset and the ISPRS Potsdam dataset. The semantic segmentation model uses cross-entropy loss as the loss function during training to update the network parameters of the semantic segmentation model.

[0021] Preferably, the remote sensing image is a high-resolution remote sensing image with a spatial resolution of less than 1m.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This invention discloses a semantic segmentation method for remote sensing images based on bi-branch dynamic attention. Addressing the problem that traditional attention mechanisms, with their independent correlation calculations, fail to consider the complex background of remote sensing images, leading to noise and ambiguity in attention weights, this invention constructs a relational attention module. This module seeks consensus among all correlation vectors to enhance appropriate correlations and suppress erroneous correlations, thereby promoting feature aggregation. Furthermore, by introducing the relational attention module into local and global dynamic attention branches, local texture details and global contextual information are acquired, respectively. Moreover, considering the diverse needs of complex geographic objects for local texture details and global contextual information, a dynamic weight allocation mechanism is designed to adaptively adjust the contributions of the two branches, thus constructing a more discriminative feature representation. This invention solves the problems of high background noise and complex and diverse ground features in remote sensing images, providing a new solution for high-resolution remote sensing image segmentation tasks and improving the performance of semantic segmentation of remote sensing images. Attached Figure Description

[0024] Figure 1 This is a diagram of the DBDAN model structure, a remote sensing image semantic segmentation method based on bi-branch dynamic attention.

[0025] Figure 2 This is a schematic diagram of the sub-module structure of the DBDAN model, a remote sensing image semantic segmentation method based on bi-branch dynamic attention.

[0026] Figure 3This is a flowchart illustrating the training and testing process of the DBDAN model in an embodiment of the present invention.

[0027] Figure 4 This is a visualization of the test results in an embodiment of the present invention. Detailed Implementation

[0028] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0029] Attention mechanisms, which embed scene information to highlight features with appropriate relevance or suppress features with erroneous relevance, are widely used in semantic segmentation of remote sensing images. However, the relevance of each query key pair is calculated independently, ignoring the relevance of other query key pairs. Generally, if a key is highly relevant to a query, its adjacent keys should also be highly relevant to that query. Otherwise, the relevance may be related to noise. Based on this observation, this invention designs a relevance attention module that refines relevance by seeking consensus among all relevant vectors, thereby extending general attention mechanisms. Simultaneously, achieving a trade-off between the effectiveness of feature embedding and the effectiveness of spatial localization is crucial for semantic segmentation of remote sensing images. This invention designs local dynamic attention branches and global dynamic attention branches to acquire local texture details and global contextual information. Furthermore, considering the different needs of complex and diverse geographic objects for local texture details and global background information, this invention designs a dynamic weight allocation mechanism to fuse the outputs of the two branches to capture more discriminative feature representations. The core of this invention is the proposal of a deep network based on feature aggregation using dual-branch dynamic attention, namely the semantic segmentation model DBDAN.

[0030] This invention provides a remote sensing image semantic segmentation method based on bi-branch dynamic attention. Specifically, it involves acquiring a trained semantic segmentation model (DBDAN), inputting the remote sensing image to be semantically segmented into the semantic segmentation model, and obtaining the semantic segmentation result of the remote sensing image. The semantic segmentation model includes an encoder module and a decoder module. The image used in this invention is preferably a remote sensing image, and more preferably a high-resolution remote sensing image with a spatial resolution of less than 1 meter.

[0031] The following provides a detailed description of the specific structure and principles of the aforementioned semantic segmentation model, DBDAN. Figure 1 The diagram shows the overall structure of the DBDAN model, which includes an encoder module and a decoder module. The encoder module is used to extract semantic features, while the decoder is used to enhance the coarse feature representation output by the encoder module through local dynamic attention branches, global dynamic attention branches, and dynamic weight allocation mechanisms. The enhanced feature representation and the low-level feature representation are then concatenated along the channel dimension to restore the image spatial resolution.

[0032] It should be noted that the remote sensing images to be semantically segmented in this embodiment of the invention need to undergo data preprocessing first, and then be processed in batches during the training phase. The semantic segmentation model is trained using training samples from each batch. The dimensions of the remote sensing images are (B×3×H×W), where H and W are the height and width of the remote sensing images, respectively; B is the batch size of the input, and the value of B depends on the sample size of each batch during the training phase, but is not limited in this embodiment of the invention; during the prediction phase, the value of B can be set to 1. The remote sensing images to be semantically segmented are derived from the ISPRSVaihingen dataset and the ISPRS Potsdam dataset.

[0033] In the encoder module of DBDAN of the present invention, the remote sensing image to be semantically segmented is first input into the backbone network for feature extraction. The features output by the backbone network are used as coarse feature representations, and the features output by the low layer of the backbone network are used as low layer feature representations.

[0034] In this embodiment, the intermediate processes of the encoder module and the dimensions of each parameter are further described to help those skilled in the art better understand the principle of using this encoder module. First, the remote sensing image to be semantically segmented is input into the backbone network for feature extraction. The features output by the backbone network are used as the coarse feature representation R, with dimensions (B×H′×W′×C). The features output by the lower layers of the backbone network are used as the lower-level feature representation L, with dimensions (B×H″×W″×C), where C is the number of feature channels in the coarse feature representation.

[0035] The DBDAN decoder module of the present invention mainly includes a local dynamic attention branch, a global dynamic attention branch, and a dynamic weight allocation mechanism branch. Both the local dynamic attention branch and the global dynamic attention branch have a relational attention module. In the decoder module, the coarse feature representation is first input into the local dynamic attention branch to obtain the local feature representation. Then, the coarse feature representation is input into the global dynamic attention branch to obtain the global feature representation. Finally, the coarse feature representation is input into the dynamic weight allocation mechanism branch to obtain scalar weights for both the local and global feature representations. The global feature representation scalar weights are then multiplied element-wise with the global feature representation to obtain the global refined feature representation. The local feature representation scalar weights are then multiplied element-wise with the local feature representation to obtain the local refined feature representation. The local refined feature representation is then added element-wise with the global refined feature representation to obtain the refined feature representation. Finally, the refined feature representation and the coarse feature representation are added element-wise to obtain the enhanced feature representation. The enhanced feature representation and the low-level feature representation are concatenated along the channel direction to obtain the output feature representation. The output feature representation is then upsampled to obtain the semantic segmentation result of the remote sensing image.

[0036] In this embodiment, the intermediate processes of the encoder module and the dimensions of each parameter are further described to help those skilled in the art better understand the principle of using this decoder module. First, the coarse feature representation is input into the local dynamic attention branch to obtain the local feature representation D. l The coarse feature representation is input into the global dynamic attention branch to obtain the global feature representation D. g Both the local and global feature representations have dimensions of (B×H′×W′×C). The coarse feature representation is input into the dynamic weight allocation mechanism branch to obtain the local feature representation scalar weight α used to adjust the contribution of local and global information. l and global feature representation scalar weight α g The global feature representation is element-wise multiplied by its scalar weights to obtain the global refined feature representation. The local feature representation is then element-wise multiplied by its scalar weights to obtain the local refined feature representation. Finally, the local refined feature representation is element-wise added to the global refined feature representation to obtain the refined feature representation S = D. l ×α l +D g ×α gThe refined feature representation and the coarse feature representation are added element-wise to obtain the enhanced feature representation P, which has the same dimensions (B×H′×W′×C). The enhanced feature representation and the low-level feature representation are concatenated along the channel direction to obtain the output feature representation. The output feature representation is then upsampled to the size of the input remote sensing image to be semantically segmented to obtain the semantic segmentation result of the remote sensing image.

[0037] In the local dynamic attention branch of this invention, the coarse feature representation is sliced ​​to obtain a patch representation. The patch representation is passed through a convolutional layer to obtain a first local feature map. The convolutional layer has a kernel size of 1, a stride of 1, and padding of 0. The patch representation is then input into another convolutional layer to obtain a second local feature map. The first local feature map and the second local feature map are multiplied element-wise to obtain initial local attention weights. The initial local attention weights are input into the relational attention module for denoising to obtain denoised local attention weights. The denoised local attention weights and the initial local attention weights are added element-wise to obtain updated local attention weights. The updated local attention weights are then subjected to a Softmax operation to obtain enhanced local attention weights. The patch representation is passed through a convolutional layer to obtain a local value vector. The enhanced local attention weights and the local value vector are multiplied element-wise to obtain a third local feature map. The third local feature map is passed through a convolutional layer to obtain a fourth local feature map. The fourth local feature map and the patch representation are added element-wise to obtain a fifth local feature map. The fifth local feature map is then patched back to obtain the final local feature representation.

[0038] It should be noted that the main purpose of the local dynamic attention branch in this embodiment is to extract local information from the remote sensing image. Furthermore, a relational attention module is introduced to enhance the extraction of local feature representations. Specifically, as... Figure 2 As shown, the coarse feature representation R∈(B×H′×W′×C) is sliced ​​to obtain a series of patch representations X, whose dimensions are (B×(N)). h ×N w )×(h×w)×C), where N h and N wLet represent the number of patches along the height and width dimensions after slicing, respectively. The patch representation is passed through a convolutional layer to obtain a first local feature map. The patch representation is then input into another convolutional layer to obtain a second local feature map. The first and second local feature maps are multiplied element-wise to obtain initial local attention weights. These initial local attention weights are then input into the relational attention module for denoising to obtain denoised local attention weights. The denoised local attention weights and the initial local attention weights are then added element-wise to obtain updated local attention weights. These updated local attention weights are then subjected to a Softmax operation to obtain enhanced local attention weights. The patch representation is passed through a convolutional layer to obtain a local value vector. The enhanced local attention weights and the local value vector are then multiplied element-wise to obtain a third local feature map. The third local feature map is passed through a convolutional layer to obtain a fourth local feature map. The fourth local feature map is then added element-wise to the patch representation to obtain a fifth local feature map. The fifth local feature map is then patched back to obtain the final local feature representation D. l Its spatial dimensions are restored to (B×H′×W′×C).

[0039] In the global dynamic attention branch of this invention, the patch representation is sequentially passed through a convolutional layer and an average pooling layer to obtain a global descriptor. The patch representation is then passed through a convolutional layer to obtain a first global feature map. The first global feature map and the global descriptor are multiplied element-wise to obtain the initial global attention weights. The initial global attention weights are input into the relational attention module for denoising to obtain denoised global attention weights. The denoised global attention weights and the initial global attention weights are added element-wise to obtain updated global attention weights. The updated global attention weights are subjected to a Softmax operation to obtain enhanced global attention weights. The enhanced local attention weights are multiplied element-wise with the global descriptor to obtain a second global feature map. The second global feature map is passed through one of the aforementioned convolutional layers to obtain a third global feature map. The third global feature map is added element-wise with the patch representation to obtain a fourth global feature map. The fourth global feature map is then sliced ​​and restored to obtain the final global feature representation.

[0040] It should be noted that the main purpose of the global dynamic attention branch in this embodiment is to extract global information from the remote sensing image. Furthermore, a relational attention module is introduced to enhance the extraction of global feature representations. Specifically, as... Figure 2As shown, in the global dynamic attention branch, the patch representation X is passed through a convolutional layer and an average pooling layer to obtain the global descriptor Z. The patch representation is then passed through a convolutional layer to obtain the first global feature map. The first global feature map and the global descriptor are multiplied element-wise to obtain the initial global attention weights. The initial global attention weights are input into the relational attention module for denoising to obtain the denoised global attention weights. The denoised global attention weights and the initial global attention weights are added element-wise to obtain the updated global attention weights. The updated global attention weights are then subjected to a Softmax operation to obtain the enhanced global attention weights. The enhanced local attention weights are then multiplied element-wise with the global descriptor to obtain the second global feature map. The second global feature map is passed through a convolutional layer to obtain the third global feature map. The third global feature map is added element-wise with the patch representation to obtain the fourth global feature map. The fourth global feature map is then sliced ​​and restored to obtain the final global feature representation D. g Its spatial dimensions are restored to (B×H′×W′×C).

[0041] The relational attention module in this invention is applied to both the local dynamic attention branch and the global dynamic attention branch. The input is the initial attention weight matrix generated from the query key pairs in both branches, and the output is enhanced attention weights after removing extra background noise and ambiguity. It leverages the principle that high relevance of query key pairs should result in similar relationships between adjacent pixels of the query and key, thereby improving the generation of the relevance map. Specifically, it seeks relevance consistency around each key to enhance the appropriate relevance extracted from relevant query key pairs and suppress erroneous relevance in irrelevant pairs.

[0042] In the relational attention module of this invention, the initial attention weight matrix A is sequentially passed through a fully connected layer to obtain a query vector Q and a key vector K. The query vector and the key vector are then passed through a normalization layer to obtain a first intermediate vector Q′ and a second intermediate vector K′. Each column of the initial attention weight matrix is ​​used as a value vector V. The value vector is then passed through a normalization layer to obtain a third intermediate vector V′. The first intermediate vector Q′, the second intermediate vector K′, and the third intermediate vector V′ are then subjected to an attention operation to obtain the denoised attention weight A′. The functional form of the denoised attention weight A′ is as follows:

[0043]

[0044] Where f represents the Softmax operation, W′ a represents the linear transformation weight, used to adjust aggregation correlation; C represents a constant; T represents matrix transpose.

[0045] In this embodiment, the main purpose of the relational attention module is to denoise the attention operations in the local dynamic attention branch and the global dynamic attention branch. Specifically, as follows: Figure 2 As shown, each column of the initial attention weight matrix A is first regarded as a sequence of related vectors, and a linear transformation is performed on it to generate a query vector Q and a key vector K, both of which have a dimension of N×D. The sequence of related vectors itself is used as a value vector V, with a dimension of N×N. Then, the query vector Q, the key vector K, and the value vector V are normalized to obtain vectors Q′, K′, and V′. Finally, attention operations are performed to generate denoised attention weights.

[0046] In the dynamic weight allocation mechanism branch of this invention, the input feature representation is input into an average pooling layer to obtain a first intermediate feature representation. The first intermediate feature representation is then passed through two convolutional layers for dimensionality reduction to obtain a second intermediate feature representation. The second intermediate feature representation is then subjected to a Softmax operation to obtain the scalar weights of the input feature representation.

[0047] In this embodiment, the purpose of the dynamic weight allocation mechanism is to adaptively integrate local and global information, thereby addressing the different needs of complex and diverse features in remote sensing images for local texture details and global context. Specifically, as... Figure 1 As shown, the dynamic weight allocation mechanism branch generates scalar weights to adjust global and local information. This module first performs average pooling on the input feature representation, then performs dimensionality reduction through two convolutional layers, and finally performs a softmax operation to obtain the global feature representation scalar weights α used to adjust global and local information. g Local feature representation scalar weight α l The dynamic weight allocation mechanism branches follow α. l +α g The =1 constraint is used to compress the kernel space and simplify the learning of the two weights. Where α... l α g The function form is:

[0048] [α l α g ]=σ(H d (δ(H r (AvgPool(R)))))

[0049] Where δ is the ReLU activation function, and AvgPool represents the average pooling layer; H d For convolutional layers; σ is the sigmoid activation function; H r It is a convolutional layer.

[0050] It should be noted that, in the embodiments of the present invention, the above-mentioned convolutional layers are all standard convolutional layers with a kernel size of 1, a stride of 1, and padding of 0.

[0051] It should be noted that, in this embodiment of the invention, the patch is represented by a size of 32×32.

[0052] It should be noted that the aforementioned semantic segmentation model, DBDAN, is pre-trained using labeled training data before being used for actual semantic segmentation. Data augmentation can be performed on the training data to expand the training sample. The loss function used for training the semantic segmentation model is cross-entropy loss; the specific training process can be found in existing semantic segmentation model training methods and will not be elaborated further.

[0053] It should be noted that the encoder module used in the embodiments of the present invention can be from any open-source and general-purpose model, and can be selected according to the actual situation. In the embodiments of the present invention, the ResNet50 model is used as the backbone network, and the ResNet50 is loaded with pre-trained weights learned on the Image-Net dataset. Its implementation method belongs to the prior art. For details, please refer to the prior art literature: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778. It will not be elaborated further here.

[0054] It should be noted that the pre-trained ImageNet dataset used in the embodiments of the present invention comes from the existing technical literature Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge[J]. International journal of computer vision, 2015, 115: 211-252. In the ImageNet dataset, each training sample is in the form of several images and the true category of the images. The true category of the image is used as the label value. The model inputs the image and outputs the predicted category of the image. This will not be elaborated further.

[0055] It should be noted that the ISPRS Vaihingen and ISPRS Potsdam datasets used in the embodiments of this invention can be from any and universally applicable remote sensing image datasets, and can be selected according to actual conditions. The implementation methods of the ISPRS Vaihingen and ISPRS Potsdam datasets used in the embodiments of this invention are both existing technologies. Specifically, the ISPRS Vaihingen dataset can be found at: https: / / isprs.org / education / benchmarks / UrbanSemLab / 2d-sem-label-vaihingen.aspx; the ISPRS Potsdam dataset can be found at:

[0056] https: / / www.isprs.org / education / benchmarks / UrbanSemLab / 2d-sem-label-potsdam.aspx; I won't go into details about that.

[0057] The above-described remote sensing image semantic segmentation method based on bi-branch dynamic attention will be applied to a specific embodiment to demonstrate its technical effects.

[0058] Example

[0059] The network structure of the semantic segmentation model DBDAN used in this embodiment is as described above and will not be repeated here. Figure 3 As shown, the overall process of semantic segmentation of remote sensing images can be divided into three stages: data preprocessing, model training, and image prediction.

[0060] 1. Data Preprocessing Stage

[0061] For the obtained raw remote sensing images (using the ISPRS Vaihingen and ISPRS Potsdam datasets as examples in this embodiment), data preprocessing is performed. First, the raw remote sensing images are cut into 512×512 size, and then the cut images are randomly rotated and flipped to perform data augmentation.

[0062] 2. Model Training

[0063] Step 1. Select a subset of samples from the ISPRS Vaihingen and ISPRS Potsdam datasets to construct the training dataset, and divide the training dataset into batches of a fixed batch size, with a total of N.

[0064] Step 2. Select a batch of training samples with index i sequentially from the training dataset, and train the semantic segmentation model DBDAN using each batch of training samples. During training, calculate the cross-entropy loss function for each training sample. And based on the total loss of all training samples in the batch The network parameters in the entire model are adjusted until all batches of the training dataset have participated in the model training. After reaching the specified number of iterations, the model converges and training is complete, where i∈{0,1,…,N}.

[0065] 3. Image Prediction

[0066] A subset of samples was reselected from the ISPRS Vaihingen and ISPRS Potsdam datasets to construct a test dataset. The remote sensing images in the test dataset were used as input to the pre-trained semantic segmentation model DBDAN. Finally, a probability vector with each class of changed pixels was predicted. The class with the highest probability was selected as the change classification output through activation functions such as Sigmoid, thereby achieving semantic segmentation.

[0067] In this embodiment, the test visualization results are as follows: Figure 4 As shown in Table 1, the test data results are as follows:

[0068] Table 1 Test Data Results

[0069] Vaihingen 91.57 84.64 91.74 Potsdam 93.28 87.86 91.94

[0070] Depend on Figure 4 As shown in Table 1, the semantic segmentation model of this invention can effectively process remote sensing images and produce segmentation results. Relying on the relevance attention module, it effectively solves the shortcomings of traditional attention mechanisms applied to the visual processing of remote sensing images. Through local dynamic attention branches and global dynamic attention branches, it makes full use of local texture details and global context information for feature aggregation. At the same time, it introduces a dynamic weight allocation mechanism to further enhance the feature aggregation capability, thereby improving the semantic segmentation performance of remote sensing images and providing a new solution for the application of attention mechanisms in the field of remote sensing image segmentation.

[0071] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on bi-branch dynamic attention, characterized in that, The specific process is as follows: Obtain a trained semantic segmentation model, input the remote sensing image to be semantically segmented into the semantic segmentation model, and obtain the semantic segmentation result of the remote sensing image. The semantic segmentation model includes an encoder module and a decoder module. In the encoder module, the remote sensing image to be semantically segmented is first input into the backbone network for feature extraction. The features output by the backbone network are used as coarse feature representations, and the features output by the low layer of the backbone network are used as low layer feature representations. The decoder module includes a local dynamic attention branch, a global dynamic attention branch, and a dynamic weight allocation mechanism branch. Both the local and global dynamic attention branches contain a relational attention module. In the decoder module, the coarse feature representation is first input into the local dynamic attention branch to obtain a local feature representation. Then, the coarse feature representation is input into the global dynamic attention branch to obtain a global feature representation. Finally, the coarse feature representation is input into the dynamic weight allocation mechanism branch to obtain scalar weights for the local and global feature representations. The scalar weights of the local feature representation are multiplied element-wise with the global feature representation to obtain a global refined feature representation. The scalar weights of the local feature representation are multiplied element-wise with the local feature representation to obtain a local refined feature representation. The local refined feature representation is added element-wise with the global refined feature representation to obtain a refined feature representation. The refined feature representation and the coarse feature representation are added element-wise to obtain an enhanced feature representation. The enhanced feature representation and the low-level feature representation are concatenated along the channel direction to obtain an output feature representation. The output feature representation is upsampled to obtain the semantic segmentation result of the remote sensing image.

2. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 1, characterized in that, In the local dynamic attention branch, the coarse feature representation is sliced ​​to obtain a patch representation. This patch representation is then passed through a convolutional layer to obtain a first local feature map. The convolutional layer has a kernel size of 1, a stride of 1, and padding of 0. The patch representation is then input into another convolutional layer to obtain a second local feature map. The first and second local feature maps are then multiplied element-wise to obtain initial local attention weights. These initial local attention weights are then input into the relational attention module for denoising to obtain denoised local attention weights. Finally, the denoised local attention weights and the initial local attention weights are... The element-wise addition is performed again to obtain the updated local attention weights. The updated local attention weights are then subjected to a Softmax operation to obtain enhanced local attention weights. The patch representation is then passed through a convolutional layer to obtain a local value vector. The enhanced local attention weights and the local value vector are then multiplied element-wise to obtain a third local feature map. The third local feature map is then passed through another convolutional layer to obtain a fourth local feature map. The fourth local feature map is then added element-wise with the patch representation to obtain a fifth local feature map. Finally, a patch restoration operation is performed on the fifth local feature map to obtain the final local feature representation.

3. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 2, characterized in that, In the global dynamic attention branch, the patch representation is passed through a convolutional layer and an average pooling layer to obtain a global descriptor. The patch representation is then passed through a convolutional layer to obtain a first global feature map. The first global feature map and the global descriptor are multiplied element-wise to obtain initial global attention weights. The initial global attention weights are input into the relational attention module for denoising to obtain denoised global attention weights. The denoised global attention weights and the initial global attention weights are added element-wise to obtain updated global attention weights. The updated global attention weights are subjected to a Softmax operation to obtain enhanced global attention weights. The enhanced local attention weights and the global descriptor are multiplied element-wise to obtain a second global feature map. The second global feature map is passed through a convolutional layer to obtain a third global feature map. The third global feature map is added element-wise to the patch representation to obtain a fourth global feature map. The fourth global feature map is sliced ​​and restored to obtain the final global feature representation.

4. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 3, characterized in that, In the dynamic weight allocation mechanism branch, the input feature representation is input into an average pooling layer to obtain a first intermediate feature representation. The first intermediate feature representation is then passed through two convolutional layers for dimensionality reduction to obtain a second intermediate feature representation. The second intermediate feature representation is then subjected to a Softmax operation to obtain the scalar weights of the input feature representation.

5. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 3, characterized in that, In the relational attention module, the initial attention weight matrix is... The query vector is obtained by passing the data through a fully connected layer. Key vector The query vector and the key vector are passed through a normalization layer to obtain the first intermediate vector. Second intermediate vector Each column of the initial attention weight matrix is ​​used as a value vector. The value vector is passed through a normalization layer to obtain a third intermediate vector. , the first intermediate vector The second intermediate vector The third intermediate vector Perform attention operations to obtain the denoised attention weights. The denoised attention weights The function form is: ， in, This indicates the Softmax operation. This represents the linear transformation weight, used to adjust aggregation correlation; Represents a constant; This indicates the matrix transpose.

6. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 2, characterized in that, The patch represents a size of .

7. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 1, characterized in that, The dynamic weight allocation mechanism branch follows Constraints, among which, The local feature represents the scalar weight; The global feature represents the scalar weight.

8. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 1, characterized in that, The backbone network is the ResNet-50 model.

9. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 1, characterized in that, The training samples of the semantic segmentation model are labeled training data, which are obtained by processing original remote sensing images. The original remote sensing images are from the ISPRS Vaihingen dataset and the ISPRS Potsdam dataset. The semantic segmentation model uses cross-entropy loss as the loss function during training to update the network parameters of the semantic segmentation model.

10. The remote sensing image semantic segmentation method based on bi-branch dynamic attention as described in claim 1, characterized in that, The remote sensing image is a high-resolution remote sensing image with a spatial resolution of less than 1m.

Images