A Semantic Segmentation Method for Remote Sensing Images Based on Dual-Domain Optimized Class-Aware Attention

Abstract

This invention discloses a semantic segmentation method for remote sensing images based on dual-domain optimized class-aware attention. Specifically, it involves acquiring a trained semantic segmentation model with encoder and decoder modules, inputting the remote sensing image to be semantically segmented into the model, and obtaining the semantic segmentation result. The decoder module includes a semantic domain optimization module and a spatial domain optimization module, with the spatial domain optimization module incorporating a nested attention module. In the semantic domain, this invention uses class confidence as the scaling standard to extract multi-confidence scale class representations, shortening the distance between pixels and their corresponding classes to distinguish intra-class differences. In the spatial domain, this invention uses nested attention to find consensus in pixel-class relationships, enhancing correct correlations and suppressing erroneous correlations. This invention, combined with dual-domain optimized class-aware attention, provides a novel solution for high-resolution remote sensing image segmentation tasks, improving the accuracy of semantic segmentation of remote sensing images.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision, specifically relating to a semantic segmentation method for remote sensing images based on dual-domain optimized class-aware attention. Background Technology

[0002] Remote sensing imagery, as a special type of image, is characterized by its rich information, high dimensionality, and complexity, and is widely used in resource surveys, environmental monitoring, and urban planning. In recent years, the rapid development of remote sensing technology has made it possible to conduct detailed observations of surface features using remote sensing images, but it has also brought challenges to image processing. Traditional remote sensing image segmentation methods can generally be divided into thresholding methods, region growing methods, edge detection methods, clustering methods, and model-driven methods. Limited by the manual extraction of features, these methods struggle to achieve satisfactory results in segmenting high-resolution remote sensing images, and are also susceptible to noise interference due to their weak robustness.

[0003] In recent years, the rise of deep learning techniques, especially convolutional neural networks (CNNs), has yielded significant results in the field of remote sensing image segmentation. For example, fully convolutional networks (FCNs) are an end-to-end supervised learning method that replaces fully connected layers with convolutional layers for pixel-level classification. However, due to their fixed geometry, FCNs are limited by local receptive fields and short-range contextual information. To capture longer-range dependencies, researchers have subsequently designed a series of context modeling methods, including spatial context modeling and relational context modeling.

[0004] Spatial context modeling methods aim to enhance pixel feature representations by integrating spatial context information across multiple scales. For example, PSPNet uses pyramid pooling modules to process input images at multiple scales; DeepLab employs dilated convolutions to increase the receptive field and preserve more detail; and DMNet adaptively captures local and global contextual information through dynamic multi-scale filters. However, these methods focus on capturing homogeneous contextual dependencies and are likely to ignore class differences, potentially leading to unreliable context if confused classes occur in the scene.

[0005] Relational context modeling methods, based on spatial attention mechanisms, compute pixel-level similarity in images and weighted aggregate heterogeneous contextual information, thus achieving significant performance in semantic segmentation. NonLocal was the first to apply self-attention mechanisms to the image domain, enabling a single feature at any location to perceive features at all other locations. Based on this, CBAM and DANet designed serial and parallel spatial and channel attention mechanisms, respectively, to model features. Furthermore, FLANet and SAPNet highlighted the attention deficit problem, finding that it can impair the ability to represent features and cannot be solved by simply stacking attention blocks in different directions. To address this, these two methods reconstructed the self-attention mechanism into global attention that can be captured in a single block. Considering the complex background and foreground-background imbalance characteristics of remote sensing images, these methods often introduce a large amount of background context due to dense affinity operations, leading to suboptimal segmentation performance.

[0006] Unlike pixel-level relation modeling, several studies have generated class representations and used them as representative features to mitigate background contextual interference. For example, ACFNet and OCRNet compute the relationship between pixels and object regions by aggregating the representations of pixels within object regions and use a weighted aggregation of all object region representations to enhance the representation of each pixel. CCANet includes a novel class-constrained coarse-grained attention that can constrain class information to obtain explicit long-range contextual information. CPNet constructs a contextual prior layer with embedded affinity loss supervision to explicitly capture intra-class and inter-class contextual relevance. However, these methods have two significant drawbacks: (1) in ordinary attention, independent relevance computation leads to noise and ambiguity in pixel-class relation weights; (2) class representations with single confidence cannot distinguish intra-class differences, resulting in semantically confusing class-level semantic modeling. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to extract richer class representations to distinguish intra-class differences and mine consensus in pixel-class relationships to optimize pixel-class relationship weights.

[0008] The specific technical solution adopted in this invention is as follows:

[0009] A remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention is proposed. The specific steps are as follows: a trained semantic segmentation model is obtained, the remote sensing image to be semantically segmented is input into the semantic segmentation model, and the semantic segmentation result of the remote sensing image is obtained.

[0010] The semantic segmentation model includes an encoder module and a decoder module. In the encoder module, the remote sensing image to be semantically segmented is first processed by a backbone network to extract features and obtain a coarse feature representation. The decoder module includes a semantic domain optimization module and a spatial domain optimization module, with the spatial domain optimization module incorporating a nested attention module. In the decoder module, the coarse feature representation is first input into a 1×1 convolution for pre-classification to obtain a class probability distribution. The coarse feature representation and the class probability distribution are then input into the semantic domain optimization module to obtain a multi-confidence scale class representation and a global class representation. The spatial domain optimization module further processes the coarse feature representation and the multi-confidence scale class representation to obtain optimized pixel-class relationships. Finally, the pixel-class relationships and the global class representation undergo an affinity operation to obtain an enhanced semantic representation. The enhanced semantic representation is then processed using bilinear interpolation to restore its resolution to that of the remote sensing image to be semantically segmented, thus obtaining the semantic segmentation result of the remote sensing image.

[0011] Preferably, the backbone network adopts the HRNetv2-w32 model and loads pre-trained weights learned on the ImageNet dataset.

[0012] Preferably, the specific process in the semantic domain optimization module is as follows:

[0013] Based on the category probability distribution, we find the feature representing the k-th category in R, and the region containing this feature forms a set, i.e., the first category region. Simultaneously, we find the probability of the k-th category in the category probability distribution, and the region containing this probability forms a set, i.e., the second category region. We divide the second category region by columns, sort the elements in each column in descending order of their relative probabilities, and select the first and second highest probability values ​​among the K categories in each column. The absolute difference between the first and second highest probability values ​​is used as the confidence score W for whether each pixel belongs to the k-th category. k Using the aforementioned confidence level W k The confidence level W k Sort the pixels in descending order according to their relative confidence levels, and select the first pixel in the sorted sequence up to the index (m×N). k For all pixels within the range of the pixel ( / M), calculate the confidence level of each pixel to determine whether it belongs to the k-th category and the m-th confidence scale. Where m∈[1,M], M is the number of confidence scales; N k The number of feature representations belonging to the k-th category; for each category, calculate the class center for the m-th confidence scale of each category. From the class center Calculate the class center for each confidence scale for each category to obtain the multi-confidence scale class representation S. m; at the class center In the process, the class center corresponding to the largest confidence scale for each category is selected as the global class representation S. g .

[0014] Preferably, the number of confidence scales M in the semantic domain optimization module is 8.

[0015] Preferably, the specific process of the spatial domain optimization module is as follows:

[0016] The coarse feature representation R and the multi-confidence scale class representation S are compared. m Dimension reshaping is performed separately, and then an affinity operation is performed between the coarse feature representations after dimension reshaping and the multi-confidence scale class representations after dimension reshaping to obtain the initial pixel-class relationship A. m ;The initial pixel-class relationship A m The input is fed into the nested attention module to obtain the optimized pixel-class relationship A. g .

[0017] Preferably, the specific process of the nested attention module is as follows:

[0018] From the initial pixel-class relationship A m Obtain the pixel-class relationship A after enhanced consensus in the spatial domain. ′ m The pixel-class relationship A after enhanced consensus in the spatial domain ′ m The function form is:

[0019]

[0020] in, Both represent fully connected layers, with Softmax(·) indicating the layer type; H ′ The roughness feature represents the height of R; W ′ The roughness feature represents the width of R;

[0021] Then, the pixel-class relationship A after consensus enhancement in the spatial domain will be... ′ m Through a feedforward layer FC(A) is obtained after the feedforward layer. ′ m The pixel-class relationship FC(A) after passing through the feedforward layer will be... ′ m ) and pixel-class relationship A after enhanced consensus in the spatial domain ′ m By adding them together, we obtain the optimized pixel-class relationship A. g .

[0022] Preferably, the semantic segmentation model is trained in advance using labeled training data before being used for actual semantic segmentation.

[0023] Preferably, the semantic segmentation model uses cross-entropy loss as the loss function during training to update the network parameters of the semantic segmentation model.

[0024] Preferably, the remote sensing image is a high-resolution remote sensing image with a spatial resolution of less than 1m.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] This invention proposes a semantic segmentation method for remote sensing images based on dual-domain optimized class-aware attention. Addressing the problem that single-confidence class representations in ordinary class-level attention cannot distinguish intra-class differences, this invention uses class confidence as a scale standard in the semantic domain to extract multi-confidence scale class representations, thereby shortening the distance between pixels and their corresponding classes and thus distinguishing intra-class differences.

[0027] To address the issue of pixel-class relationship weight ambiguity caused by independent correlation calculations in ordinary attention, this invention uses nested attention in the spatial domain to find consensus in pixel-class relationships, thereby enhancing correct correlations and suppressing incorrect correlations.

[0028] Furthermore, this invention combines dual-domain optimized class-aware attention to provide a novel solution for high-resolution remote sensing image segmentation tasks and can improve the performance of semantic segmentation of remote sensing images. Attached Figure Description

[0029] Figure 1 Here is a model structure diagram of the semantic segmentation model (DOCNet) of the present invention; wherein, (a) is a structure diagram of the semantic segmentation model (DOCNet); (b) is a structure diagram of the semantic domain optimization module; (c) is a structure diagram of the spatial domain optimization module; and (d) is a schematic diagram of the nested attention module.

[0030] Figure 2 This is a flowchart illustrating the training and testing process of the semantic segmentation model (DOCNet) in an embodiment of the present invention.

[0031] Figure 3 This is a visualization of the test results in an embodiment of the present invention. Detailed Implementation

[0032] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0033] To enhance context awareness and eliminate background noise, spatial attention mechanisms, capable of modeling long-range dependencies, are widely used for semantic segmentation of remote sensing images. Many methods introduce semantically richer class representations instead of pixel representations in spatial attention to achieve class-level modeling. However, due to the complex backgrounds, large intra-class variance, and foreground-background imbalance in remote sensing images, spatial attention-based methods introduce a large amount of background context during intensive affinity operations, leading to unsatisfactory segmentation results. Furthermore, while general class-aware attention methods generate class representations as representative features to mitigate background context interference, the correlation calculation between pixel-class relationships is independent, and a single-confidence class representation cannot distinguish intra-class differences, resulting in semantically confused class-level semantic modeling. In addition, in ordinary attention, the calculation of inter-element correlations is independent, which introduces noise and ambiguity into the pixel-class relationship weights.

[0034] To shorten the distance between pixels and their corresponding categories and distinguish intra-class differences, this invention uses class confidence as the scaling criterion in the semantic domain to extract multi-confidence scale class representations. Furthermore, in the spatial domain, this invention uses nested attention to find consensus in pixel-class relationships to enhance correct relevance and suppress incorrect relevance. The core of this invention is the proposal of a deep network model based on dual-domain optimized class-aware attention and class-level context aggregation, namely the semantic segmentation model DOCNet. It should be noted that each module in this network model has high portability and can be applied to most networks.

[0035] This invention proposes a semantic segmentation model, DOCNet, to address the two aforementioned problems in remote sensing image segmentation. Specifically, in the semantic domain, this invention first extracts multi-confidence scale class representations using class confidence as the scaling standard, thereby reducing the distance between pixels and their corresponding classes and effectively mitigating intra-class discrepancies. In the spatial domain, this invention designs a nested attention module that uses the original pixel-class relationship as a query and seeks consensus to enhance correct relevance and suppress erroneous relevance. The proposed semantic segmentation model, DOCNet, integrates a dual-domain optimization strategy and class awareness, thereby achieving accurate and effective class-level context modeling.

[0036] This invention provides a remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention. Specifically, it involves acquiring a trained semantic segmentation model, DOCNet, which includes an encoder module and a decoder module. The remote sensing image to be semantically segmented is then input into the aforementioned semantic segmentation model to obtain the semantic segmentation result of the remote sensing image. The image used in this invention is preferably a remote sensing image, and more preferably a high-resolution remote sensing image with a spatial resolution of less than 1 meter.

[0037] The following section provides a detailed description of the specific structure and principles of the aforementioned semantic segmentation model, DOCNet.

[0038] In the encoder module of the aforementioned semantic segmentation model DOCNet, features are first extracted through a backbone network, and the features output by the backbone network are used as coarse feature representation R.

[0039] The decoder module described above mainly consists of a semantic domain optimization module (SEDO) and a spatial domain optimization module (SPDO). The spatial domain optimization module includes a nested attention module (NAM). The decoder module takes the coarse feature representation output by the encoder as input. During operation, the decoder module first inputs the coarse feature representation R into a 1×1 convolution for pre-classification to obtain the class probability distribution D. Then, the coarse feature representation and the class probability distribution are input together into the semantic domain optimization module to obtain the multi-confidence scale class representation S. m and global class representation S g The spatial domain optimization module further refines the coarse feature representation R and the multi-confidence scale class representation S. m The process is performed to obtain the optimized pixel-class relationship; finally, the pixel-class relationship A is... g and global class representation S g Perform affinity operations to obtain the enhanced semantic representation R. a The enhanced semantic representation is restored to the resolution of the remote sensing image to be semantically segmented by bilinear interpolation, thus obtaining the semantic segmentation result of the remote sensing image.

[0040] The specific structure of the semantic segmentation model DOCNet of this invention will be described in detail below. Figure 1 (a) shows the overall structure of the semantic segmentation model DOCNet, including the encoder module and the decoder module. The encoder module is used to extract coarse semantic features; the decoder includes a semantic domain optimization module (SEDO) and a spatial domain optimization module (SPDO), which are used to obtain enhanced semantic representations by extracting multi-confidence scale class representations and mining pixel-class relationship consensus, and restore the enhanced semantic representations to the input image size.

[0041] It should be noted that the remote sensing images to be semantically segmented in this embodiment of the invention need to undergo data preprocessing first, and then be processed in batches during the training phase. The semantic segmentation model is trained using training samples from each batch. The dimensions of the remote sensing images are (B×3×H×W), where H and W are the height and width of the remote sensing images, respectively; B is the batch size of the input, and the value of B depends on the sample size of each batch during the training phase, but is not limited in this embodiment of the invention; during the prediction phase, the value of B can be set to 1. The remote sensing images to be semantically segmented are derived from the LoveDA dataset.

[0042] In this embodiment, the intermediate processes and the dimensions of each parameter of the encoder and decoder modules described above are further described below to help those skilled in the art better understand the principle of using the encoder and decoder modules. Specifically, for the encoder module, its input is the remote sensing image to be semantically segmented, with dimensions (B×3×H×W). First, the backbone network selected in this embodiment is HRNetv2-w32, which has been pre-trained on the Image-Net dataset. The image to be segmented is input into the backbone network for feature extraction to obtain a relatively coarse feature representation R, with dimensions (B×C×H′×W′), where C is the number of feature channels in the coarse feature representation, H... ′ Let H / 4 and W′ be H / 4 and W / 4, respectively. During decoder operation, the coarse feature representation R is first input into a 1×1 convolution for pre-classification, yielding the class probability distribution D, whose dimensions are (B×K×H). ′ ×W ′ Then, the Semantic Domain Optimization (SEDO) module generates a multi-confidence scale class representation S based on the coarse feature representation R and the class probability distribution D. m and global class representation S g Their dimensions are (B×M×K×C) and (B×K×C), respectively. The Spatial Domain Optimization Module (SPDO) further refines the coarse feature representation R and the multi-confidence scale class representation S. m Processing is performed to obtain the optimized pixel-class relationship A. g Its dimensions are (B×K×H) ′ W ′ Finally, pixel-class relationship A g Relationship with global class S g Performing an affinity operation (element-wise multiplication) yields an enhanced semantic representation. Its dimensions are (B×C×H) ′ W ′ ),in, This is matrix multiplication; (·) T This represents the matrix transpose. Finally, the enhanced semantic representation is bilinearly interpolated to obtain the enhanced semantic representation R. aThe dimensions are reshaped to (B×C×H′×W′), which yields the semantic segmentation result of the input remote sensing image.

[0043] It should be noted that this invention takes into account the significant intra-class differences in remote sensing images. For example, the shape, color, and texture of buildings vary greatly in different regions and from different viewpoints. Extracting global class centers based solely on information from each class is insufficient to identify intra-class differences. Furthermore, unlike a single global class center, class representations with small-scale semantics (i.e., containing only high confidence) can accelerate the modeling of salient features for the corresponding class, while class representations with large-scale semantics (i.e., containing all confidence levels) are helpful for modeling global features of the class. Therefore, the aforementioned Semantic Domain Optimization (SEDO) module first divides the coarse feature representation and itself into several class regions based on the class probability distribution, and calculates the confidence of each pixel belonging to a specific class at a specific scale based on the difference in probability values. The class center for each class is calculated for the confidence at all scales, thus obtaining a multi-confidence scale class representation, where the class representation corresponding to the largest confidence scale is used as the global class representation.

[0044] The Semantic Domain Optimization Module (SEDO) in this invention primarily aims to generate multi-confidence scale class representations to address the problem that a single global class center cannot identify significant intra-class differences in remotely sensed images. The intermediate processes and parameters of the SEDO module are further described below to facilitate a better understanding of the principles behind its implementation. Figure 1 As shown in (b), the specific method is as follows: First, according to the category probability distribution D, the rough feature representation R and itself are divided into several categories. The feature of the kth category in the rough feature representation R is found, and the region where this feature is located forms a set, that is, the first category region R. k ={R [i,j,*] |argmax(D [i,j,*] )=k}, and at the same time find the probability of the k-th class in the class probability distribution D, and form a set by the region where the probability is located, that is, the second class region D. k ={D [i,j,*] |argmax(D [i,j,*] )=k}, where the dimensions of both the first and second type regions are (B×N) k ×C), k is the category label, N k Let T be the number of feature representations belonging to the k-th category. For ease of explanation, the batch dimension B is omitted from all coordinates, meaning all images in a batch are selected by default. A sorting sampling function T is introduced. i j (x), which returns all pixels within the range of index i to index j, sorted in descending order of variable x, starting from index 1. This invention will use the second type of region D...k Divide the data by column, sort the pixels in each column in descending order of their relative probability, and select the first and second highest probability values ​​from the K categories in each column. Then, use the highest probability value... Second largest probability value absolute difference between As the confidence score for whether each pixel belongs to the k-th category, i.e. Its dimensions are (B×N) k ×1). Furthermore, the confidence score W for whether each pixel belongs to the k-th category is used. k Select the first sorted pixel up to the index (m×N) k For all pixels within the range of the pixel ( / M), calculate the confidence level for each pixel to determine whether it belongs to the k-th category and the m-th confidence scale. Its dimensions are (B×(m×N)) k / M)×1), where m∈[1,M], and M is the maximum confidence scale. For each class, calculate the class center of the m-th confidence scale for each class, i.e. Its dimensions are (B×1×C), where, [n] represents the index. From the above class center... Calculate the class center for each confidence scale for each category to obtain the multi-confidence scale class representation. Its dimensions are (B×M×K×C). In addition, this invention extracts a global class representation for class-context integration in the decoder, that is, at the aforementioned class center. In the process, the class center corresponding to the largest confidence scale (i.e., the Mth confidence scale) of each category is selected as the global class representation. Its dimensions are (B×K×C).

[0045] It should be noted that the number of confidence scales in the semantic domain optimization module of the present invention can be selected according to the actual situation. In this embodiment, the number of confidence scales M is set to 8, that is, there are a total of M confidence scales. These confidence scales are sorted and numbered from 1 to M in ascending order. Therefore, the largest confidence scale corresponds to the Mth confidence scale.

[0046] It should be noted that the Spatial Domain Optimization (SPDO) module in this invention is based on a Nested Attention Module (NAM). It uses the original pixel-class relationship as the query and seeks consensus within a specific spatial range to enhance correct relevance and suppress incorrect relevance. The aforementioned Spatial Domain Optimization (SPDO) module first calculates the pixel-class relationship in the coarse feature representation R and the relationship between the pixel and the multi-confidence scale class representation S. mThe relationship between pixels and classes is then analyzed, and a nested attention module (NAM) is used to find consensus in the pixel-class relationship. Finally, an optimized pixel-class relationship is obtained through a feedforward layer.

[0047] In this embodiment, the purpose of the Spatial Domain Optimization (SPDO) module is to enhance pixel-class relationships by leveraging the relationship between pixels and multi-confidence scale class representations and their consensus. The intermediate processes and parameters of the SPDO module are further described below to facilitate a better understanding of the principles behind its implementation. Figure 1 As shown in (c), the Spatial Domain Optimization Module (SPDO)

[0048] First, calculate the pixel representation in the coarse feature representation R and the multi-confidence scale class representation S. m The relationship between the coarse feature representation R and the multi-confidence scale class representation S m Perform dimensional reshaping separately to obtain a size of (B×C×H) ′ W ′ The rough feature representation of ) and the size of (B×MK×H) ′ W ′ The multi-confidence scale class representation is used to perform an affinity operation between the dimension-reshaped coarse feature representation and the dimension-reshaped multi-confidence scale class representation to obtain the initial pixel-class relationship. Its dimensions are (B×MK×H) ′ W ′ ), This represents matrix multiplication. Subsequently, as... Figure 1 As shown in (d), the initial pixel-class relationship A m The input is fed into a nested attention module (NAM) to obtain the optimized pixel-class relationship A. g .

[0049] It should be noted that in the Nested Attention Module (NAM) of this invention, the initial pixel-class relationship A... m The pixel-class relationship A with enhanced consensus in the spatial domain is obtained. ′ m The dimensions are (B×K×H) ′ W ′ Finally, the pixel-class relationship A after consensus enhancement in the spatial domain will be... ′ m Through a feedforward layer Then, it is compared with the pixel-class relationship A after consensus enhancement in the spatial domain. ′ m By adding them together, we obtain the optimized pixel-class relationship A. g =A ′ m +FC(A′ m Its dimensions are (B×K×H) ′ W ′ The pixel-class relationship A above, after enhancing consensus in the spatial domain. ′ m The function form is as follows:

[0050]

[0051] in, Both represent fully connected layers used to project the original relation representation to accelerate relational interactions.

[0052] It should be noted that the aforementioned semantic segmentation model, DOCNet, is pre-trained using labeled training data before being used for actual semantic segmentation. Data augmentation can be performed on the training data to expand the training sample. The loss function used in training the semantic segmentation model is cross-entropy loss; the specific training process can be found in existing semantic segmentation model training methods and will not be elaborated further.

[0053] It should be noted that the LoveDA dataset used in the embodiments of the present invention can be from any and universal remote sensing image dataset, and can be selected according to the actual situation. The LoveDA dataset used in the embodiments of the present invention is implemented in the prior art, and can be referred to in the technical literature: Wang J, Zheng Z, Ma A, et al. LoveDA: A remotesensing land-cover dataset for domain adaptive semantic segmentation[J]. arXivpreprint arXiv:2110.08733,2021.

[0054] The following example demonstrates the technical effects of the remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention in a specific embodiment.

[0055] Example

[0056] The semantic segmentation model DOCNet used in this embodiment has the specific network structure described above and will not be repeated here. Figure 2 As shown, the overall process of semantic segmentation of remote sensing images can be divided into three stages: data preprocessing, model training, and image prediction.

[0057] 1. Data Preprocessing Stage

[0058] For the obtained raw remote sensing images (using the LoveDA dataset as an example in this embodiment), image preprocessing is performed. First, the raw remote sensing images are cut into 512×512 size, and then the cut images are randomly rotated and flipped to perform data augmentation.

[0059] 2. Model Training

[0060] Step 1. Select a subset of samples from the LoveDA dataset to construct the training dataset, and divide the training dataset into batches of a fixed batch size, with a total of N.

[0061] Step 2. Select a batch of training samples with index i sequentially from the training dataset, and train the semantic segmentation model DOCNet using each batch of training samples. During training, calculate the cross-entropy loss function for each training sample. And based on the total loss of all training samples in the batch The network parameters in the entire model are adjusted until all batches of the training dataset have participated in the model training. After reaching the specified number of iterations, the model converges and training is complete, where i∈{0,1,…,N}.

[0062] 3. Image Prediction

[0063] A subset of samples is reselected from the LoveDA dataset to construct a test dataset. The remote sensing images in the test dataset are used as input to the pre-trained semantic segmentation model DOCNet. Finally, a probability vector with each pixel class is predicted. The class with the highest probability is selected as the final output by activation functions such as Sigmoid, thereby achieving semantic segmentation.

[0064] In this embodiment, the test visualization results are as follows: Figure 3 As shown in Table 1, the test data results are as follows:

[0065] Table 1 Test Data Results

[0066] Dataset Back Building Road Water Barren Forest Agri mIoU LoveDA 46.6 59.3 56.9 81.5 21.3 46.2 65.2 53.9

[0067] Depend on Figure 3 As shown in Table 1, the semantic segmentation model DOCNet of this invention can effectively process segmentation results for remote sensing images. It relies on the semantic domain optimization module to fully distinguish intra-class differences and on the spatial domain optimization module to fully mine consensus in pixel-class relationships and optimize pixel-class relationships, thereby improving the segmentation performance of remote sensing images. It also provides a new solution for the application of class-level context modeling in the field of remote sensing image segmentation.

[0068] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on dual-domain optimized class-aware attention, characterized in that: A trained semantic segmentation model is obtained, and the remote sensing image to be semantically segmented is input into the semantic segmentation model to obtain the semantic segmentation result of the remote sensing image. The semantic segmentation model includes an encoder module and a decoder module. In the encoder module, the remote sensing image to be semantically segmented is first processed by a backbone network to extract features and obtain a coarse feature representation. The decoder module includes a semantic domain optimization module and a spatial domain optimization module, and the spatial domain optimization module has a nested attention module. In the decoder module, firstly, the coarse feature representation is input into a... Pre-classification is performed in convolution to obtain class probability distributions. The coarse feature representation and class probability distributions are input together into the semantic domain optimization module to obtain multi-confidence scale class representations and global class representations. The spatial domain optimization module further processes the coarse feature representations and multi-confidence scale class representations to obtain optimized pixel-class relationships. Finally, the pixel-class relationships and global class representations are subjected to affinity operations to obtain enhanced semantic representations. The enhanced semantic representations are then subjected to bilinear interpolation to restore the resolution of the enhanced semantic representations to the resolution of the remote sensing image to be semantically segmented, thereby obtaining the semantic segmentation result of the remote sensing image. The specific process of the nested attention module is as follows: From the initial pixel-class relationship Obtain pixel-class relationships with enhanced consensus in the spatial domain The pixel-class relationship enhanced with consensus in the spatial domain The function form is: in, Both represent fully connected layers. express; The roughness feature represents Height; The roughness feature represents The width; Then, the pixel-class relationship after consensus enhancement in the spatial domain will be further enhanced. Through a feedforward layer , obtain the pixel-class relationship after the feedforward layer The pixel-class relationship after passing through the feedforward layer Pixel-class relationships with enhanced consensus in the spatial domain By adding them together, we obtain the optimized pixel-class relationship. .

2. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 1, characterized in that, The backbone network uses the HRNetv2-w32 model and loads pre-trained weights learned from the ImageNet dataset.

3. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 1, characterized in that, The specific process in the semantic domain optimization module is as follows: Find the coarse feature representation based on the category probability distribution. The Middle The features of each category are used to form a set, i.e., the first category region, by the regions where each feature is located; simultaneously, the region containing the first category probability distribution is found. The probability of each category is represented by a set of regions containing that probability, known as the second category region. This second category region is then divided by columns. The elements in each column are sorted in descending order of their relative probabilities. Each column is then selected... The first and second highest probability values ​​in each category are used to determine whether each pixel belongs to the first category. Confidence of each category Using the aforementioned confidence level The confidence level Sort the pixels in descending order according to their relative confidence levels, and select the first pixel in the sorted sequence up to the next pixel with the index... For all pixels within the range of the given pixel, calculate whether each pixel belongs to the first pixel. Category 1, No. 2 Confidence level of each confidence scale ,in , The number of confidence scales; For belonging to the first Number of feature representations for each category; For each category, calculate the first value for each category. Class center of confidence scale ; by the class center Calculate the class center for each confidence scale for each category to obtain the multi-confidence scale class representation. ; at the class center In the process, the class center corresponding to the highest confidence scale for each category is selected as the global class representation. .

4. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 3, characterized in that, In the semantic domain optimization module, the number of confidence scales It is 8.

5. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 3, characterized in that, The specific process of the spatial domain optimization module is as follows: Representing coarse features Representation of multi-confidence scale Dimension reshaping is performed separately, and then an affinity operation is performed between the coarse feature representations after dimension reshaping and the multi-confidence scale class representations after dimension reshaping to obtain the initial pixel-class relationships. ; the initial pixel-class relationship The input is fed into the nested attention module to obtain the optimized pixel-class relationship. .

6. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 1, characterized in that, The semantic segmentation model is trained in advance using labeled training data before being used for actual semantic segmentation.

7. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 1, characterized in that, The semantic segmentation model uses cross-entropy loss as the loss function during training to update the network parameters of the semantic segmentation model.

8. The remote sensing image semantic segmentation method based on dual-domain optimized class-aware attention as described in claim 1, characterized in that, The remote sensing image is a high-resolution remote sensing image with a spatial resolution of less than 1m.

Images