A remote sensing image semantic segmentation method and device based on a diffusion model

Abstract

The application discloses a kind of based on diffusion model's remote sensing image semantic segmentation method and device, belong to image processing technical field, including the following steps: remote sensing image is inputted to denoising diffusion probability model and carries out feature extraction, and the feature map of model each level output is aggregated to obtain feature map collection;Feature map collection is inputted into multi-resolution multi-level feature fusion module, carries out resolution alignment, channel-space attention semantic enhancement, and by gate network dynamically adjusts each level feature output weight, obtains the feature map collection after enhanced fusion;The feature map collection after enhanced fusion is inputted into pixel-level label classification network, first modeling is graph structure representation and is updated and inquired to pixel-level label classification.Through the whole model is trained and applied to the semantic segmentation of new remote sensing image, the application can effectively improve the accuracy and robustness of remote sensing image classification under complex scene.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and apparatus for semantic segmentation of remote sensing images based on a diffusion model. Background Technology

[0002] Semantic segmentation, a core task in computer vision, aims to assign a specific category label to each pixel in an image, thereby achieving pixel-level scene resolution. High-precision semantic segmentation is of great significance in the application and deep understanding of remote sensing images. However, due to the inherent characteristics of remote sensing images, this task faces significant challenges in practice, mainly in two key aspects: large-scale variations and perspective changes, as well as category imbalance. Remote sensing images cover a wide area, and the features of land cover, lighting conditions, seasonal changes, and other factors vary greatly in different regions. This results in the same type of land cover appearing very differently in different areas. Furthermore, the diverse shooting perspectives of remote sensing images lead to significant differences in the frequency and area occupied by different types of land cover, further increasing the difficulty of segmentation.

[0003] Traditional algorithms, such as thresholding and edge detection, while simple to implement, fall short when processing complex or low-resolution remote sensing images. From a technical perspective, there are two main bottlenecks: First, traditional algorithms only possess a local perceptual field, severely limiting their ability to capture global image information and making it difficult to grasp the overall relationships and spatial layout of features within the image. Second, traditional algorithms lack effective strategies to address large-scale differences and class imbalances in remote sensing images, resulting in poor segmentation accuracy and performance.

[0004] With the rapid development of machine learning technology, semantic segmentation techniques based on machine learning have made significant progress since the fully convolutional network (FCN) model, strongly promoting the development of this field. To address the problem of large-scale variations in remote sensing images, a series of new algorithms and plugins related to attention mechanisms and receptive fields have emerged. The Deeplab series enhances the receptive field by extending convolution and conditional random fields (CRF). They utilize spatial pyramid pooling (ASPP) modules and residual networks to obtain multi-scale features, improving segmentation performance to some extent. The Unet architecture, with its superior multi-scale feature capture capabilities, has inspired many methods to enhance global semantic information. Integrating the Transformer architecture provides an innovative method to capture global semantics by modeling long-range contextual information, thereby improving segmentation results. Some studies combine traditional convolutional neural networks (CNNs) with Unet and Transformers to achieve complementary effects. For example, ST Unet introduces a novel Swin Transformer and CNN parallel dual encoder to improve segmentation accuracy. The latest model, Samba, employs state-space modeling (SSM) to capture multi-level and global semantic information.

[0005] However, when dealing with complex and varied remote sensing images, the methods mentioned above still have problems such as unstable segmentation results when dealing with class imbalance, and there is an urgent need to explore more effective image semantic segmentation solutions. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a method and apparatus for semantic segmentation of remote sensing images based on a diffusion model. This method deeply mines multi-scale features of remote sensing images through a denoising diffusion probability model, achieves resolution alignment and semantic enhancement using a multi-resolution, multi-level feature fusion module, and introduces a dynamic adaptive graph construction and lightweight graph relationship reasoning module to effectively model the feature relationships between nodes in the graph structure. It also improves the robustness of pixel-level label classification by combining a spatially constrained classification head, and alleviates the class imbalance problem by jointly optimizing the model using focus loss and cross-entropy loss. Therefore, this method can significantly improve the segmentation accuracy of remote sensing images in complex scenes.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a remote sensing image semantic segmentation method based on a diffusion model, comprising the following steps:

[0009] The remote sensing image is input into the denoising diffusion probability model for feature extraction, and the feature maps output from each level of the model are aggregated to obtain a feature map collection.

[0010] The feature map assembly is input into a multi-resolution, multi-level feature fusion module. First, the feature map assembly is aligned to the resolution, then semantically enhanced by a channel-spatial attention layer, and finally a gating network is used to dynamically adjust the output weights of each level of features to obtain the enhanced and fused feature map assembly.

[0011] The enhanced and fused feature map set is input into the pixel-level label classification network. First, the enhanced and fused feature map set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the relationship between the node features in the graph structure is modeled and updated through a lightweight graph relationship inference module. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure.

[0012] A remote sensing image semantic segmentation model, including a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, was trained using a remote sensing image dataset and a total loss including focus loss and cross-entropy loss. This model was then applied to the semantic segmentation of new remote sensing images.

[0013] Preferably, the step of inputting the remote sensing image into the denoising diffusion probability model for feature extraction, and aggregating the feature maps output from each level of the model to obtain a feature map collection, includes:

[0014] The remote sensing image is input into a pre-trained denoising diffusion probability model. The remote sensing image undergoes a forward diffusion noise addition process, and then passes through a denoising network based on the Unet architecture. At the decoding output of Unet, feature maps of different resolutions are obtained. Based on the high semantic level information and detail edge information of the obtained feature maps, some feature maps with these key features are selected, and then they are spliced ​​and aggregated in the channel dimension to finally obtain a feature map collection.

[0015] Preferably, when using a denoising diffusion probability model for feature extraction, the time step, a fundamental parameter in the model, is adjusted. To control the noise level in the obtained feature maps, and further to divide different time steps The feature map collection below is fused by overlaying the channel dimensions and used as input for the subsequent multi-resolution, multi-level feature fusion module.

[0016] Preferably, the step of first aligning the feature map collection at resolution and then performing semantic enhancement through a channel-spatial attention layer includes:

[0017] The feature map collection is resolution aligned using bilinear upsampling or deformable convolution, and channel-spatial attention is introduced to dynamically weight important regions. For the ... Layer feature map The alignment and enhancement process is as follows:

[0018] ,

[0019] in, This represents the feature map after resolution alignment and semantic enhancement via a channel-space attention layer. Indicates upsampling, and These represent the channel attention weights and spatial attention weights, respectively. and These represent channel-wise multiplication and space-wise multiplication, respectively.

[0020] Preferably, the subsequent gating network dynamically adjusts the feature output weights at each level, including:

[0021] Dynamic gating weights are designed to balance the contributions of features at different levels. The output features of each level are represented as follows:

[0022] ,

[0023] in, This represents the output characteristics of each level. Indicates the total number of floors. Indicates the first Layer-gated networks, Represents the first in the feature map set. Layer feature map, This represents the semantic enhancement of the first layer after resolution alignment and channel-space attention layer. Layer feature map.

[0024] Preferably, the gating network is based on constructing an output weight graph. In this implementation, 1 represents the number of channels. Representing spatial dimensions, by analyzing each layer of feature maps within the feature map assembly. Calculate local variance Initial weights are generated by lightweight convolution. Then, after Softmax normalization, we get And satisfy .

[0025] Preferably, the step of modeling the enhanced fused feature map assembly as a graph structure representation through a dynamic adaptive graph construction layer includes:

[0026] Input the enhanced fusion of feature maps , Indicates batch size, Indicates the number of channels. Indicates spatial dimensions, Reorganized into a node matrix , Based on Construct an adaptive adjacency matrix:

[0027] ,

[0028] in, Represents the adaptive adjacency matrix The Middle Line number Column elements, This represents the Softmax function. and This represents a linear transformation achieved by convolution. and Indicates adjacent nodes, Indicates the distance between nodes. Indicates a local radius constraint mask, superscript This represents the transpose, and finally, the node matrix is ​​used. With adaptive adjacency matrix Combined, they are constructed into a dynamic graph structure representation.

[0029] Preferably, the lightweight graph relation reasoning module includes at least two multilayer perceptron networks.

[0030] Preferably, the pixel-level label classification by the spatially constrained classification head based on the updated graph structure includes:

[0031] Based on the updated graph structure Perform feature recombination:

[0032] ,

[0033] in, This represents the feature map after recombination. Represents the feature recombination function. Indicates batch size, Indicates the number of channels. Indicates spatial dimensions;

[0034] Perform local context aggregation on the recombined feature map:

[0035] ,

[0036] in, This represents the feature map after local context aggregation. This represents depthwise separable convolution. Represents the convolution kernel Size;

[0037] based on and Achieve pixel-level tag classification:

[0038] ,

[0039] in, Indicates pixel position The classification probability at that location. This represents the Softmax function. Represents the classifier weight matrix. Indicates feature splicing, and They represent and Middle pixel position The eigenvector at that location.

[0040] Secondly, embodiments of the present invention also provide a remote sensing image semantic segmentation device based on a diffusion model, which is implemented using the above-mentioned remote sensing image semantic segmentation method based on a diffusion model, including: a feature extraction and aggregation module, a multi-resolution enhancement and fusion module, a graph structure reasoning and classification module, and a model training and application module;

[0041] The feature extraction and aggregation module is used to input remote sensing images into a denoising diffusion probability model for feature extraction, and to aggregate the feature maps output by each level of the model to obtain a feature map collection.

[0042] The multi-resolution enhancement and fusion module is used to input the feature map collection into the multi-resolution and multi-level feature fusion module. The feature map collection is first aligned in resolution, then semantically enhanced by a channel-spatial attention layer, and then dynamically adjusted by a gating network to obtain the enhanced and fused feature map collection.

[0043] The graph structure reasoning classification module is used to input the enhanced and fused feature graph set into the pixel-level label classification network. First, the enhanced and fused feature graph set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the lightweight graph relationship reasoning module models and updates the relationship between the node features in the graph structure. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure.

[0044] The model training application module is used to train a remote sensing image semantic segmentation model, including a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, using a remote sensing image dataset and a total loss including focus loss and cross-entropy loss, and then apply it to the semantic segmentation of new remote sensing images.

[0045] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0046] (1) This invention utilizes a denoising diffusion probability model to gradually extract multi-level and multi-resolution semantic features from complex remote sensing images, which contain rich ground feature information. By aggregating these multi-level features to form a feature map collection, it provides a more comprehensive and robust feature representation foundation for subsequent segmentation tasks, which helps to enhance the model's adaptability to complex scenes, noise interference and imaging differences, and significantly improves segmentation accuracy.

[0047] (2) This invention, through a carefully designed multi-resolution, multi-level feature fusion module, dynamically emphasizes and enhances feature responses from rare classes and small-scale targets via a channel-spatial attention layer and a gating network, while suppressing interference from dominant classes such as background, thereby significantly improving the recall rate and segmentation accuracy for minority classes and small targets. Simultaneously, through the dynamic adaptive graph construction layer and lightweight graph relation reasoning module in the pixel-level label classification network, it can explicitly model long-range semantic relationships and spatial context between pixels, enabling easily confused categories to be more accurately identified and distinguished within the relational context, thus significantly improving visual effects.

[0048] (3) After the denoising diffusion probability model of the present invention is pre-trained on a large-scale remote sensing dataset, its parameters can be fixed and used as a general feature extraction backbone network. When applied to a new specific task or different types of datasets, only the subsequent multi-resolution multi-level feature fusion module and pixel-level label classification network need to be trained, which greatly reduces the requirements for computing resources and sample quantity of the new dataset, shortens the training time, and avoids the risk of overfitting, so that the model can be quickly and efficiently adapted to diverse remote sensing application scenarios. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the remote sensing image semantic segmentation method based on a diffusion model provided in an embodiment of the present invention.

[0051] Figure 2 This is an example of an 18-layer feature map output from the Unet network decoding provided in this embodiment of the invention;

[0052] Figure 3 These are different time steps provided in the embodiments of the present invention. t Visual comparison of feature maps below;

[0053] Figure 4This is an overall architecture diagram of the remote sensing image semantic segmentation model provided in this embodiment of the invention;

[0054] Figure 5 This invention provides a comparison of the visual effects of RGB remote sensing images segmented under different semantic segmentation algorithm models.

[0055] Figure 6 This is a schematic diagram of the structure of the remote sensing image semantic segmentation device based on the diffusion model provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0057] The inventive concept of this invention is as follows: Addressing the problems of insufficient feature extraction, poor multi-level feature fusion, and limited pixel-level classification accuracy in existing remote sensing image semantic segmentation technologies, this invention provides a remote sensing image semantic segmentation method and apparatus based on a diffusion model. It extracts and aggregates multi-scale features layer by layer using a denoising diffusion probability model to construct a feature map ensemble rich in semantic information. It strengthens the expression of key features using multi-resolution alignment and channel-space attention mechanisms, and dynamically balances hierarchical contributions using a gating network. Furthermore, it introduces a dynamic graph structure modeling and lightweight graph inference module to capture long-distance dependencies between pixels, and optimizes local consistency through spatially constrained classification heads. Finally, it introduces focus loss to suppress class imbalance, improving the accuracy and robustness of remote sensing image classification in complex scenes.

[0058] like Figure 1 As shown in the embodiment, a remote sensing image semantic segmentation method based on a diffusion model is provided, including the following steps:

[0059] S1: Input the remote sensing image into the denoising diffusion probability model for feature extraction, and aggregate the feature maps output from each level of the model to obtain a feature map collection.

[0060] S1.1, Construct a remote sensing image dataset.

[0061] In this embodiment, image data is downloaded from the open-source ISPRIS remote sensing image dataset and augmented. Simultaneously, the images are transformed to obtain corresponding NumPy format masks as labels, which are then divided into training, validation, and test sets in a 6:2:2 ratio. The dataset is then combined into fixed-batch datasets using the data loader in the PyTorch framework, with the original images and corresponding label masks packaged together as model input.

[0062] S1.2, Feature map extraction and aggregation.

[0063] In this embodiment, remote sensing images from the remote sensing image dataset are... Inputting a pre-trained denoising diffusion probability model, the remote sensing image undergoes a forward diffusion denoising mathematical process to obtain a denoised image. Then, a pre-trained denoising network based on the Unet architecture is used to obtain feature maps of different resolutions at the decoding output of the Unet network. Based on the high semantic level information and detail edge information of the obtained feature maps, partial feature maps with these key features are selected, and then these are concatenated and aggregated along the channel dimension to finally obtain a feature map collection. At the decoding output of the Unet network, 18 layers of feature maps with different resolutions and semantic levels can be obtained, such as... Figure 2 As shown.

[0064] Meanwhile, when using a denoising diffusion probability model for feature extraction, the time step, a fundamental parameter in the model, can be adjusted. (This represents the number of noise-adding steps; more steps result in stronger noise and more severe damage to the original image) to control the noise level in the obtained feature map. Visual effects at different time steps t are compared below. Figure 3 As shown, different time step parameter values ​​have different effects on the image, greatly enriching the information content of the feature map and helping to increase the robustness and generalization of the model during training, resulting in more stable and excellent performance on datasets with more diverse shooting conditions. This also verifies the impact of different time steps on the final semantic segmentation result and visual performance. Therefore, using a feature map set containing only a single time step as input to the subsequent semantic segmentation network output head is insufficient, as it does not carry enough information. Therefore, to further enhance the robustness and generalization of the entire algorithm architecture, different time steps can be used... The feature map collection below is fused by overlaying the channel dimensions and used as input for the subsequent multi-resolution, multi-level feature fusion module.

[0065] S2, the feature map collection is input into the multi-resolution, multi-level feature fusion module. First, the feature map collection is aligned to the resolution, then semantically enhanced by the channel-spatial attention layer, and then a gating network is used to dynamically adjust the output weights of each level of features to obtain the enhanced and fused feature map collection.

[0066] S2.1, Multi-scale feature alignment and semantic enhancement.

[0067] In this embodiment, due to the large differences in the scale of ground features in remote sensing images (such as buildings and small roads), it is necessary to address the issue of feature map resolution and semantic gap. Therefore, the obtained feature map set is input into a multi-resolution, multi-level feature fusion module, which sequentially enhances each feature map at different resolutions and levels, ultimately obtaining an enhanced and fused feature map set.

[0068] First, the feature maps in the feature map collection are aligned in resolution using bilinear upsampling or deformable convolution to unify the feature map size.

[0069] Then, a channel-space attention mechanism is introduced to dynamically weight important regions, for the first... Layer feature map The alignment and enhancement process is as follows:

[0070] ,

[0071] in, This represents the feature map after resolution alignment and semantic enhancement via a channel-space attention layer. This indicates upsampling (up to the highest resolution). and These represent channel attention weights (focusing on important channels) and spatial attention weights (focusing on important regions), respectively. and These represent channel-wise multiplication and space-wise multiplication, respectively.

[0072] S2.2, cross-level adaptive fusion.

[0073] In this embodiment, dynamic gating weights are designed to balance the contributions of features at different levels, and the output features at each level are represented as follows:

[0074] ,

[0075] in, This represents the output characteristics of each level. Indicates the total number of floors. Indicates the first Layer-gated networks, Represents the first in the feature map set. Layer feature map, This represents the semantic enhancement of the first layer after resolution alignment and channel-space attention layer. Layer feature map.

[0076] Gated networks are based on constructing an output weight graph. In this implementation, 1 represents the number of channels. Representing spatial dimensions, by analyzing each layer of feature maps within the feature map assembly. Calculate local variance Initial weights are generated by lightweight convolution. Then, after Softmax normalization, we get And satisfy .

[0077] Using the aforementioned method, feature maps at different resolutions and levels are sequentially enhanced to achieve multi-scale complementarity. This allows for the preservation of edge details (such as building outlines) for high-resolution features and the capture of macroscopic structures (such as farmland areas) for low-resolution features. Furthermore, through dynamic gating network adjustment, reliable features are automatically enhanced and noise is suppressed in complex scenes (such as shadow occlusion). In addition, by incorporating an attention mechanism, key responses in spatial and channel dimensions are strengthened to address the uneven distribution of ground features in remote sensing images (such as linear roads and scattered vegetation). Through the synergistic effect of resolution alignment, attention mechanisms, and dynamic gating, the performance of remote sensing image semantic segmentation models in complex scenes can be effectively improved.

[0078] S3 inputs the enhanced and fused feature map set into the pixel-level label classification network. First, the enhanced and fused feature map set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the relationship between the node features in the graph structure is modeled and updated through a lightweight graph relationship inference module. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure.

[0079] S3.1, Dynamic Adaptive Graph Construction Layer.

[0080] In this embodiment, the enhanced fusion feature graph set, which is a regular grid feature, is transformed into a graph structure representation, and an adaptive connection based on feature similarity is established to preserve spatial locality constraints.

[0081] Specifically, it includes:

[0082] (1) Input the enhanced and fused feature map set , Indicates batch size, Indicates the number of channels. Indicates spatial dimensions;

[0083] (2) Reorganized into a node matrix , ;

[0084] (3) Based on Construct an adaptive adjacency matrix:

[0085] ,

[0086] in, Represents the adaptive adjacency matrix The Middle Line number Column elements, This represents the Softmax function. and This represents a linear transformation achieved by a 1×1 convolution. and Indicates adjacent nodes, Indicates the distance between nodes. Indicates a local radius constraint mask (avoiding the computational overhead of fully connected components), superscript Indicates transpose;

[0087] (4) By using the node matrix With adaptive adjacency matrix Combined, they are constructed into a dynamic graph structure representation.

[0088] S3.2, Lightweight Graph Relation Reasoning Module.

[0089] In this embodiment, to explicitly model the binary relationships between features, a lightweight two-layer multilayer perceptron (MLP) network is designed by jointly utilizing feature differences and spatial relationships to update the relationships between node features in the graph structure.

[0090] S3.3, Spatial Constraint Classification Header.

[0091] In this embodiment, local spatial consistency constraints are used to avoid complex upsampling (which relies on the high-resolution output of multi-resolution, multi-level feature fusion modules) and to simplify the classification decision layer.

[0092] Specifically, it includes:

[0093] (1) Based on the updated graph structure Perform feature recombination:

[0094] ,

[0095] in, This represents the feature map after recombination. Represents the feature recombination function. Indicates batch size, Indicates the number of channels. Indicates spatial dimensions;

[0096] (2) Perform local context aggregation on the recombined feature map:

[0097] ,

[0098] in, This represents the feature map after local context aggregation. This represents depthwise separable convolution. Represents the convolution kernel Size;

[0099] (3) Based on and Achieve pixel-level tag classification:

[0100] ,

[0101] in, Indicates pixel position The classification probability at that location. This represents the Softmax function. Represents the classifier weight matrix. Indicates feature splicing, and They represent and Middle pixel position The eigenvector at that location.

[0102] S4 uses a remote sensing image dataset and a total loss including focus loss and cross-entropy loss to train a remote sensing image semantic segmentation model, which includes a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, and applies it to the semantic segmentation of new remote sensing images.

[0103] In the embodiments, based on remote sensing image datasets, such as... Figure 4 The remote sensing image semantic segmentation model shown includes a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network. Iterative training is performed using the backpropagation mechanism in conjunction with the loss function to calculate and update the model's weights until the loss function of the model training decreases to a stable value. When the model is in a convergent state, the training ends.

[0104] At the same time, focus loss and cross-entropy loss This approach combines various methods to alleviate class imbalance in remote sensing images while maintaining stable training, resulting in a total loss. Represented as:

[0105] ,

[0106] in, As a constant coefficient, 0.3 is the most reasonable value selected after experimental testing.

[0107] Finally, import the trained weight model into the PyTorch framework, switch the model state to evaluation mode, and perform model inference on the image that needs semantic segmentation. The inference result image is the corresponding prediction mask.

[0108] In summary, the remote sensing image semantic segmentation method based on a diffusion model provided in this embodiment of the invention was tested on the classic open-source dataset Vaihingen. The comparison results are shown in Table 1 below. Figure 5 As shown.

[0109] Table 1. Data metrics of the dataset under different semantic segmentation algorithm models

[0110]

[0111] As can be seen, the method provided by this invention is superior to the previously established models and algorithms, achieving improvements of 0.77% and 1.38% in the average intersection-union ratio (mIoU) and average F1 (mF1) metrics, respectively.

[0112] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention provides a remote sensing image semantic segmentation device 600 based on a diffusion model, including: a feature extraction and aggregation module 610, a multi-resolution enhancement and fusion module 620, a graph structure reasoning and classification module 630, and a model training and application module 640.

[0113] The feature extraction and aggregation module 610 is used to input remote sensing images into a denoising diffusion probability model for feature extraction, and to aggregate the feature maps output by each level of the model to obtain a feature map collection.

[0114] The multi-resolution enhancement and fusion module 620 is used to input the feature map collection into the multi-resolution and multi-level feature fusion module. First, the feature map collection is aligned in resolution, then semantic enhancement is performed through the channel-spatial attention layer, and then a gating network is used to dynamically adjust the feature output weights of each level to obtain the enhanced and fused feature map collection.

[0115] The graph structure reasoning classification module 630 is used to input the enhanced and fused feature map set into the pixel-level label classification network. First, the enhanced and fused feature map set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the lightweight graph relationship reasoning module models and updates the relationship between the node features in the graph structure. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure.

[0116] The model training application module 640 is used to train a remote sensing image semantic segmentation model, including a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, using a remote sensing image dataset and a total loss including focus loss and cross-entropy loss, and then apply it to the semantic segmentation of new remote sensing images.

[0117] It should be noted that the remote sensing image semantic segmentation device based on the diffusion model provided in the above embodiments belongs to the same inventive concept as the remote sensing image semantic segmentation method based on the diffusion model. For details of its specific implementation process, please refer to the embodiments of the remote sensing image semantic segmentation method based on the diffusion model, which will not be repeated here.

[0118] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on a diffusion model, characterized in that, Includes the following steps: The remote sensing image is input into a denoising diffusion probability model for feature extraction. The feature maps output from each level of the model are then aggregated to obtain a feature map collection. This process includes: inputting the remote sensing image into a pre-trained denoising diffusion probability model; performing a forward diffusion noise addition process on the remote sensing image; then passing it through a denoising network based on the UNet architecture; obtaining feature maps of different resolutions at the decoding output of UNet; selecting feature maps with key features based on their high semantic level information and detail edge information; and then concatenating and aggregating these feature maps along the channel dimension to obtain the final feature map collection. During feature extraction using the denoising diffusion probability model, the time step, a fundamental parameter in the model, is adjusted. To control the noise level in the obtained feature maps, and further to analyze different time steps The feature map collection below is fused by channel-dimensional superposition and used as input for subsequent multi-resolution multi-level feature fusion modules; The feature map assembly is input into a multi-resolution, multi-level feature fusion module. First, the feature map assembly is aligned to the resolution, then semantically enhanced by a channel-spatial attention layer, and finally a gating network is used to dynamically adjust the output weights of each level of features to obtain the enhanced and fused feature map assembly. The enhanced and fused feature map set is input into the pixel-level label classification network. First, the enhanced and fused feature map set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the relationship between the node features in the graph structure is modeled and updated through a lightweight graph relationship inference module. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure. A remote sensing image semantic segmentation model, including a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, was trained using a remote sensing image dataset and a total loss including focus loss and cross-entropy loss. This model was then applied to the semantic segmentation of new remote sensing images.

2. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1, characterized in that, The process of first aligning the feature map collection at different resolutions, and then performing semantic enhancement through a channel-spatial attention layer, includes: The feature map collection is resolution aligned using bilinear upsampling or deformable convolution, and channel-spatial attention is introduced to dynamically weight important regions. For the ... Layer feature map The alignment and enhancement process is as follows: ， in, This represents the feature map after resolution alignment and semantic enhancement via a channel-space attention layer. Indicates upsampling, and These represent the channel attention weights and spatial attention weights, respectively. and These represent channel-wise multiplication and space-wise multiplication, respectively.

3. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1, characterized in that, The subsequent gating network dynamically adjusts the feature output weights at each level, including: Dynamic gating weights are designed to balance the contributions of features at different levels. The output features of each level are represented as follows: ， in, This represents the output characteristics of each level. Indicates the total number of floors. Indicates the first Layer-gated networks, Represents the first in the feature map set. Layer feature map, This represents the semantic enhancement of the first layer after resolution alignment and channel-space attention layer. Layer feature map.

4. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1 or 3, characterized in that, Gated networks are based on constructing an output weight graph. In this implementation, 1 represents the number of channels. Representing spatial dimensions, by analyzing each layer of feature maps within the feature map assembly. Calculate local variance Initial weights are generated by lightweight convolution. Then, after Softmax normalization, we get And satisfy .

5. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1, characterized in that, The process of modeling the enhanced and fused feature map assembly as a graph structure representation through a dynamic adaptive graph construction layer includes: Input the enhanced fusion of feature maps , Indicates batch size, Indicates the number of channels. Indicates spatial dimensions, Reorganized into a node matrix , Based on Construct an adaptive adjacency matrix: ， in, Represents the adaptive adjacency matrix The Middle Line number Column elements, This represents the Softmax function. and This represents a linear transformation achieved by convolution. and Indicates adjacent nodes, Indicates the distance between nodes. Indicates a local radius constraint mask, superscript This represents the transpose, and finally, the node matrix is ​​used. With adaptive adjacency matrix Combined, they are constructed into a dynamic graph structure representation.

6. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1, characterized in that, The lightweight graph relation reasoning module includes at least two multilayer perceptron networks.

7. The remote sensing image semantic segmentation method based on a diffusion model according to claim 1, characterized in that, The pixel-level label classification based on the updated graph structure by the spatially constrained classification head includes: Based on the updated graph structure Perform feature recombination: ， in, This represents the feature map after recombination. Represents the feature recombination function. Indicates batch size, Indicates the number of channels. Indicates spatial dimensions; Perform local context aggregation on the recombined feature map: ， in, This represents the feature map after local context aggregation. This represents depthwise separable convolution. Represents the convolution kernel Size; based on and Achieve pixel-level tag classification: ， in, Indicates pixel position The classification probability at that location. This represents the Softmax function. Represents the classifier weight matrix. Indicates feature splicing, and They represent and Middle pixel position The eigenvector at that location.

8. A remote sensing image semantic segmentation device based on a diffusion model, implemented using the remote sensing image semantic segmentation method based on a diffusion model as described in any one of claims 1 to 7, characterized in that, include: The module includes a feature extraction and aggregation module, a multi-resolution enhancement and fusion module, a graph structure reasoning and classification module, and a model training and application module. The feature extraction and aggregation module is used to input remote sensing images into a denoising diffusion probability model for feature extraction, and to aggregate the feature maps output by each level of the model to obtain a feature map collection. The multi-resolution enhancement and fusion module is used to input the feature map collection into the multi-resolution and multi-level feature fusion module. The feature map collection is first aligned in resolution, then semantically enhanced by a channel-spatial attention layer, and then dynamically adjusted by a gating network to obtain the enhanced and fused feature map collection. The graph structure reasoning classification module is used to input the enhanced and fused feature graph set into the pixel-level label classification network. First, the enhanced and fused feature graph set is modeled as a graph structure representation through a dynamic adaptive graph construction layer. Then, the lightweight graph relationship reasoning module models and updates the relationship between the node features in the graph structure. Finally, the spatial constraint classification head performs pixel-level label classification based on the updated graph structure. The model training application module is used to train a remote sensing image semantic segmentation model, including a denoising diffusion probability model, a multi-resolution multi-level feature fusion module, and a pixel-level label classification network, using a remote sensing image dataset and a total loss including focus loss and cross-entropy loss, and then apply it to the semantic segmentation of new remote sensing images.

Images