An optical remote sensing image weakly supervised salient object detection method

Abstract

This application discloses a weakly supervised salient target detection method for optical remote sensing images, relating to the field of optical remote sensing image processing technology. The method includes: acquiring a training sample set of optical remote sensing images, each training sample including the optical remote sensing image and its graffiti-annotated mask; constructing a network architecture for a salient target detection model, including a conditional guidance module, a denoising module, and a knowledge distillation module; training the salient target detection model using the optical remote sensing image training sample set; and using the trained salient target detection model to detect salient targets in the optical remote sensing image under test to obtain salient target detection results. This application combines the stepwise generation capability of the diffusion model with the knowledge transfer mechanism of knowledge distillation, achieving high-precision salient target detection in optical remote sensing images under weak supervision, effectively improving the accuracy of salient target detection results.

Description

Technical Field

[0001] This application relates to the field of optical remote sensing image processing technology, and in particular to a method for detecting weakly supervised salient targets in optical remote sensing images. Background Technology

[0002] Salience object detection, a fundamental task in computer vision, aims to quickly locate and segment the most visually attractive target regions from complex scenes by simulating human visual attention mechanisms. With the development of deep learning technology, salient object detection methods based on convolutional neural networks have gradually become mainstream and are now widely used in agriculture, forestry, and environmental science.

[0003] With the advancement of satellite and aerial photography technologies, the application of salient target detection in optical remote sensing images is becoming increasingly widespread. However, optical remote sensing images are often plagued by complex background interference from various elements such as buildings, vegetation, and water bodies, and the diverse types of remote sensing targets exhibit significant differences in imaging scale and morphological features. This results in existing salient target detection methods based on convolutional neural networks performing poorly and with low accuracy in optical remote sensing images. Summary of the Invention

[0004] To address the aforementioned problems and technical requirements, this application proposes a weakly supervised salient target detection method for optical remote sensing images. The technical solution of this application is as follows: A method for weakly supervised salient target detection in optical remote sensing images includes the following steps: Obtain a training sample set of optical remote sensing images. Each optical remote sensing image training sample includes an optical remote sensing image and its graffiti annotation mask, wherein the graffiti annotation mask indicates the salient targets of the optical remote sensing image. A network architecture for a salient target detection model is constructed and trained using a training sample set of optical remote sensing images. The salient target detection model includes a conditional guidance module, a denoising module, and a knowledge distillation module. The conditional guidance module extracts multi-scale features from the optical remote sensing images and generates conditional guidance information and conditional prediction masks. The denoising module generates a noise mask and recovers the salient target from the noise mask based on the conditional guidance information to obtain a denoised prediction mask. The knowledge distillation module uses the conditional prediction mask generated by the conditional guidance module as a soft label to guide the denoising module to recover the salient target and to guide the feature learning within the conditional guidance module. The trained salient target detection model is used to detect salient targets in the optical remote sensing image under test to obtain salient target detection results.

[0005] The further technical solution is as follows: the conditional guidance module includes a cascaded feature extraction encoder and a convolutional decoder; the feature extraction encoder is designed based on the pyramid vision Transformer and includes multiple stacked coding layers. Different coding layers extract guidance features of different scales from the optical remote sensing image and then transmit them to the convolutional decoder. The convolutional decoder includes decoding layers that correspond one-to-one with the coding layers of the feature extraction encoder; the optical remote sensing image of the training sample is input into the feature extraction encoder, and after being processed by multiple coding layers to extract guidance features of multiple scales, it is transmitted to the convolutional decoder. The guidance features of multiple scales extracted by the feature extraction encoder are decoded by multiple decoding layers to obtain conditional guidance information and conditional prediction masks of multiple scales; as the number of coding layers increases, the scale of the extracted guidance features decreases, and as the number of decoding layers increases, the scale of the decoded conditional guidance information and conditional prediction masks increases. Each decoding layer consists of two convolutional blocks cascaded together and a prediction output layer. Each convolutional block includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function layer connected sequentially from input to output. The convolutional decoder employs a top-down feature transfer mechanism. The first decoding layer inputs the guiding features extracted from the last encoding layer into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the last encoding layer. Starting from the second decoding layer, the conditional guiding information output from the previous decoding layer is upsampled and fused with the guiding features extracted from the corresponding encoding layer. This fusion is then input into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the corresponding encoding layer. The prediction output layer consists of a cascaded 1×1 convolutional layer and a Sigmoid activation function layer. The prediction output layer maps the conditional guiding information to a conditional prediction mask.

[0006] The further technical solution is as follows: the denoising module includes a noise mask generation module, a denoising encoder, and a denoising decoder; the noise mask generation module, based on the forward process of the diffusion model, progressively adds Gaussian noise to the graffiti annotation mask of the optical remote sensing image training samples according to the iteration time step to obtain multiple noise masks, each noise mask corresponding to one iteration time step; the denoising encoder is designed based on the Transformer encoder, and includes multiple stacked denoising coding layers, which correspond one-to-one with the coding layers of the feature extraction encoder of the conditional guidance module; the denoising decoder includes denoising decoding layers that correspond one-to-one with the denoising coding layers of the denoising encoder; The optical remote sensing image and noise mask of the training sample are concatenated and input into the denoising encoder. After multiple denoising coding layers, noise coding features of multiple scales are extracted and then transmitted to the denoising decoder. The noise coding features of multiple scales extracted by the denoising encoder are decoded by multiple denoising decoding layers to obtain denoising prediction masks of multiple scales. As the number of denoising coding layers increases, the scale of the extracted noise coding features decreases, and as the number of denoising decoding layers increases, the scale of the decoded denoising prediction mask increases.

[0007] The further technical solution is that each denoising coding layer includes multiple denoising coding modules, and each denoising coding module includes a Transformer feature extraction module, a conditional enhancement module, a multi-head attention mechanism module, a layer normalization layer and a feedforward network processing module connected sequentially from input to output; The conditional enhancement module integrates the iteration time step information into the noise features extracted by the Transformer feature extraction module, and fuses it with the guiding features extracted by the conditional guidance module to obtain the conditional enhancement features. The conditional enhancement features are processed by the multi-head attention mechanism module, the layer normalization layer and the feedforward network processing module to obtain denoising coding features with time and space awareness characteristics. The iteration time step information indicates the iteration time step corresponding to the noise mask.

[0008] The further technical solution is that each denoising decoding layer includes two 3×3 convolutional layers connected sequentially from input to output and a denoising output layer; The first denoising decoding layer fuses the denoised coding features extracted from the last denoising coding layer and the conditional guidance information extracted from the last decoding layer of the convolutional decoder of the conditional guidance module. The two 3×3 convolutional layers input to the denoising decoding layer then decode to obtain denoised decoding features with the same scale as the denoised coding features extracted from the last denoising coding layer. Starting from the second decoding layer, the denoised decoding features output from the previous denoising decoding layer are upsampled and fused with the denoised coding features extracted from the corresponding denoising coding layer and the conditional guidance information extracted from the corresponding decoding layer of the convolutional decoder of the conditional guidance module. The two 3×3 convolutional layers input to the denoising decoding layer then decode to obtain denoised decoding features with the same scale as the denoised coding features extracted from the corresponding denoising coding layer. The denoising output layer includes a cascaded convolutional prediction head and a Sigmoid activation function layer. The denoising output layer maps the denoised decoding features and outputs them as a denoised prediction mask.

[0009] Its further technical solution is that the knowledge distillation module includes a conditional denoising distillation module and a conditional guided self-distillation module; The conditional denoising distillation module uses the conditional prediction mask generated by the last decoding layer of the conditional guidance module. As a soft tag, the denoising module is guided to recover the fine structure of the salient target based on the difference between the soft tag and the denoising prediction mask generated by each decoding layer except the first decoding layer in the denoising module. The conditional self-distillation module uses the conditional prediction mask generated by the last decoding layer of the conditional self-distillation module. As a soft tag, the differences between the soft tag and the conditional prediction masks generated by each decoding layer in the conditional guidance module (excluding the first and last decoding layers) guide the consistency of features among the decoding layers within the conditional guidance module.

[0010] The further technical solution involves training a salient object detection model, including: The training samples from optical remote sensing images are input into the network architecture of the established salient object detection model. Based on the conditional prediction mask and denoised prediction mask output by the salient object detection model, and combined with the graffiti annotation mask of the optical remote sensing image training samples, a joint loss function is calculated. The model is trained using the optical remote sensing image training samples based on the joint loss function. Among them, condition-guided loss Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the graffiti annotation mask of the training samples of the optical remote sensing image; denoising loss Used to measure the difference between the denoised prediction mask generated by the denoising module and the graffiti annotation mask of the training samples of the optical remote sensing image; knowledge distillation loss. Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the denoising prediction mask generated by the denoising module, as well as the difference between features within the conditional guidance module.

[0011] The further technical solution is that the feature extraction encoder of the condition guidance module includes M stacked encoding layers, and the denoising encoder of the denoising module includes M stacked denoising encoding layers. The knowledge distillation loss ,in, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module The self-distillation loss, where i is an integer parameter and j is an integer parameter and .

[0012] A further technical solution is that the conditional prediction mask generated by the last decoding layer of the conditional guidance module. The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss ,in, It is the loss weight of the M+1-ith denoising decoding layer of the denoising module.

[0013] A further technical solution is that the conditional prediction mask generated by the last decoding layer of the conditional guidance module. The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module Self-distillation loss ,in, It is the loss weight of the (M+1-j)th decoding layer of the conditional guidance module.

[0014] The beneficial technical effects of this application are: This application proposes a weakly supervised salient target detection method for optical remote sensing images. Based on a diffusion model architecture, it constructs two branches: a conditional guidance module and a denoising module. The conditional guidance module accurately extracts image features from the optical remote sensing image and incorporates them as prior knowledge into the denoising process, providing effective guidance information on the location, boundary, and shape of salient targets. Simultaneously, the denoising module, based on a Transformer-CNN hybrid architecture, fully combines the advantages of Transformer in capturing long-range dependencies and global relationship modeling with the efficient local feature extraction capabilities of CNN, significantly improving detection accuracy. The multi-step iterative approach to recover salient targets based on the diffusion model ensures the acquisition of fine target boundaries and regions even with incomplete annotation information under weak supervision. This effectively reduces the dependence of traditional convolutional neural network-based salient target detection methods on pixel-by-pixel precise annotation data, greatly reducing annotation costs while improving detection performance.

[0015] By introducing a knowledge distillation mechanism into the diffusion model architecture, the knowledge transfer mechanism of knowledge distillation is effectively combined with the progressive generation capability of the diffusion model, further improving model performance. The conditional denoising module transfers knowledge from the conditional guidance module to the denoising module, accelerating the convergence process of the denoising module and effectively solving the problems of training instability and slow convergence speed caused by the multiple iterations of denoising in the diffusion model. The conditional guidance self-distillation module promotes the downward transmission of high-level semantic information generated by the conditional guidance module to the intermediate feature representation, making the soft labels generated by the conditional guidance module more discriminative, thereby providing more accurate supervision for the denoising module.

[0016] Moreover, compared to traditional knowledge distillation methods, the conditional guidance module in this application is not a pre-trained static teacher network, but a dynamic knowledge source optimized simultaneously with the denoising module. It compensates for the lack of information in weak supervision by generating high-quality soft labels. This online distillation strategy enables the conditional guidance module and the denoising module to promote each other and make progress together, forming a virtuous cycle. It effectively overcomes the problem of inaccurate boundary localization under weak supervision, and achieves high-precision salient target detection in optical remote sensing images under weak supervision, significantly improving the accuracy of salient target localization results and the boundary refinement effect. Attached Figure Description

[0017] Figure 1 This is a structural diagram of a salient target detection model in one embodiment.

[0018] Figure 2 This is a structural diagram of the conditional guidance module in one embodiment.

[0019] Figure 3 This is a structural diagram of the denoising encoding module of the denoising model.

[0020] Figure 4 This is a schematic diagram of the PR curve in the comparative experiment.

[0021] Figure 5 This is a comparison chart of the salience target detection results in the comparative experiment. Detailed Implementation

[0022] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] This application proposes a weakly supervised salient target detection method for optical remote sensing images. The specific steps of this salient target detection method are as follows: Step 1: Obtain an optical remote sensing image training sample set. Each optical remote sensing image training sample includes an optical remote sensing image and its graffiti annotation mask, wherein the graffiti annotation mask indicates the salient targets of the optical remote sensing image.

[0024] To mitigate the excessive annotation costs associated with traditional salient object detection methods that heavily rely on pixel-by-pixel fully supervised annotation, this application employs a weakly supervised salient object detection method based on graffiti annotation. Graffiti annotation is a sparse annotation method that only requires marking key regions of the target object with simple lines or colors in the image, rather than drawing a complete mask pixel by pixel. This method effectively reduces annotation costs.

[0025] Optical remote sensing image data is downloaded from publicly available datasets (such as ORSSD and EORSSD) and cloud platforms (such as GEE), and then graffiti annotations are generated using methods such as manual annotation or automated algorithm generation. For example, drawing software is used to mark key areas such as salient targets and backgrounds in the acquired optical remote sensing images with simple lines or colors; these lines and colors serve as the graffiti annotation results. Then, a script is written to batch convert the graffiti annotation results into binary images containing salient targets and backgrounds; these binary images are the graffiti annotation masks. Each optical remote sensing image and one graffiti annotation mask constitute an optical remote sensing image training sample.

[0026] Step 2: Construct the network architecture of the salient target detection model and train the salient target detection model using the training sample set of optical remote sensing images.

[0027] Traditional salient target detection methods rely on pixel-by-pixel labeled masks as supervision, directly modeling complex data distributions, resulting in low computational efficiency and a susceptibility to mode collapse. While computational efficiency improves under weak supervision, incomplete labeling information often makes it difficult to obtain fine boundaries and accurate target regions, leading to low accuracy in salient target detection. Unlike traditional methods, diffusion models establish a mapping relationship from noise to image data and employ iterative denoising to gradually recover salient targets. This allows for continuous optimization of prediction results in each denoising iteration, gradually approaching accuracy even if initial labeling is inaccurate. Therefore, to improve the accuracy of salient target detection under weak supervision, this application constructs a salient target detection model for optical remote sensing images based on a diffusion model architecture.

[0028] Based on the diffusion model architecture, two branches are constructed: a conditional guidance module and a denoising module. One branch generates prior knowledge, and the other guides the denoising process based on the prior knowledge. To further improve computational efficiency, a knowledge distillation mechanism is introduced into the diffusion model architecture. This not only facilitates knowledge transfer between different branches but also enables self-distillation within the conditional guidance module, effectively combining the knowledge transfer mechanism of knowledge distillation with the progressive generation capability of the diffusion model. The salient target detection model includes a conditional guidance module, a denoising module, and a knowledge distillation module. The conditional guidance module extracts multi-scale features from optical remote sensing images and generates conditional guidance information and conditional prediction masks. The denoising module generates a noise mask and recovers the salient target from the noise mask based on the conditional guidance information to obtain the denoised prediction mask. The knowledge distillation module uses the conditional prediction mask generated by the conditional guidance module as a soft label to guide the denoising module in recovering the salient target and to guide feature learning within the conditional guidance module.

[0029] Considering the vast differences in the size of ground features in optical remote sensing images, such as buildings, roads, rivers, and mountains, multi-scale features are needed to simultaneously capture details and semantic information. The Pyramid Visual Transformer (PVT) combines global relational modeling capabilities with the computational efficiency of multi-scale pyramids, ensuring the extraction of guiding features at multiple scales while effectively improving computational efficiency. This application experimentally verifies that the performance at four scales is superior. The network architecture of the salient object detection model based on a four-layer pyramid structure is as follows: Figure 1 As shown.

[0030] The conditional guidance module, as a prior knowledge generation module, plays a guiding role in the results of salient target detection. Especially in the practical application of a well-trained salient target detection model, the model input is only an optical remote sensing image. The model needs to directly recover the salient target from the generated random noise mask, which requires the conditional guidance module to provide more accurate prior knowledge.

[0031] In one embodiment, the conditional guidance module includes a cascaded feature extraction encoder and a convolutional decoder. The feature extraction encoder is designed based on the Pyramid Vision Transformer and includes multiple stacked coding layers. Different coding layers extract guidance features of different scales from the optical remote sensing image and then transmit them to the convolutional decoder. The convolutional decoder includes decoding layers that correspond one-to-one with the coding layers of the feature extraction encoder. The optical remote sensing image of the training sample is input into the feature extraction encoder, and after being processed by multiple coding layers to extract guidance features of multiple scales, it is transmitted to the convolutional decoder. The guidance features of multiple scales extracted by the feature extraction encoder are decoded by multiple decoding layers to obtain conditional guidance information and conditional prediction masks of multiple scales. As the number of coding layers increases, the scale of the extracted guidance features decreases, while as the number of decoding layers increases, the scale of the decoded conditional guidance information and conditional prediction masks increases.

[0032] Each decoding layer consists of two convolutional blocks cascaded together and a prediction output layer. Each convolutional block includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function layer connected sequentially from input to output. The convolutional decoder employs a top-down feature transfer mechanism. The first decoding layer inputs the guiding features extracted from the last encoding layer into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the last encoding layer. Starting from the second decoding layer, the conditional guiding information output from the previous decoding layer is upsampled and fused with the guiding features extracted from the corresponding encoding layer. This fusion is then input into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the corresponding encoding layer. The prediction output layer consists of a cascaded 1×1 convolutional layer and a Sigmoid activation function layer. The prediction output layer maps the conditional guiding information to a conditional prediction mask.

[0033] The conditional guidance module with 4 encoding layers and 4 decoding layers has the following specific structure: Figure 2 As shown. Figure 2 The feature extraction encoder has four encoding layers. The spatial resolution of the guided features extracted by the feature extraction encoder decreases by a factor of two as the number of layers increases. The convolutional decoder has four decoding layers, each corresponding to one of the encoding layers of the feature extraction encoder. The spatial resolution of the conditional guidance information and conditional prediction mask decoded by the decoding layers of the convolutional decoder increases by a factor of two as the number of layers increases. The first encoding layer corresponds to the fourth decoding layer, the second encoding layer corresponds to the third decoding layer, the third encoding layer corresponds to the second decoding layer, and the fourth encoding layer corresponds to the first decoding layer. The guided features extracted by the first encoding layer have the same scale as the input optical remote sensing image, and the conditional guidance information and conditional prediction mask decoded by the fourth decoding layer also have the same scale as the input optical remote sensing image. The four scale guided features extracted by the feature extraction encoder are as follows: The convolutional decoder guides features from the deepest layer. The process begins with upsampling layer by layer and fusing it with the guiding features extracted from the corresponding coding layer. The decoded conditional guiding information at four scales is as follows: The specific decoding calculation process is as follows:

[0034]

[0035] Among them, the decoding layer , This indicates a bilinear interpolation upsampling operation with a spatial resolution scaling factor of 2. This represents a feature connection operation along the channel dimension. and These represent two concatenated convolutional blocks. The four decoding layers are processed by the conditional output layer to obtain conditional prediction masks at four scales. .

[0036] The denoising module generates a noise mask and recovers the salient target from the noise mask based on conditional guidance information. During model training, the noise mask is generated from the graffiti annotation mask through the forward pass of the diffusion model; during the inference process of model application, the noise mask is a randomly generated Gaussian noise mask. Recovering the salient target mask from the noise mask must ensure both the accurate location and detailed information of the detected salient targets. This requires the denoising model to have both global relational modeling and local feature extraction capabilities. The Transformer model is characterized by its excellent global modeling ability, while the CNN convolutional neural network is characterized by its high detail extraction capability. Therefore, this application combines Transformer and CNN for denoising.

[0037] In one embodiment, the denoising module includes a noise mask generation module, a denoising encoder, and a denoising decoder; the noise mask generation module generates a graffiti-annotated mask for training samples of optical remote sensing images based on a forward process of a diffusion model. Multiple noise masks are obtained by progressively adding Gaussian noise according to each iteration time step, with each noise mask corresponding to one iteration time step; the noise mask corresponding to any generated iteration time step t is... ,in, It is standard Gaussian noise. This is the noise figure. This process aims to homogenize the data distribution, making the final noise distribution easier to model.

[0038] The denoising encoder is designed based on the Transformer encoder and includes multiple stacked denoising coding layers, each corresponding one-to-one with the coding layers of the feature extraction encoder in the conditional guidance module. The denoising decoder includes denoising decoding layers that correspond one-to-one with the denoising coding layers of the denoising encoder. Figure 1 Taking the 4-layer structure shown as an example, the first denoising coding layer corresponds to the first coding layer of the feature extraction encoder of the conditional guidance module, and the fourth denoising coding layer corresponds to the fourth coding layer of the feature extraction encoder of the conditional guidance module; the first denoising decoding layer of the denoising decoder corresponds to the fourth denoising coding layer, and the fourth denoising decoding layer of the denoising decoder corresponds to the first denoising coding layer.

[0039] To increase the information content of the input model, the optical remote sensing images and noise masks of the training samples are concatenated and then input into the denoising encoder. After passing through multiple denoising coding layers, noise coding features at multiple scales are extracted and then transmitted to the denoising decoder. The noise coding features extracted by the denoising encoder at multiple scales are decoded by multiple denoising decoding layers to obtain denoised prediction masks at multiple scales. As the number of denoising coding layers increases, the scale of the extracted noise coding features decreases, while as the number of denoising decoding layers increases, the scale of the decoded denoised prediction mask increases. Similarly, the scale changes by a factor of 2 for each layer.

[0040] The RGB three-channel optical remote sensing images and noise mask of the training samples are used. By splicing along the channel dimension, a four-channel input is constructed. Then the input Basic features are extracted through a single convolutional layer. Then The noise coding features at four scales are obtained by extracting features from the input four-layer denoising coding layer.

[0041] Each denoising coding layer comprises multiple denoising coding modules. Each module includes, sequentially from input to output, a Transformer feature extraction module, a conditional enhancement module, a multi-head attention mechanism module, a layer normalization layer, and a feedforward network processing module. The conditional enhancement module incorporates iteration time step information into the noise features extracted by the Transformer feature extraction module and fuses it with the guiding features extracted by the conditional guidance module to obtain conditionally enhanced features. These enhanced features are then processed by the multi-head attention mechanism module, layer normalization layer, and feedforward network processing module to obtain denoising coding features with temporal and spatial awareness. The iteration time step information indicates the iteration time step corresponding to the noise mask. The structure of the denoising coding module is as follows: Figure 3 As shown, the specific processing procedure is as follows: (1) In the diffusion model, the iteration time step t is an important parameter in the model generation process, representing the progress of the diffusion process. For each iteration time step t, the model needs to know which step the diffusion has reached in order to adjust the denoising operation in a timely manner. Therefore, the iteration time step information is particularly important. This is achieved by encoding the iteration time step into an iteration time step embedding. The iteration time step information is represented by this. For any denoising coding layer i, the conditional enhancement module incorporates the iteration time step information into the noise features extracted by the Transformer feature extraction module. In, and the guidance features extracted by the conditional guidance module. Fusion yields conditional enhancement features This preprocessing method ensures that temporal information and conditional semantics can effectively guide the subsequent denoising process.

[0042] (2) Enhanced conditional enhancement features After layer normalization, the data is fed into a multi-head attention mechanism to capture long-range dependencies between different spatial locations, resulting in the following output features:

[0043] in, This indicates a multi-head self-attention mechanism. The layer normalization operation is represented. The multi-head self-attention mechanism and the layer normalization operation adopt conventional model structures and processing methods, and the specific details will not be elaborated in this application.

[0044] (3) Output characteristics A The denoised coding features are obtained again after layer normalization and feedforward network processing. for:

[0045] in, This represents a feedforward network, consisting of two linear layers with the GELU activation function.

[0046] pass Figure 3 The network structure shown enables the denoising encoding module to fully perceive temporal conditions and conditional semantic guidance, improving the model's ability to model complex scenes. The denoising encoder with four denoising encoding layers extracts denoised features at four scales, respectively. .

[0047] Based on the multi-scale denoising coding features extracted by the denoising encoder, a multi-layer superimposed denoising decoding layer is constructed to gradually upsample and restore the spatial resolution of the output result. Each denoising decoding layer consists of two 3×3 convolutional layers connected sequentially from input to output, and a denoising output layer. The first denoising decoding layer fuses the denoised coding features extracted from the last denoising coding layer with the conditional guidance information extracted from the last decoding layer of the convolutional decoder in the conditional guidance module. The two 3×3 convolutional layers input to this denoising decoding layer then decode to obtain denoised decoding features with the same scale as the denoised coding features extracted from the last denoising coding layer. Starting from the second decoding layer, the denoised decoding features output from the previous denoising decoding layer are upsampled and fused with the denoised coding features extracted from the corresponding denoising coding layer and the conditional guidance information extracted from the corresponding decoding layer of the convolutional decoder in the conditional guidance module. The two 3×3 convolutional layers input to this denoising decoding layer then decode to obtain denoised decoding features with the same scale as the denoised coding features extracted from the corresponding denoising coding layer. The denoising output layer consists of a cascaded convolutional prediction head and a sigmoid activation function layer. The denoising output layer maps the denoised decoding features and outputs a denoised prediction mask. The specific processing procedure of the denoising decoding layer is as follows: (1) First, information fusion is performed based on the denoising coding features extracted by the denoising coding layer and the conditional guidance information extracted by the decoding layer of the convolutional decoder of the conditional guidance module to obtain the fused features as follows:

[0048]

[0049] in, It is the fusion feature of the first denoising and decoding layer. This indicates channel-dimensional concatenation; the fused features obtained after information fusion from the four denoising decoding layers are processed by two 3×3 convolutional modules respectively to obtain denoising decoding features at four scales. .

[0050] (2) A convolutional prediction head is connected to the denoised decoding features, and a denoised prediction mask is generated by using the Sigmoid activation function:

[0051] in, Indicates the convolutional prediction head. This represents the Sigmoid activation function. The denoised decoding features at the four scales are processed by the denoised output layer to obtain denoised prediction masks at the four scales, respectively. .

[0052] Since the diffusion model is based on multi-step iterative denoising, although it has high accuracy, the repeated iterations also increase the computational load. To further improve the computational efficiency of the model, this application introduces the knowledge distillation method into the diffusion model. Knowledge distillation provides high-quality soft labels for the denoising module to compensate for the lack of graffiti annotation information, while further accelerating the denoising convergence.

[0053] In one embodiment, the knowledge distillation module includes a conditional denoising distillation module and a conditional guided self-distillation module. The conditional prediction mask output by the conditional guidance module is directly used as a soft label to perform knowledge distillation on the denoising module, accelerating the convergence of the denoising model and improving prediction accuracy. Simultaneously, a self-distillation process is incorporated within the conditional guidance module to enhance its own representational capabilities and improve the quality of soft label generation.

[0054] The conditional denoising distillation module uses the conditional prediction mask generated by the last decoding layer of the conditional guidance module as a soft label. Based on the difference between the soft label and the denoising prediction mask generated by each decoding layer in the denoising module except for the first decoding layer, the module guides the denoising module to recover the fine structure of the salient target. The conditional self-distillation module uses the conditional prediction mask generated by the last decoding layer of the conditional guidance module as a soft label. Based on the difference between the soft label and the conditional prediction masks generated by each decoding layer in the conditional guidance module except for the first and last decoding layers, it guides the consistency of features among the decoding layers within the conditional guidance module.

[0055] Based on the salient target detection model established above, the specific method for training this model using a training sample set of optical remote sensing images is as follows: The training samples from optical remote sensing images are input into the network architecture of the established salient object detection model. Based on the conditional prediction mask and denoised prediction mask output by the salient object detection model, and combined with the graffiti annotation mask from the training samples of the optical remote sensing images, the joint loss function is calculated. The model is trained using training samples from optical remote sensing images based on the joint loss function.

[0056] Among them, condition-guided loss Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the graffiti annotation mask of the training samples of the optical remote sensing image; denoising loss Used to measure the difference between the denoised prediction mask generated by the denoising module and the graffiti annotation mask of the training samples of the optical remote sensing image; knowledge distillation loss. Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the denoising prediction mask generated by the denoising module, as well as the difference between features within the conditional guidance module.

[0057] Since the knowledge distillation module includes a conditional denoising distillation module and a conditional guided self-distillation module, the knowledge distillation loss is calculated from the conditional denoising loss and the self-distillation loss. In one embodiment, the feature extraction encoder of the conditional guided module includes M stacked coding layers, and the denoising encoder of the denoising module includes M stacked denoising coding layers; Knowledge distillation loss ,in, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module The self-distillation loss, where i is an integer parameter and j is an integer parameter and .

[0058] This application minimizes the information difference between the true and predicted distributions by minimizing the KL divergence between the output distributions of the conditional guidance module and the denoising module, thereby causing the denoised prediction mask output by the denoising module to approximate the soft label output by the conditional guidance module. Specifically, the denoised prediction mask and soft label output by each denoising decoding layer of the denoising module (excluding the first denoising decoding layer) are calculated respectively. The conditional denoising loss is obtained by applying the KL divergence loss between the layers and assigning corresponding loss weights according to the accuracy of the prediction results of each denoising decoding layer. .

[0059] In one embodiment, the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss ,in, This refers to the loss weight of the (M+1-i)th denoising decoding layer in the denoising module. Since a deeper denoising decoding layer results in an image resolution that more closely approximates the original optical remote sensing image for the obtained denoised prediction mask, a deeper denoising decoding layer corresponds to a lower loss weight. The specific loss weight value is customized according to actual needs. For a conditional guidance module with 4 decoding layers and a denoising module with 4 denoising decoding layers, this application sets the loss weight of the 4th denoising decoding layer. =0.5, the loss weight of the third denoising and decoding layer =0.1, the loss weight of the second denoising and decoding layer =0.1.

[0060] Similarly, this application minimizes the information difference between the true and predicted distributions by minimizing the KL divergence of the output distributions of each decoding layer within the conditional guidance module, thereby prompting the conditional guidance module to output higher-quality soft tags. Specifically, the conditional prediction mask and soft tag outputs of each decoding layer (excluding the first and last decoding layers) of the conditional guidance module are calculated respectively. The conditional self-distillation loss is obtained by applying the KL divergence loss between the layers and assigning corresponding loss weights according to the accuracy of the prediction results of each decoding layer. .

[0061] In one embodiment, the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module Self-distillation loss ,in, This refers to the loss weight of the (M+1-j)th decoding layer of the conditional guidance module. The specific loss weight value can be customized according to actual needs. For a conditional guidance module with 4 decoding layers, this application sets the loss weight of the 3rd denoising decoding layer. =0.1, the loss weight of the second denoising and decoding layer =0.1. By setting KL divergence constraints, the conditional guidance module forms consistency constraints across different levels, which enhances the representation learning ability of the conditional guidance module, thereby generating more discriminative feature representations, ensuring the generation of accurate prior knowledge, and improving the accuracy of the denoising module's prediction results.

[0062] For the training process of the conditional guidance module, cross-entropy loss and structure-aware loss are used for supervision to optimize the network parameters of the conditional guidance module. Specific calculation methods for cross-entropy loss and structure-aware loss can be found in existing technologies and will not be elaborated upon here. Conditional Guidance Loss The calculation formula is:

[0063] in, It is the truth value of the conditional prediction mask. Scr It is the pixel set of the graffiti annotation mask. G It is a structure-aware gating mechanism. d Is and Partial derivatives in the direction, x , yThese are the width and height directions of the image. a It is a constant coefficient. I It is an identity matrix.

[0064] For the training process of the denoising module, cross-entropy loss and structure-aware loss are also used for supervision to optimize the network parameters of the denoising module. The calculation formula is:

[0065] in, It is the ground truth of the denoising prediction mask. It is the denoised prediction mask output by the last denoised decoding layer of the denoising module.

[0066] During training, the optical remote sensing image training sample set is divided into a training set and a validation set. This application uses training sample data with a batch size of 4 for iteration and uses the Adam optimizer for parameter updates. The initial learning rate is set to... The learning rate was dynamically adjusted using a cosine annealing strategy, and a total of 25 training cycles were performed. Finally, the model parameters with the best performance on the validation set were selected as the final training result, resulting in a salient object detection model.

[0067] Step 3: Use the trained salient target detection model to detect salient targets in the optical remote sensing image under test to obtain the salient target detection results. The denoised prediction mask output by the last layer of the denoising decoder in the denoising module of the salient target detection model is the final salient target detection result.

[0068] The effectiveness of the proposed method was further verified through comparative experiments. The ORSSD and EORSSD remote sensing image datasets, both annotated with graffiti, were used to perform salient target detection using the proposed method. The ORSSD dataset contains 800 optical remote sensing images, with 600 used for training and 200 for testing. This dataset covers various complex scenes and target types. The EORSSD dataset is an extended version of ORSSD, containing 2000 optical remote sensing images, with 1400 used for training and 600 for testing. It covers a wider variety of land cover types. The specific experimental environment was: Python 3.9, CPU: i9-12900K, GPU: GTX-3090Ti, and 64GB of memory.

[0069] Comparative experiments selected 23 advanced algorithms from existing salient target detection methods, categorized into traditional optical remote sensing image salient target detection methods, deep learning-based fully supervised optical remote sensing image salient target detection methods, and deep learning-based weakly supervised optical remote sensing image salient target detection methods. Experiments were conducted on the EORSSD and ORSSD datasets, and the experimental data were compared with the method in this application. Four metrics were calculated to evaluate the final salient target detection results: mean absolute error (MAE). ), structural metric ( ), F-metric ( ) and E metric ( The specific experimental results are shown in the table below: Table 1 Comparison of saliency target detection results between this application and other methods

[0070] The table presents a quantitative comparison of the overall performance of the proposed method and other methods on benchmark datasets for salient target detection in optical remote sensing images, where Ours represents the proposed method. According to the evaluation metrics data of the comparative experiments in the table, the proposed method achieves the best results on all metrics across both benchmark datasets compared to traditional optical remote sensing image salient target detection algorithms. Compared to weakly supervised deep learning-based optical remote sensing image salient target detection algorithms, the proposed method outperforms existing methods in all four metrics on the ORSSD dataset; on the EORSSD dataset, it only… In terms of metrics, it is slightly lower than GLSIN by 0.6%, but outperforms existing methods in the other three metrics. Compared with deep learning-based fully supervised optical remote sensing image salient target detection algorithms, under weakly supervised annotation training conditions with incomplete pixel-level annotations and imprecise boundary supervision, this invention achieves better results on two datasets. , and The indicator is still better than half of the fully supervised methods. The metrics outperform most fully supervised methods.

[0071] The PR curves (precision-recall curves) of this application method are compared with those of other existing methods as follows: Figure 4 As shown in the figure, the red curve represents the method of this application. It can be seen that the method of this application can handle salient targets in complex scenes well under different threshold ranges. The visualization comparison results of the salient target mask of the salient target detection results of this application method and other existing methods are shown in the figure. Figure 5 As shown, compared with other methods, the method of this application can handle various challenging scenarios and generate prediction result maps with more accurate target localization and more detailed boundaries.

[0072] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A method for weakly supervised salient target detection in optical remote sensing images, characterized in that, The salient target detection method includes: Obtain a training sample set of optical remote sensing images. Each optical remote sensing image training sample includes an optical remote sensing image and its graffiti annotation mask, wherein the graffiti annotation mask indicates the salient targets of the optical remote sensing image. A network architecture for a salient target detection model is constructed and trained using a training sample set of optical remote sensing images. The salient target detection model includes a conditional guidance module, a denoising module, and a knowledge distillation module. The conditional guidance module extracts multi-scale features from the optical remote sensing images and generates conditional guidance information and a conditional prediction mask. The conditional guidance module includes a cascaded feature extraction encoder and a convolutional decoder. The feature extraction encoder is designed based on the pyramid vision Transformer. The denoising module generates a noise mask and recovers the salient target from the noise mask based on the conditional guidance information to obtain a denoised prediction mask. The denoising module includes a noise mask generation module, a denoising encoder, and a denoising decoder. The noise mask generation module, based on the forward process of the diffusion model, progressively adds Gaussian noise to the graffiti annotation masks of the training samples of the optical remote sensing image according to the iteration time step to obtain multiple noise masks, each noise mask corresponding to one iteration time step; the denoising encoder is designed based on the Transformer encoder and includes multiple stacked denoising coding layers, which correspond one-to-one with the coding layers of the feature extraction encoder of the conditional guidance module; the denoising decoder includes denoising decoding layers that correspond one-to-one with the denoising coding layers of the denoising encoder; the knowledge distillation module is used to use the conditional prediction mask generated by the conditional guidance module as a soft label to guide the denoising module to recover the salient target, and to guide the feature learning within the conditional guidance module; The trained salient target detection model is used to detect salient targets in the optical remote sensing image under test to obtain salient target detection results; Each denoising coding layer includes multiple denoising coding modules, and each denoising coding module includes a Transformer feature extraction module, a conditional enhancement module, a multi-head attention mechanism module, a layer normalization layer, and a feedforward network processing module connected sequentially from input to output. The conditional enhancement module integrates the iteration time step information into the noise features extracted by the Transformer feature extraction module, and fuses it with the guiding features extracted by the conditional guidance module to obtain the conditional enhancement features. The conditional enhancement features are processed by the multi-head attention mechanism module, the layer normalization layer, and the feedforward network processing module to obtain denoising coding features with time and space awareness characteristics; the iteration time step information indicates the iteration time step corresponding to the noise mask. The knowledge distillation module includes a conditional denoising distillation module and a conditional guided self-distillation module; The conditional denoising distillation module uses the conditional prediction mask generated by the last decoding layer of the conditional guidance module as a soft label. Based on the difference between the soft label and the denoising prediction mask generated by each decoding layer in the denoising module except for the first decoding layer, the module guides the denoising module to recover the fine structure of the salient target. The condition-guided self-distillation module uses the condition prediction mask generated by the last decoding layer of the condition-guided module as a soft label. Based on the difference between the soft label and the condition prediction masks generated by each decoding layer in the condition-guided module except for the first and last decoding layers, it guides the characteristics of each decoding layer in the condition-guided module to be consistent.

2. The salient target detection method according to claim 1, characterized in that, The feature extraction encoder consists of multiple stacked coding layers. Different coding layers extract guiding features of different scales from the optical remote sensing image and then transmit them to the convolutional decoder. The convolutional decoder includes decoding layers that correspond one-to-one with the coding layers of the feature extraction encoder. The optical remote sensing image of the training sample is input into the feature extraction encoder, and after being extracted by multiple coding layers to obtain guiding features of multiple scales, it is transmitted to the convolutional decoder. The guiding features of multiple scales extracted by the feature extraction encoder are decoded by multiple decoding layers to obtain conditional guiding information and conditional prediction masks of multiple scales. As the number of coding layers increases, the scale of the extracted guiding features decreases; as the number of decoding layers increases, the scale of the conditional guiding information and conditional prediction mask obtained from decoding increases. Each decoding layer consists of two convolutional blocks cascaded together and a prediction output layer. Each convolutional block includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function layer connected sequentially from input to output. The convolutional decoder employs a top-down feature transfer mechanism. The first decoding layer inputs the guiding features extracted from the last encoding layer into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the last encoding layer. Starting from the second decoding layer, the conditional guiding information output from the previous decoding layer is upsampled and fused with the guiding features extracted from the corresponding encoding layer. This fusion is then input into the two convolutional blocks of the decoding layer to decode conditional guiding information with the same scale as the guiding features extracted from the corresponding encoding layer. The prediction output layer consists of a cascaded 1×1 convolutional layer and a Sigmoid activation function layer. The prediction output layer maps the conditional guiding information to a conditional prediction mask.

3. The salient target detection method according to claim 2, characterized in that, The optical remote sensing image and noise mask of the training sample are concatenated and input into the denoising encoder. After multiple denoising coding layers, noise coding features of multiple scales are extracted and then transmitted to the denoising decoder. The noise coding features of multiple scales extracted by the denoising encoder are decoded by multiple denoising decoding layers to obtain denoising prediction masks of multiple scales. As the number of denoising coding layers increases, the scale of the extracted noise coding features decreases, while as the number of denoising decoding layers increases, the scale of the decoded denoising prediction mask increases.

4. The salient target detection method according to claim 1, characterized in that, Each denoising decoding layer consists of two 3×3 convolutional layers connected sequentially from input to output and a denoising output layer; The first denoising decoding layer fuses the denoised coding features extracted by the last denoising coding layer and the conditional guidance information extracted by the last decoding layer of the convolutional decoder of the conditional guidance module. The two 3×3 convolutional layers input to the denoising decoding layer decode the denoised decoding features with the same scale as the denoised coding features extracted by the last denoising coding layer. Starting from the second decoding layer, the denoised decoding features output by the previous denoising decoding layer are upsampled and fused with the denoised coding features extracted by the corresponding denoising coding layer and the conditional guidance information extracted by the corresponding decoding layer of the convolutional decoder of the conditional guidance module. The two 3×3 convolutional layers input to the denoising decoding layer decode the denoised decoding features with the same scale as the denoised coding features extracted by the corresponding denoising coding layer. The denoising output layer consists of a cascaded convolutional prediction head and a sigmoid activation function layer. The denoising output layer outputs the denoised decoded feature map as a denoised prediction mask.

5. The salient target detection method according to claim 1, characterized in that, The training yields a salient object detection model including: The training samples from optical remote sensing images are input into the network architecture of the established salient object detection model. Based on the conditional prediction mask and denoised prediction mask output by the salient object detection model, and combined with the graffiti annotation mask of the optical remote sensing image training samples, a joint loss function is calculated. The model is trained using the optical remote sensing image training samples based on the joint loss function. Among them, condition-guided loss Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the graffiti annotation mask of the training samples of the optical remote sensing image; denoising loss Used to measure the difference between the denoised prediction mask generated by the denoising module and the graffiti annotation mask of the training samples of the optical remote sensing image; knowledge distillation loss. Used to measure the difference between the conditional prediction mask generated by the conditional guidance module and the denoising prediction mask generated by the denoising module, as well as the difference between features within the conditional guidance module.

6. The salient target detection method according to claim 5, characterized in that, The feature extraction encoder of the conditional guidance module includes M stacked encoding layers, and the denoising encoder of the denoising module includes M stacked denoising encoding layers. The knowledge distillation loss ,in, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss, It is the conditional prediction mask generated by the last decoding layer of the conditional bootstrapping module. The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module The self-distillation loss, where i is an integer parameter and j is an integer parameter and .

7. The salient target detection method according to claim 6, characterized in that, The conditional prediction mask generated by the last decoding layer of the conditional guidance module The denoising prediction mask generated by the (M+1-i)th denoising decoding layer of the denoising module Conditional denoising loss ,in, It is the loss weight of the M+1-ith denoising decoding layer of the denoising module.

8. The salient target detection method according to claim 6, characterized in that, The conditional prediction mask generated by the last decoding layer of the conditional guidance module The conditional prediction mask generated by the (M+1-j)th decoding layer of the conditional guidance module Self-distillation loss ,in, It is the loss weight of the (M+1-j)th decoding layer of the conditional guidance module.

Images