A remote sensing target detection method based on diffusion model

Abstract

The present invention belongs to the technical field of pattern recognition, and specifically discloses a remote sensing target detection method based on a diffusion model, comprising the following steps: obtaining remote sensing image training data; using the remote sensing image training data to train a neural network to obtain a remote sensing target detection model; the remote sensing target detection model comprises a visual feature extraction module, a multi-scale feature extraction module and an instance-level target detection module; the visual feature extraction module is a pre-trained basic diffusion model; the multi-scale feature extraction module is used to extract features from an initial feature map to obtain a multi-level feature map; the target remote sensing image data is input into the remote sensing target detection model, and the detection result of the target remote sensing image is output through the remote sensing target detection model. The present invention can reduce the demand for training resources, shorten the learning evolution time, and improve the adaptability and scalability of the model. The problem of relatively complex training resources in the remote sensing image data detection process in the prior art is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of pattern recognition, and in particular relates to a remote sensing target detection method based on a diffusion model. Background Art

[0002] In recent years, with the extensive exploration of deep learning models, significant progress has been made in computer vision tasks. Among them, diffusion models have emerged as a class of powerful and versatile generative models, achieving remarkable success in a variety of computer vision tasks. Inspired by the physical process of diffusion, diffusion models provide a powerful framework for modeling the distribution of complex data and generating high-fidelity images. Diffusion models gradually transform simple noise distributions into target data distributions through a series of reversible steps. This iterative denoising process is usually guided by a neural network, enabling diffusion models to generate highly detailed and coherent images. The application of diffusion models in computer vision covers a wide range of tasks, including image generation, image restoration, super-resolution, etc. Their versatility and robustness make them suitable for solving standard and challenging vision problems, paving the way for new innovations and improvements in image processing and synthesis. As the research community continues to explore and refine these models, diffusion-based methods will play a key role in the future of computer vision, providing new tools and methods for creating, analyzing, and understanding visual data.

[0003] Remote sensing object detection is a rapidly growing area in computer vision that focuses on identifying and localizing objects from aerial images, which are typically captured by drones, satellites, or other airborne platforms. This technology has attracted much attention due to its wide range of applications in various fields such as urban planning, agriculture, environmental monitoring, disaster management, and defense. The unique perspective provided by remote sensing images allows for comprehensive analysis of large and complex landscapes, which is usually not possible with ground-based methods. However, compared with traditional object detection tasks, remote sensing object detection faces unique challenges. The high altitude at which images are captured leads to variations in object scale, occlusion, and background environment, all of which complicate the detection process. Remote sensing object detection utilizes advanced deep learning models, especially convolutional neural networks (CNNs) and their extensions, to address these challenges. These models excel at learning hierarchical features directly from raw pixel data, which enables them to robustly detect objects at different scales and orientations. Techniques such as multi-scale feature aggregation, anchor box refinement, and region proposal networks have been specifically tailored to improve the performance of object detectors in aerial imagery. However, there has been limited exploration of diffusion models in remote sensing object detection tasks. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a remote sensing target detection method based on a diffusion model, which solves the problem of relatively complex training resources in the remote sensing image data detection process in the prior art.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a remote sensing target detection method based on a diffusion model, comprising the following steps:

[0006] S1, obtaining remote sensing image training data;

[0007] S2, using remote sensing image training data to train the neural network to obtain a remote sensing target detection model;

[0008] The remote sensing target detection model includes a visual feature extraction module, a multi-scale feature extraction module, and an instance-level target detection module;

[0009] The visual feature extraction module is a pre-trained basic diffusion model, which is used to extract the image features of the remote sensing image training data and obtain the initial feature map of the remote sensing image training data;

[0010] The multi-scale feature extraction module is used to extract features from the initial feature map to obtain a multi-level feature map;

[0011] The instance-level object detection module is used to perform object detection on multi-level feature maps and output the detection results;

[0012] S3. Input the target remote sensing image data into the remote sensing target detection model, and output the detection result of the target remote sensing image through the remote sensing target detection model.

[0013] The beneficial effects of the above scheme are:

[0014] (1) In the process of target remote sensing image data detection, the present invention can fix the training parameters of the basic model VAE encoder and only train the parameters of the lightweight multi-scale feature extraction module and the instance-level target detection module, thereby reducing the demand for training resources, shortening the learning evolution time, and improving the adaptability and scalability of the model.

[0015] (2) In the process of detecting target remote sensing image data, the present invention can use the prior knowledge of the basic model to make predictions, thereby improving the accuracy and efficiency of detection.

[0016] Further, in step S2, the pre-trained basic diffusion model includes a feature encoder and a UNet network;

[0017] Among them, the feature encoder is used to extract image features of remote sensing image training data;

[0018] The UNet network is used to further encode the image features of the remote sensing image training data to obtain the initial feature map.

[0019] Furthermore, in step S2, the dimensions of the multi-level feature maps are the same.

[0020] The beneficial effect of the above further scheme is: the present invention utilizes the pre-trained basic diffusion model Stable-Diffusion to freeze all parameters of the original image encoder during the visual feature extraction process, and improves the generalization ability of the model through data enhancement operations such as random cropping during the training process.

[0021] Furthermore, in step S2, the instance-level object detection module includes 9 stacked sub-modules, and the sub-modules include Cross-Attention, Self-Attention and Feedforward Neural Network FFN.

[0022] Furthermore, Cross-Attention is used to learn the categories of remote sensing image training data;

[0023] Self-Attention is used to learn prediction boxes in remote sensing image training data;

[0024] Feed-forward neural network (FFN) is used to learn prediction masks in remote sensing image training data.

[0025] Furthermore, in step S2, the loss function of the remote sensing target detection model is for:

[0026]

[0027] in, represents the classification loss, represents the predicted probability, Indicates the instance category, represents the detection box loss, represents the prediction box, represents the real frame, represents the mask loss, represents the prediction mask, represents the instance mask.

[0028] Furthermore, the pre-trained basic diffusion model is Stable-Diffusion, and the parameters of the feature encoder of the pre-trained basic diffusion model remain unchanged.

[0029] The beneficial effects of the above further scheme are: improving the efficiency and accuracy of remote sensing image target detection, and at the same time, by optimizing the model structure and loss function design, making the entire system more robust and adaptable, providing powerful tools and methods for solving complex practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1Shown is a flow chart of a remote sensing target detection method based on a diffusion model.

[0031] Figure 2 Shown is a schematic diagram of the model structure of a remote sensing target detection method based on a diffusion model. DETAILED DESCRIPTION

[0032] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0033] like Figure 1 As shown, a remote sensing target detection method based on a diffusion model includes the following steps:

[0034] S1. Obtain remote sensing image training data.

[0035] S2, using remote sensing image training data to train the neural network to obtain a remote sensing target detection model;

[0036] The remote sensing target detection model includes a visual feature extraction module, a multi-scale feature extraction module, and an instance-level target detection module;

[0037] The visual feature extraction module is a pre-trained basic diffusion model, which is used to extract the image features of the remote sensing image training data and obtain the initial feature map of the remote sensing image training data;

[0038] The multi-scale feature extraction module is used to extract features from the initial feature map to obtain a multi-level feature map;

[0039] The instance-level object detection module is used to perform object detection on multi-level feature maps and output the detection results.

[0040] In this embodiment, in step S2, the pre-trained basic diffusion model includes a feature encoder and a UNet network;

[0041] Among them, the feature encoder is used to extract image features of remote sensing image training data;

[0042] The UNet network is used to further encode the image features of the remote sensing image training data to obtain the initial feature map.

[0043] In this embodiment, the dimensions of the multi-level feature maps in step S2 are the same.

[0044] Exemplarily, the dimension of the multi-level feature map represents the dimension of the pixel-by-pixel vector in the multi-level feature map. For example, the multi-level feature map can be a 3-level feature map (each level of feature map can be obtained by upsampling or downsampling the previous level of image), and its size (number of pixels) is respectively 1:1 of the original image compared to the remote sensing image in the remote sensing image training data. , and . Among them, each pixel is represented by a vector, and the dimension of the vector is the dimension of the multi-level feature map.

[0045] In this embodiment, the instance-level target detection module includes 9 stacked sub-modules, and the sub-modules include Cross-Attention, Self-Attention and Feedforward Neural Network FFN.

[0046] In this embodiment, Cross-Attention is used to learn the categories of remote sensing image training data;

[0047] Self-Attention is used to learn prediction boxes in remote sensing image training data;

[0048] Feed-forward neural network (FFN) is used to learn prediction masks in remote sensing image training data.

[0049] In this embodiment, in step S2, the loss function of the remote sensing target detection model is for:

[0050]

[0051] in, represents the classification loss, represents the predicted probability, Indicates the instance category, represents the detection box loss, represents the prediction box, represents the real frame, represents the mask loss, represents the prediction mask, represents the instance mask.

[0052] In this embodiment, the pre-trained basic diffusion model is Stable-Diffusion, and the parameters of the feature encoder of the pre-trained basic diffusion model remain unchanged.

[0053] For example, the basic diffusion model Stable-Diffusion (SD) is used to freeze all parameters of the original image encoder during the training of the visual feature extraction module. The input image can be uniformly scaled to 1024×1024, and data enhancement operations such as random cropping are performed during the training process to improve the generalization ability of the model.

[0054] For example, the feature encoder VAE encoder and UNet network in SD can be used to obtain the original representation of the pre-trained model. First, the RGB image can be used as input, the image features can be extracted through the feature encoder VAE encoder, and then the features can be further encoded through the UNet network. In this process, the parameters in the feature encoder VAE encoder remain unchanged, while the parameters in UNet are optimized during the training process, thereby achieving transfer learning of remote sensing image features.

[0055] For example, in order to extract features with semantic relevance and discriminative power while avoiding increasing computational complexity, a multi-scale feature extraction module in step S2 may be constructed. The multi-scale feature extraction module may utilize a lightweight convolutional network to learn representative semantic features from multiple intermediate feature layers of the Stable-Diffusion backbone.

[0056] Optionally, the multi-scale feature extraction module in step S2 may also generate a multi-scale pyramid feature map to facilitate the execution of subsequent tasks.

[0057] Exemplarily, the instance-level object detection module in step S2 can be designed and constructed using the Decoder module of Mask2former (i.e., image segmentation model). The instance-level object detection module can be composed of 9 stacked sub-modules, each of which includes Cross-Attention, Self-Attention, and Feedforward Neural Network FFN. These sub-modules are responsible for predicting the category of the target, the prediction box, and the corresponding segmentation results, respectively, and effectively distinguishing between targets through learnable queries.

[0058] For example, the tokens in the query learn different target instances, and learn the predicted probability p, predicted box b, and predicted mask m corresponding to the target through three branches respectively.

[0059] like Figure 2 As shown, Figure 2 The figure is a schematic diagram of the model structure of a remote sensing target detection method based on a diffusion model. Figure 2In , the instance-level object detection module can include three branches, namely, classification branch, regression branch and mask branch. The loss function of the classification branch is , the loss function of the regression branch is , the loss function of the mask branch is .

[0060] Continue to refer Figure 2 , the learnable query realizes instance-level differentiation, and interacts with the multi-scale features (i.e., multi-level feature maps) extracted by the multi-scale feature extraction module through the mask attention module, self-attention module, and feedforward neural network in the instance-level object detection module to learn instance-level latent features. Subsequently, the latent features are respectively learned through the classification branch to obtain the prediction probability p, through the regression branch to obtain the prediction box b, and through the mask branch to obtain the prediction mask m.

[0061] The classification branch contains a linear layer that maps the vector dimension 256 to categories.

[0062] The regression branch is a superposition of 2 linear layers and ReLu layers. The last linear layer maps the vector dimension 256 to 4 dimensions to learn the upper left and lower right corners of the detection box respectively.

[0063] The mask branch is a superposition of three linear layers and ReLu layers. The output of the last linear layer and the last layer of multi-scale features are matrix-multiplied to obtain the mask m.

[0064] S3. Input the target remote sensing image data into the remote sensing target detection model, and output the detection result of the target remote sensing image through the remote sensing target detection model.

[0065] In this embodiment, the Adam W optimizer can also be used to optimize the network parameters of the model. The learning rate can be set to 0.0001, and the batch size (ie, batch size) can be selected to be 8.

[0066] Exemplarily, a public remote sensing instance segmentation dataset can be used to verify the remote sensing target detection model.

[0067] For example, the publicly available remote sensing instance segmentation dataset may be the NWPU VHR-10 dataset. The NWPU VHR-10 dataset covers ten categories: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, track and field fields, ports, bridges, and vehicles. The NWPU VHR-10 dataset can be widely used for remote sensing image target detection tasks, and 70% of the NWPU VHR-10 dataset is used for training and 30% is used for testing. As shown in Table 1, Table 1 is a comparison result between the remote sensing target detection model and other detection models.

[0068] Table 1 Comparison results between remote sensing target detection model and other detection models

[0069]

[0070] It can be concluded from Table 1 that the remote sensing target detection model in the present invention can fully utilize the advantages of the basic diffusion model and achieve a better final accuracy rate when only a small number of target detection module parameters are trained.

[0071] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A remote sensing target detection method based on a diffusion model, characterized in that: The method comprises: S1. Obtain remote sensing image training data; S2. Using the remote sensing image training data to train a neural network to obtain a remote sensing target detection model; The remote sensing target detection model includes a visual feature extraction module, a multi-scale feature extraction module and an instance-level target detection module; The visual feature extraction module is a pre-trained basic diffusion model, which is used to extract image features of the remote sensing image training data to obtain an initial feature map of the remote sensing image training data; The multi-scale feature extraction module is used to extract features from the initial feature map to obtain a multi-level feature map; The instance-level target detection module is used to perform target detection on the multi-level feature map and output the detection result; the instance-level target detection module includes 9 layers of stacked sub-modules, and the sub-modules include Cross-Attention, Self-Attention and Feedforward Neural Network FFN; The instance-level object detection module realizes instance-level distinction through learnable query, specifically: the learnable query interacts with the multi-level feature map extracted by the multi-scale feature extraction module through the cross-attention, self-attention and feed-forward neural network FFN in the instance-level object detection module to learn instance-level latent features; the latent features are respectively learned into prediction probability p through the classification branch, prediction box b through the regression branch, and prediction mask m through the mask branch; The classification branch comprises a linear layer; The regression branch is a superposition of two linear layers and a ReLu layer; The mask branch is a superposition of 3 linear layers and ReLu layers; The pre-trained basic diffusion model is Stable-Diffusion, and the parameters of the feature encoder of the pre-trained basic diffusion model remain unchanged; The pre-trained basic diffusion model includes a feature encoder and a UNet network; Wherein, the feature encoder is used to extract image features of the remote sensing image training data; The UNet network is used to further encode the image features of the remote sensing image training data to obtain the initial feature map; S3. Input the target remote sensing image data into the remote sensing target detection model, and output the detection result of the target remote sensing image through the remote sensing target detection model.

2. The method according to claim 1, characterized in that In step S2, the dimensions of the multi-level feature maps are the same, and the dimensions of the multi-level feature maps are the dimensions of the vectors representing the pixels in the multi-level feature maps.

3. The method according to claim 1, characterized in that The cross-attention is used to learn the category of the remote sensing image training data; The self-attention is used to learn the prediction box in the remote sensing image training data; The feed-forward neural network FFN is used to learn the prediction mask in the remote sensing image training data.

4. The method according to claim 1, characterized in that: In step S2, the loss function of the remote sensing target detection model is for: in, represents the classification loss, represents the predicted probability, Indicates the instance category, represents the detection box loss, represents the prediction box, represents the real frame, represents the mask loss, represents the prediction mask, represents the instance mask.

Images