Diffusion model training method, remote sensing image generation method, electronic device, and computer-readable storage medium

Abstract

The application provides a diffusion model training method, including: obtaining a remote sensing sample set, generating corresponding content text based on sample images; using a SpDINO model as an encoder to encode the sample images into a latent space to obtain sample features, wherein the SpDINO model is obtained by using DINOv2, SAM2 and CLIP to perform multi-teacher distillation training on ViT; performing noise processing on the sample features to obtain noisy features; using a diffusion model based on a Transformer architecture to perform diffusion prediction on the noisy features based on the content text and a denoising time step to obtain predicted features, and obtaining corresponding predicted features for the noisy features of each denoising time step; calculating a diffusion model loss value based on the noisy features and the predicted features, and adjusting the parameters of the diffusion model according to the diffusion model loss value to obtain a trained diffusion model. The application uses the SpDINO model obtained by using multi-teacher distillation training as an encoder to train the diffusion model, so that the diffusion model can generate images with perfect features and high clarity.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to diffusion model training methods, remote sensing image generation methods, electronic devices, and computer-readable storage media. Background Technology

[0002] Remote sensing images typically refer to images obtained by remote sensing satellites or drone platforms. In practice, due to factors such as weather and lighting conditions, and sensor performance, it is difficult to continuously, stably, and with high quality acquire remote sensing images of a target area.

[0003] In recent years, with the development of artificial intelligence technology, image generation models have provided a new approach to acquiring remote sensing image data. These models learn the distribution patterns of sample data and generate new remote sensing images based on conditions, providing high-quality samples for subsequent tasks such as image interpretation and classification. However, existing image generation models cannot simultaneously meet the multi-dimensional requirements of remote sensing images in terms of semantics, spatiality, and texture, resulting in low-quality generated images. Furthermore, the diffusion models used in existing image generation models suffer from training instability. Summary of the Invention

[0004] This invention provides a diffusion model training method, a remote sensing image generation method, an electronic device, and a computer-readable storage medium, aiming to solve the above-mentioned problems.

[0005] According to the first aspect, a diffusion model training method is provided, the method comprising:

[0006] S1: Obtain a remote sensing sample set, which includes multiple sample images, and generate corresponding content text based on the sample images;

[0007] S2: The sample image is encoded into the latent space using the SpDINO model as an encoder to obtain sample features. The SpDINO model is obtained by multi-teacher distillation training of ViT using DINOv2, SAM2 and CLIP.

[0008] S3: Use a noise-adding model to add noise to the sample features to obtain noisy features;

[0009] S4: Construct a diffusion model based on the Transformer architecture, and use the diffusion model to perform diffusion prediction on the noisy features based on the content text to obtain the predicted features;

[0010] S5: Calculate the diffusion model loss value based on the noisy features and the predicted features, and adjust the parameters of the diffusion model according to the diffusion model loss value to obtain the trained diffusion model.

[0011] According to the second aspect, a remote sensing image generation method is provided, the method comprising:

[0012] Step 1: Obtain the conditional text describing the remote sensing image;

[0013] Step 2: Generate a conditional text representation based on the conditional text, and generate a denoised time step representation based on the denoised time step;

[0014] Step 3: Input the noise, the conditional text representation, and the denoising time step representation into the diffusion model to obtain the image feature representation; wherein, the diffusion model is trained using the diffusion model training method described above;

[0015] Step 4: Input the image features into the pre-trained decoder to obtain the remote sensing image.

[0016] According to a third aspect, an electronic device is provided, the electronic device comprising:

[0017] The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described diffusion model training method.

[0018] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described diffusion model training method.

[0019] The beneficial effects of the present invention include at least the following:

[0020] 1. This invention uses the SpDINO model obtained by multi-teacher distillation training of DINOV2, SAM2 and CLIP as an encoder to train the diffusion model. It can simultaneously train the visual representation ability, image contour segmentation ability and image-text alignment ability of the diffusion model. It can also give full play to the advantages of DINOV2's high information content and concentrated distribution, so that the trained diffusion model can generate images with complete features and high clarity based on conditional text.

[0021] 2. This invention converts noisy features and predicted features into corresponding flow velocities, and obtains the loss value of the diffusion model by calculating the loss value between the target flow velocity and the predicted flow velocity. This makes the training of the diffusion model more stable, converges faster, and has a more uniform gradient. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the training method of the diffusion model of the present invention;

[0024] Figure 2 This is a schematic diagram of the overall network structure for training the diffusion model according to the present invention;

[0025] Figure 3 This is a schematic flowchart of the remote sensing image generation method of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0027] Figure 1 This is a flowchart illustrating a training method for a diffusion model provided by the present invention. Figure 1 As shown, a training method for a diffusion model includes:

[0028] S1: Obtain a remote sensing sample set, which includes multiple sample images, and generate corresponding content text based on the sample images.

[0029] Specifically, a remote sensing image dataset is acquired and filtered to obtain a remote sensing sample set. A remote sensing image dataset refers to image data obtained from remote sensing satellites or captured by unmanned aerial vehicle (UAV) platforms. The dataset includes multiple remote sensing images. These images are filtered according to selection criteria to choose those that meet the target criteria, resulting in sample images. The remote sensing sample set is then constructed from these sample images. Selection criteria include, but are not limited to, high image resolution, clear image boundaries, and the inclusion of the target land cover type. The target land cover type refers to the category of the target land feature; for example, the selection may include uncommon land cover types such as greenhouse areas and residential areas.

[0030] The BLIP text extraction model is used to generate corresponding descriptive text for each sample image. Therefore, each sample image has a matching descriptive text.

[0031] S2: The sample image is encoded into the latent space using the SpDINO model as an encoder to obtain sample features. The SpDINO model is obtained by multi-teacher distillation training of ViT using DINOv2, SAM2 and CLIP.

[0032] Specifically, using DINOv2, SAM2, and CLIP as multi-teacher models and ViT as the student model, the trained student model is obtained through joint distillation training of the multi-teacher models. This is a new DINOv2 model, called the Super DINO (SpDINO) model. The main framework of the SpDINO model is still the ViT network, which includes multiple Transformer blocks. The SpDINO model can simultaneously possess the structure awareness of DINOv2, the semantic alignment of CLIP, and the spatial layout capabilities of SAM2.

[0033] The distillation training framework comprises a teacher model, student models, and adapter heads. The teacher models, employing pre-trained CLIP, DINOv2, and SAM2 models, extract features from the input image. The student models, using ViT, perform multi-scale feature extraction on the input image, outputting general visual features. Three adapter heads follow the student models. Each adapter head uses a two-layer Multilayer Perceptron (MLP) with a LayerNorm and Gaussian Error Linear Unit (GELU) added in between to align the output dimension of the student models with the dimensions projected from each teacher model. The input dimension of each adapter head is the embedding dimension of the student models, the intermediate dimension is the largest embedding dimension among all teacher models, and the output dimensions are matched to specific teacher models (CLIP, DINOv2, and SAM2).

[0034] Distillation training utilizes image data from datasets such as ImageNet and DataComp-1B, requiring no data labels. An adapter head is designed for each teacher, distilling both global and spatial features separately. The distillation training process involves simultaneously feeding the input image into both the student model and each teacher model. The general visual features output by the student model are transformed through different adapter heads, and then compared with the features of the corresponding teacher to calculate the multi-teacher distillation loss. Backpropagation is used to update the student model parameters, resulting in the trained student model, SpDINO. By distilling CLIP, the student model inherits strong image-text alignment capabilities. By distilling the spatial features of DINOv2, the student model obtains high-quality general visual representations. By distilling the image segmentation capabilities of SAM, the student model possesses pixel-level understanding and the ability to segment novel objects.

[0035] Figure 2 A schematic diagram of the overall network structure for training the diffusion model is shown. Figure 2As shown, the SpDINO model is used as an encoder to train the DiT diffusion model. The SpDINO model is used to encode the sample images into the latent space to obtain the sample features in the latent space.

[0036] Specifically, the sample image I is divided into blocks and input into the SpDINO model. Multiple Transformer blocks of the ViT network encode the sample image I into the latent space, outputting the sample features f in the latent space. , c represents the number of channels in the latent space. The sample feature f integrates land cover visual features, spatial contour features, and semantic features.

[0037] By using the SpDINO model, which is obtained through multi-teacher distillation training with DINOV2, SAM2, and CLIP, as the encoder to train the diffusion model, we can simultaneously train the diffusion model's visual representation ability, image contour segmentation ability, and image-text alignment ability. Furthermore, we can fully leverage the advantages of DINOV2's high information content and concentrated distribution, enabling the trained diffusion model to generate images with complete features and high clarity based on conditional text.

[0038] S3: Use a noise-adding model to add noise to the sample features to obtain noisy features.

[0039] See also Figure 2 The sample feature f is then subjected to noise processing using a noise-adding model to obtain a noisy feature x. Specifically, the noise-adding model uses a linear interpolation function of the sample feature f and the noise z to represent the noisy feature. The linear interpolation function is as follows:

[0040] (1)

[0041] in, For time steps, ∈[0,1].

[0042] Specifically, the noisy features are first represented using a general interpolation function. ,Right now:

[0043] (2)

[0044] in, and In time step Predefined noise scheduling coefficients.

[0045] When the noise scheduling is a linear scheduling, it satisfies , Then the noisy features It can be further expressed as the linear interpolation function described above.

[0046] During the noise addition process, noise Conforms to noise distribution Noise distribution Typically follows a standard Gaussian distribution; time step i follows a continuous uniform distribution between 0 and 1. , i∈[0,1]. During each noise addition process, from the noise distribution... A noise z is randomly sampled from a uniform distribution at time step. A noisy feature is obtained by randomly sampling a time step i.

[0047] Understandably, ideally, at time step for Noise addition has not yet started; the noisy feature is the sample feature f; at time step... for At that location, the noise feature is noise z.

[0048] S4: Construct a diffusion model based on the Transformer architecture, and use the diffusion model to perform diffusion prediction on the noisy features based on the content text to obtain the predicted features.

[0049] Specifically, a diffusion model (DiT) based on the Transformer architecture is constructed. The DiT model includes an input layer, a linear embedding layer, multiple Transformer blocks, and a linear prediction layer. Each Transformer block includes a normalization layer, a self-attention layer, and a feedforward network layer. The linear embedding layer further includes a temporal embedding layer and a text embedding layer.

[0050] The predicted features are obtained by back-diffusion of noisy features based on the content text using a diffusion model, specifically including:

[0051] Use the BLIP model to extract content text. Encoding as content text representation ;

[0052] The time step representation is obtained by encoding time step i using a time step encoding model. Where i∈[0,1], the time step coding model is, for example, to encode time step i using a sine function;

[0053] Represent the content text Time step representation The noisy features are input into the diffusion model, which uses temporal embedding layers and text embedding layers to represent the content text. Time step representation The input of the input to the noisy feature x is added to the input to the Transformer block, and then the self-attention mechanism is used for back-diffusion to output the predicted feature y.

[0054] S5: Calculate the diffusion model loss value based on the noisy features and the predicted features, and adjust the parameters of the diffusion model according to the diffusion model loss value to obtain the trained diffusion model.

[0055] Specifically, based on the noise characteristics The predicted features are used to calculate the diffusion model loss value, and the diffusion model parameters are adjusted according to the denoising loss value. This process is repeated until the diffusion model converges, resulting in a trained diffusion model.

[0056] Among them, based on the noise characteristics Predictive features The diffusion model loss value is calculated by obtaining the target flow velocity from the flow velocity of the noisy features and the predicted flow velocity from the flow velocity of the predicted features; the mean square error loss between the target flow velocity and the predicted flow velocity is then calculated to obtain the diffusion model loss value. Specific steps include:

[0057] (1) Calculate the noise features Target flow rate The formula for calculating the target flow velocity is:

[0058] (3)

[0059] And calculate the predicted features Predicted flow rate The formula for predicting flow velocity is:

[0060] (4)

[0061] Where f represents the sample features and i represents the time step. Noisy features Let y be the target flow velocity and y be the predicted feature. To predict flow rate.

[0062] It should be understood that during the diffusion prediction process of the diffusion model, in time step i∈[0,1], ideally, the noisy feature x is the sample feature f when time step i is 1. Then, the speed at which the noisy feature x transforms into the sample feature f can be defined as the target flow velocity of the noisy feature x, and its formula can be expressed as the above formula (3). Similarly, ideally, the predicted feature y can become the sample feature f when time step i is 1. Then, the speed at which the predicted feature y transforms into the sample feature f can be defined as the predicted flow velocity of the noisy feature y, and its formula can be expressed as the above formula (4).

[0063] (2) Calculate the target flow velocity Predicted flow velocity The model loss value is obtained by taking the mean squared error loss between the two values, as shown in the following formula:

[0064] (4)

[0065] in, For the target flow rate, To predict flow rate.

[0066] Gradient propagation is initiated, and the parameters of the diffusion model are adjusted based on the model loss value. When the diffusion model converges, a trained diffusion model is obtained. The conditions for model convergence include, but are not limited to, the diffusion model loss value reaching a preset value or the number of training iterations reaching a preset number.

[0067] This invention converts noisy features and predicted features into corresponding flow velocities, and obtains the loss value of the diffusion model by calculating the loss value between the target flow velocity and the predicted flow velocity. This makes the training of the diffusion model more stable, converges faster, and has a more uniform gradient.

[0068] Figure 3 This is a schematic flowchart of the remote sensing image generation method provided by the present invention. Figure 3 As shown, a remote sensing image generation method includes:

[0069] Step 1: Obtain the conditional text describing the remote sensing image.

[0070] Specifically, the conditional text is obtained. The conditional text describes the information the user expects to generate in the remote sensing image. The user can input the conditional text through a text input interface or by inputting conditional speech through a voice input interface, which is then converted into text to obtain the conditional text.

[0071] Step 2: Generate a conditional text representation based on the conditional text, and generate a denoised time step representation based on the denoised time step.

[0072] Specifically, a BLIP encoder is used to generate a conditional text representation based on the conditional text. Simultaneously, a time-step encoding model is used to generate a denoised time-step representation based on the denoised time step t. .

[0073] Step 3: Input the noise, the conditional text representation, and the denoising time step representation into the diffusion model to obtain the image feature representation; wherein the diffusion model is trained using the diffusion model training method as described in claim 1.

[0074] Specifically, noise Conditional text representation Representation of denoising time steps DiT is used to perform diffusion denoising processing on the input diffusion model to generate image feature representations. The diffusion model DiT is trained using the above diffusion model training method, with the denoising time step t∈[0,1].

[0075] Specifically, during the inference phase, the diffusion model employs a discrete iterative denoising method to denoise the noise. Discrete iterative denoising refers to dividing the denoising time step interval... The process is divided into N steps for segmented iteration, with a step size h = 1 / N. The result of the last step is used as the final denoising result, i.e., as the image feature representation. Output the number of steps N, which is a positive integer. N can be set as needed, for example, 10.

[0076] The process of discrete iterative denoising can be summarized as follows:

[0077] Set noise reduction steps The denoising rate of step j is calculated by using the denoising result of step j as the denoising prediction value when the final denoising is completed. Based on the denoising rate of step j and the step size, the denoising value of step j is calculated. Based on the input noise, all denoising values ​​of the previous j steps and the step size, the input value of step j+1 is determined. The input value of step j+1 is then input into the diffusion model for denoising to obtain the denoising result of step j+1. This process is repeated iteratively to obtain the actual final denoising result, where the input value of step j is noise.

[0078] The following example, with N=10, illustrates the detailed steps of discrete iterative denoising:

[0079] (1) In discrete iterative denoising, each iteration is represented by the iteration number j, and the time step is a piecewise time step. The input value to DiT at each step is The denoising result obtained is The relationship between the piecewise time step and the number of iterations is as follows:

[0080]

[0081] in, , .

[0082] (2) First iteration:

[0083] exist , At that time, input to DiT After DiT denoising, the first step denoising result is obtained. and the denoising results The denoising prediction value at the final denoising completion is used to obtain the denoising rate of the first step. for:

[0084]

[0085]

[0086] It should be understood that the final denoising completion refers to the denoising time step reaching 1.

[0087] Next, based on the denoising rate of the first step... Step 2: Determine the step size , Input value at time :

[0088]

[0089]

[0090] (2) Second iteration:

[0091] exist , At that time, input to DiT The second step of denoising results is obtained. and the denoising results The denoising prediction value at the end of the final denoising process is used to obtain the denoising rate in the second step. for:

[0092]

[0093]

[0094] Next, based on the input values ​​from the second step... The noise reduction rate in the second step The third step is to determine the step size. , Input value at time :

[0095]

[0096]

[0097] This process continues, determining the denoising rate based on the previous denoising result, and then determining the next step's input value based on the previous input value, denoising rate, and step size. The next step's input value is then input into DiT for denoising to obtain the next step's denoising result. This process is iterated continuously to obtain the denoising result for each step. .

[0098] The final noise reduction result The final denoising result of the inference stage is used as the image feature vector. Output.

[0099] Step 4: Input the image features into the pre-trained decoder to obtain the remote sensing image.

[0100] In one embodiment of the present invention, the decoder employs a Transformer network. The image feature representation is input into the pre-trained decoder constructed based on the Transformer network to generate the remote sensing image.

[0101] According to one embodiment of the present invention, in the pre-training of the decoder, the SpDINO model is used as the encoder to encode the pre-training sample images into the latent space, and the decoder is used to decode them to obtain the predicted sample images.

[0102] Specifically, the pre-training of the decoder includes:

[0103] The pre-trained sample images are input into the SpDINO model, and the SpDINO model encodes them into the latent space to obtain the second sample features;

[0104] The second sample features are input into the decoder to obtain the predicted sample image;

[0105] Calculate the loss value between the pre-trained sample images and the predicted sample images, and adjust the decoder parameters based on the loss value to obtain the pre-trained decoder.

[0106] This invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described in any of the foregoing method embodiments. Further, the electronic device further includes: at least one input device and at least one output device. The memory, processor, input device, and output device are connected via a bus. Specifically, the input device may be a camera, touch panel, physical button, or mouse, etc. The output device may specifically be a display screen. The memory may be high-speed random access memory (RAM) or non-volatile memory, such as disk storage. The memory is used to store a set of executable program code, and the processor is coupled to the memory.

[0107] This invention also provides a computer-readable storage medium, which may be disposed in the electronic device described in the foregoing embodiments. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of the foregoing method embodiments. Further, the computer-readable storage medium may also be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0108] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a diffusion model, characterized in that, The method includes: S1: Obtain a remote sensing sample set, which includes multiple sample images, and generate corresponding content text based on the sample images; S2: The sample image is encoded into the latent space using the SpDINO model as an encoder to obtain sample features. The SpDINO model is obtained by multi-teacher distillation training of ViT using DINOv2, SAM2 and CLIP. S3: Use a noise-adding model to add noise to the sample features to obtain noisy features; S4: Construct a diffusion model based on the Transformer architecture, and use the diffusion model to perform diffusion prediction on the noisy features based on the content text to obtain the predicted features; S5: Calculate the diffusion model loss value based on the noisy features and the predicted features, and adjust the parameters of the diffusion model according to the diffusion model loss value to obtain the trained diffusion model.

2. The method according to claim 1, characterized in that, S3 includes: The noisy model uses a linear interpolation function of the sample features and noise to represent the noisy features, and the linear interpolation function is as follows: in, For sample features, For noise, For time steps, , This is a noisy feature.

3. The method according to claim 2, characterized in that, The calculation of the diffusion model loss value based on the noisy features and the predicted features includes: The flow velocity of the noisy feature is calculated based on the noisy feature to obtain the target flow velocity, and the flow velocity of the predicted feature is calculated based on the predicted feature to obtain the predicted flow velocity. The mean square error loss between the target flow velocity and the predicted flow velocity is calculated to obtain the diffusion model loss value.

4. The method according to claim 3, characterized in that, The formula for calculating the target flow velocity is: The formula for calculating the predicted flow velocity is: in, For sample features, For time steps, Noisy features Let y be the target flow velocity and y be the predicted feature. To predict flow rate, .

5. The method according to claim 1, characterized in that, The diffusion model based on the Transformer architecture includes an input layer, a linear embedding layer, multiple Transformer blocks, and a linear prediction layer.

6. A method for generating remote sensing images, characterized in that, The method includes: Step 1: Obtain the conditional text describing the remote sensing image; Step 2: Generate a conditional text representation based on the conditional text, and generate a denoised time step representation based on the denoised time step; Step 3: Input the noise, the conditional text representation, and the denoising time step representation into the diffusion model to obtain the image feature representation; wherein, the diffusion model is trained using the diffusion model training method as described in claim 1; Step 4: Input the image features into the pre-trained decoder to obtain the remote sensing image.

7. The method according to claim 6, characterized in that, The decoder uses a Transformer network.

8. The method according to claim 7, characterized in that, In the pre-training of the decoder, the SpDINO model is used as the encoder to encode the pre-trained sample images into the latent space, and the decoder is used to decode them to obtain the predicted sample images.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images