Remote sensing image generation method and device based on multi-condition controllable diffusion model

The remote sensing image generation method based on a multi-condition controllable diffusion model solves the problem of insufficient single control conditions in remote sensing image generation technology, and realizes high-quality and flexible remote sensing image generation to meet different application needs.

CN118982597APending Publication Date: 2024-11-19XI AN JIAOTONG UNIV
View PDF 0 Cites 10 Cited by

Patent Information

Application Number
CN202411107175.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing remote sensing image generation technologies mainly rely on single control conditions such as text or semantic layout, which makes it difficult to fully express the complex information and structural features of remote sensing images, resulting in deficiencies in the reproduction of details and quality control of the generated images.

Method used

A method based on a multi-condition controllable diffusion model is adopted. Remote sensing images are collected and preprocessed to identify surface characteristic information and construct a dataset. A basic diffusion model is built and a multi-condition control network is introduced. The diffusion model is trained using the feature condition dataset. The model performance is optimized by combining a joint training strategy and evaluation indicators to generate high-quality remote sensing images.

Benefits of technology

It improves the quality and usability of remote sensing images, accurately extracts surface characteristic information, enhances the accuracy and processing efficiency of information extraction, strengthens the stability and flexibility of the generative model, and can generate diverse remote sensing images that meet specific needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982597A_ABST
    Figure CN118982597A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image generation method and device based on a multi-condition controllable diffusion model. Collection, preprocessing, feature recognition and data set construction of remote sensing images are carried out. And constructing and training a basic diffusion model, and introducing a multi-condition control network to generate a final diffusion model. A controlled diffusion model is obtained by constructing a remote sensing data set comprising a geographic feature condition and a multi-modal condition and adopting an initial training and joint training strategy. The final diffusion model and the controlled diffusion model are fused into an initial multi-condition controllable model, and conversion from condition information input to remote sensing image output is achieved. And comparing a final remote sensing image of the initial multi-condition controllable diffusion model with a real image by using an evaluation index, iteratively optimizing the model according to an evaluation result, and generating an efficient multi-condition controllable diffusion model, thereby solving the problem that complex information and structural features of the remote sensing image are difficult to comprehensively express in the prior art and improving the accuracy of the remote sensing image. And therefore, the problem that the generated image is insufficient in detail reproduction and quality control can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a remote sensing image generation method and device based on a multi-condition controllable diffusion model. Background Art

[0002] As a special type of image, remote sensing images have the characteristics of high resolution, large area coverage, and rich geographic information. They are widely used in many fields such as agricultural monitoring, urban planning, and environmental protection.

[0003] Remote sensing image generation technology aims to make up for the shortcomings of actual remote sensing image data acquisition and provide rich image resources for related research and applications.

[0004] However, current remote sensing image generation technology mainly relies on single control conditions such as text or semantic layout for image generation. However, this single control strategy is difficult to fully express the complex information and structural characteristics of remote sensing images, resulting in deficiencies in detail reproduction and quality control of the generated images. Summary of the invention

[0005] In an embodiment of the present application, a remote sensing image generation method based on a multi-condition controllable diffusion model is provided to solve the problem that the current remote sensing image generation technology mainly relies on a single control condition such as text or semantic layout for image generation, which makes it difficult to fully express the complex information and structural characteristics of the remote sensing image, resulting in deficiencies in detail reproduction and quality control of the generated image.

[0006] In a first aspect, an embodiment of the present application provides a remote sensing image generation method based on a multi-condition controllable diffusion model, the method comprising: collecting remote sensing images, preprocessing the remote sensing images, identifying surface characteristic information in the remote sensing images and constructing a data set; constructing a basic diffusion model suitable for remote sensing image processing, and using the data set to train the basic diffusion model; constructing a multi-condition control network in the basic diffusion model, and using the characteristic condition data set to train the multi-condition control network to generate a final diffusion model; constructing a generation diffusion model, and constructing a remote sensing data set including condition information; wherein the condition information includes geographic feature conditions and multimodal conditions; using The remote sensing data set performs initial training on the generated diffusion model, and uses a joint training strategy to perform final training on the generated diffusion model to obtain a controlled diffusion model; the final diffusion model and the controlled diffusion model are integrated to construct an initial multi-condition controllable diffusion model, the initial multi-condition controllable diffusion model receives input of condition information, and outputs a final remote sensing image; the output final remote sensing image is compared with a real image using an evaluation index, the performance of the initial multi-condition controllable diffusion model is evaluated, and an evaluation result is obtained; the initial multi-condition controllable diffusion model is iteratively trained and optimized according to the evaluation result to generate a final multi-condition controllable diffusion model.

[0007] In a possible implementation, the preprocessing of the remote sensing image includes: using a trained deep learning denoising network to remove noise in the remote sensing image; correcting the geometric distortion in the remote sensing image to restore the true shape and positional relationship of the remote sensing image; adjusting the brightness and / or contrast of the remote sensing image to make remote sensing images from different sources visually comparable; and increasing the amount and diversity of data in the remote sensing image through data augmentation operations.

[0008] In a possible implementation, the identifying of surface characteristic information in remote sensing images and constructing a data set include: standardizing the preprocessed remote sensing images, extracting feature maps representing high-dimensional features in the remote sensing images using a pretrained feature extraction network; processing the feature maps using a semantic segmentation network, identifying and segmenting different semantic regions in the feature maps; matching the feature maps with predefined conditional templates through feature matching technology to identify surface characteristic information in the remote sensing images; and integrating the preprocessed remote sensing images and the identified surface characteristic information to construct a data set.

[0009] In a possible implementation, the basic diffusion model suitable for remote sensing image processing is constructed and the basic diffusion model is trained using a data set, including: converting the remote sensing image into its variable representation in the corresponding latent space through a variational autoencoder; minimizing the reconstruction error loss function of the variational autoencoder to learn the distribution of feature representations formed after the remote sensing image is mapped to the latent space through the variational autoencoder; creating a first joint condition guidance mechanism to integrate multimodal conditions into a unified representation; wherein the multimodal conditions include text description conditions, content image conditions, and metadata conditions; introducing a cross-attention mechanism in the denoising process to enhance mutual reference and adjustment between multimodal conditions; minimizing the error between predicted noise and actual added noise, introducing a classifier-free guidance mechanism to adjust the guidance weight, and controlling the degree of dependence of the basic diffusion model on the multimodal conditions.

[0010] In a possible implementation, the creation of the first joint conditional guidance mechanism includes: converting the text description into a word embedding vector through a text encoder of a comparative language-image pre-training model, and using the word embedding vector as a first encoding; converting the content image into a feature map through an image encoder, and using a feedforward network to process the feature map to obtain a second encoding; obtaining a third encoding after the metadata passes through a linear layer; and fusing the first encoding, the second encoding, and the third encoding to create a unified representation, thereby creating the first joint conditional guidance mechanism.

[0011] In a possible implementation, a multi-condition control network is constructed in the basic diffusion model, and a feature condition data set is used to train the multi-condition control network to generate a final diffusion model, including: decoupling the geographic feature conditions in the remote sensing image into a variety of feature condition data, and constructing a feature condition data set; wherein the geographic feature conditions include a segmentation mask and a road map; and using a second joint condition guidance mechanism to fuse the multiple feature condition data to control the image generation process in the basic diffusion model.

[0012] In a possible implementation, a second joint condition guidance mechanism is used to fuse multiple feature condition data, including: using a feature extraction network to extract key feature maps from the feature condition data; using a feature fusion network to perform multi-scale fusion and feature weighting on the extracted key feature maps.

[0013] In one possible implementation, the use of the remote sensing dataset to initially train the generative diffusion model includes: during the initial training process, taking text encoding and variable representation of the remote sensing image in the corresponding latent space as input, and optimizing the generative diffusion model parameters to minimize the difference between the generated image and the real image.

[0014] In a possible implementation, the method of performing final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model includes: during the final training process, adjusting the value of the guide weight to control the degree of dependence of the generated diffusion model on the conditional information, thereby obtaining the controlled diffusion model.

[0015] In a second aspect, an embodiment of the present application provides a remote sensing image generation device based on a multi-condition controllable diffusion model, the device comprising: a collection module for collecting remote sensing images, preprocessing remote sensing images, identifying surface characteristic information in remote sensing images and constructing a data set; a first construction module for constructing a basic diffusion model suitable for remote sensing image processing, and training the basic diffusion model using a data set; a second construction module for constructing a multi-condition control network in the basic diffusion model, training the multi-condition control network using a feature condition data set, and generating a final diffusion model; a third construction module for constructing a generated diffusion model and constructing a remote sensing data set including condition information; wherein the condition information includes geographic feature conditions and multimodal conditions ; A training module is used to perform initial training on the generated diffusion model using the remote sensing data set, and to perform final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model; a fusion module is used to fuse the final diffusion model and the controlled diffusion model to construct an initial multi-condition controllable diffusion model, which receives input of conditional information and outputs a final remote sensing image; an evaluation module is used to compare the output final remote sensing image with a real image using an evaluation index, evaluate the performance of the initial multi-condition controllable diffusion model, and obtain an evaluation result; an optimization module is used to iteratively train and optimize the initial multi-condition controllable diffusion model according to the evaluation result to generate a final multi-condition controllable diffusion model.

[0016] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0017] The embodiment of the present application provides a remote sensing image generation method based on a multi-condition controllable diffusion model, which effectively improves the quality and usability of remote sensing images by using preprocessing technology, accurately extracts surface characteristic information from preprocessed remote sensing images, improves the accuracy of information extraction, reduces manual intervention, and improves processing efficiency. The constructed basic diffusion model realizes fine control of the remote sensing image generation process by introducing a multi-condition control network. Using a feature condition data set for training not only improves the generalization ability of the basic diffusion model, but also enables the basic diffusion model to generate a variety of remote sensing images according to different conditions, thereby generating a final diffusion model. In addition, the application of a joint training strategy further enhances the stability and performance of the generated diffusion model, ensuring the high quality and reliability of the generated image. The constructed initial multi-condition controllable diffusion model can receive a variety of conditional information inputs including geographic feature conditions and multimodal conditions, and generate remote sensing images that meet specific needs accordingly. This flexibility greatly expands the application scenarios of the model, enabling it to adapt to image processing tasks with different needs. By comparing evaluation indicators, the performance of the initial multi-condition controllable diffusion model is comprehensively evaluated. According to the evaluation results, the initial multi-condition controllable diffusion model is iteratively trained and optimized to continuously improve the quality and accuracy of the generated images. This process ensures the efficiency and reliability of the final multi-condition controllable diffusion model in practical applications. This solves the problem that the current remote sensing image generation technology mainly relies on a single control condition such as text or semantic layout for image generation, which makes it difficult to fully express the complex information and structural characteristics of remote sensing images, resulting in insufficient detail reproduction and quality control of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a remote sensing image generation method based on a multi-condition controllable diffusion model provided in an embodiment of the present application;

[0020] Figure 2 A specific flow chart for preprocessing remote sensing images provided in an embodiment of the present application;

[0021] Figure 3 A specific flow chart for identifying surface characteristic information in remote sensing images and constructing a data set provided in an embodiment of the present application;

[0022] Figure 4A specific flow chart for constructing a basic diffusion model suitable for remote sensing image processing and using a data set to train the basic diffusion model provided in an embodiment of the present application;

[0023] Figure 5 A specific flow chart of creating a first joint condition guidance mechanism provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of a process for constructing a multi-condition control network provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of using a joint training strategy to perform final training on a generated diffusion model to obtain a controlled diffusion model provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a remote sensing image generation device based on a multi-condition controllable diffusion model provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of a remote sensing image generation server based on a multi-condition controllable diffusion model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0030] The present application embodiment provides a remote sensing image generation method based on a multi-condition controllable diffusion model, such as Figure 1 As shown, the method includes steps S101 to S108. Among them, Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a remote sensing image generation method based on a multi-condition controllable diffusion model. In the case that the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0031] S101: Collect remote sensing images, pre-process the remote sensing images, identify surface feature information in the remote sensing images and construct a data set.

[0032] Specifically, remote sensing images can be acquired and collected through sensors on remote sensing platforms such as aircraft and satellites.

[0033] Figure 2 A specific flow chart for preprocessing remote sensing images provided in the embodiment of the present application is as follows: Figure 2 As shown, it includes steps S201 to S204.

[0034] S201: Use the trained deep learning denoising network to remove noise from remote sensing images.

[0035] Specifically, the deep learning denoising network in this application is constructed based on a convolutional neural network (CNN), which uses a large amount of training data to learn the complex relationship between noise and remote sensing image content. A large amount of training data is noisy images and their corresponding noise-free images. The network architecture of the deep learning denoising network includes multiple convolutional layers, activation functions, and pooling layers, which are used to extract different levels of features of remote sensing images layer by layer. In the training stage, the noisy remote sensing image is input into the deep learning denoising network, and a denoised output image is generated. Subsequently, this output image is compared with the original noise-free image, and the performance of the deep learning denoising network is evaluated by calculating the difference between the two, that is, the error. The back propagation algorithm is used to adjust and optimize the weight parameters in the deep learning denoising network, aiming to minimize the error and enhance the denoising ability of the deep learning denoising network. After the above training process, the deep learning denoising network can learn to identify and remove various types of noise, including but not limited to salt and pepper noise and Gaussian noise, while retaining key details and edge information in the remote sensing image. For RGB color remote sensing images, the deep learning denoising network can simultaneously process the three color channels of red (R), green (G) and blue (B), ensuring that while removing noise, the information between different color channels remains consistent and of high quality, thereby maintaining the overall visual effect of the remote sensing image.

[0036] S202: Correcting geometric distortion in the remote sensing image to restore the true shape and position relationship of the remote sensing image.

[0037] Specifically, remote sensing images will produce geometric distortion due to the combined effect of multiple factors during the collection process. These factors include shooting angle factors, sensor internal factors and earth factors, such as objective lens distortion and earth rotation. Remote sensing images can be corrected using mathematical methods such as geometric transformation and coordinate transformation. These transformations can include translation, rotation, scaling, affine transformation, etc. to eliminate geometric distortion in remote sensing images.

[0038] S203: Adjust the brightness and / or contrast of the remote sensing image to make the remote sensing images from different sources visually comparable.

[0039] Specifically, the brightness and / or contrast of the remote sensing images are adjusted so that remote sensing images from different sources have similar brightness and contrast visually and are thus comparable.

[0040] S204: Increasing the quantity and diversity of data in the remote sensing image through data augmentation operations.

[0041] Specifically, the data augmentation operations include random rotation, flipping and / or scaling operations, which can be used alone or in combination to increase the quantity and diversity of data in remote sensing images. By randomly applying these operations, the number of training samples can be significantly increased and more variation factors can be introduced.

[0042] Figure 3 The specific flow chart for identifying surface characteristic information in remote sensing images and constructing a data set provided in the embodiment of the present application is as follows: Figure 3 As shown, it includes steps S301 to S304.

[0043] S301: Standardize the preprocessed remote sensing image and use a pre-trained feature extraction network to extract a feature map representing high-dimensional features in the remote sensing image.

[0044] Specifically, the Z-score normalization method can be used to map the pixel values ​​of remote sensing images to a range of 0 to 1, which helps to ensure the consistency of data distribution between different remote sensing images, thereby improving the efficiency and stability of subsequent model training. The pre-trained feature extraction network can be a multi-layer convolutional neural network, which can automatically learn and extract high-dimensional feature representations in remote sensing images and present these features in the form of feature maps.

[0045] S302: Using a semantic segmentation network to process the feature map, identify and segment different semantic areas in the feature map.

[0046] Specifically, the semantic segmentation network is used to understand the complex scenes in remote sensing images and can identify and segment different semantic regions in the feature map. Specifically, it can distinguish and mark various geographic objects or natural elements such as roads, buildings, water bodies, vegetation, etc. in the image. This process assigns a category label to each pixel through pixel-level classification, thereby accurately outlining the boundaries of each semantic region.

[0047] S303: Match the feature map with the predefined condition template through feature matching technology to identify the surface characteristic information in the remote sensing image.

[0048] Specifically, the conditional template is constructed based on a deep understanding and prior knowledge of the surface characteristics, including key indicators such as land use type (such as farmland, forest, urban area, etc.) and vegetation coverage (such as high coverage, medium coverage, low coverage, etc.). Through feature matching technology, the feature map can be automatically compared with the various conditions in the template library to accurately identify the corresponding surface characteristics in the remote sensing image. This process not only improves the accuracy of information extraction, but also significantly enhances the degree of automation of data processing.

[0049] S304: Integrate the preprocessed remote sensing image and the identified surface characteristic information to construct a data set.

[0050] Specifically, the pre-processed remote sensing images are integrated with the surface characteristics information identified by feature matching technology. This integration process ensures a close correlation between the remote sensing image data and its corresponding surface characteristics (such as land use type, vegetation coverage, etc.). Subsequently, a comprehensive and accurate dataset is constructed using these integrated data. This dataset not only contains the original information of the remote sensing image, but also adds rich surface characteristics information.

[0051] S102: Construct a basic diffusion model suitable for remote sensing image processing and use the dataset to train the basic diffusion model.

[0052] Figure 4 The specific flow chart of constructing a basic diffusion model suitable for remote sensing image processing and using a data set to train the basic diffusion model is as follows: Figure 4 As shown, it includes steps S401 to S405.

[0053] S401: Convert the remote sensing image into its variable representation in the corresponding latent space through a variational autoencoder.

[0054] Specifically, the variational autoencoder used is a frozen variational autoencoder (VAE). Using the encoder and decoder parts of the variational autoencoder, the remote sensing image is converted into its variable representation in the corresponding latent space. The expression of this process is: x∈R C×H×W →z∈R C′×H′×W′. Where x is the remote sensing image, C is the number of channels of the remote sensing image, for RGB images, C = 3, H is the height of the remote sensing image, W is the width of the remote sensing image, z is the variable of the remote sensing image in the corresponding latent space, C′ is the number of channels of the remote sensing image in the corresponding latent space, H′ is the height of the remote sensing image in the corresponding latent space, and W′ is the width of the remote sensing image in the corresponding latent space. The goal of this step is not to directly learn the conditional data distribution p(x|τ) of the remote sensing image based on the text condition τ, but to focus on learning the feature distribution of the mapped image feature vector, that is, the feature distribution p(z|τ) of the remote sensing image in the corresponding latent space.

[0055] S402: Minimize the reconstruction error loss function of the variational autoencoder to learn the distribution of feature representations formed after the remote sensing image is mapped to the latent space through the variational autoencoder.

[0056] Specifically, the distribution of feature representations formed after the remote sensing image is mapped to the latent space through the variational autoencoder should not only be able to reconstruct the main features of the remote sensing image, but also capture the distribution patterns of different types of land objects (such as water bodies, vegetation, buildings, etc.) in the latent space.

[0057] Minimizing the reconstruction error loss function of the variational autoencoder includes the reconstruction loss and the KL divergence term. The expression of the reconstruction error loss function is: L VAE =E z~Q(z∣x) [log P(x|z)-D KL (Q(z|x)|P(z))], where L VAE is the reconstruction error loss function of the variational autoencoder, x is the remote sensing image, z is the variable of the remote sensing image in the corresponding latent space, i.e., the latent variable, E z~Q(z∣x) is the expectation operator, which means averaging all possible values ​​of the latent variable z under its conditional distribution Q(z|x). Q(z|x) is the approximate posterior distribution of the latent variable z generated by the variational autoencoder based on the remote sensing image x. log P(x|z) is the log-likelihood function, which represents the logarithm of the probability that the decoder generates the remote sensing image x given the latent variable z. P(x|z) is the generated distribution of the decoder, which can capture how to map from the latent space back to the original data space. D KL (Q(z|x)|P(z)) is the Kullback-Leibler (KL) divergence, which is used to quantify the difference between two probability distributions. It measures the difference between the encoder's approximate posterior distribution Q(z|x) of the latent variable and the prior distribution P(z). In the variational autoencoder, the encoder maps the remote sensing image to the latent space, and the decoder reconstructs the image from the latent space.

[0058] S403: Create a first joint condition guidance mechanism to integrate multimodal conditions into a unified representation, wherein the multimodal conditions include text description conditions, content image conditions, and metadata conditions.

[0059] Figure 5 A specific flow chart of creating a first joint condition guidance mechanism provided in an embodiment of the present application is as follows: Figure 5 As shown, it includes steps S501 to S504.

[0060] S501: Convert the text description into a word embedding vector by comparing the text encoder of the language-image pre-training model, and use the word embedding vector as the first encoding.

[0061] S502: Convert the content image into a feature map through an image encoder, and use a feedforward network to process the feature map to obtain a second code.

[0062] S503: The metadata passes through the linear layer to obtain a third encoding.

[0063] S504: The first encoding, the second encoding, and the third encoding are integrated to create a unified representation, thereby creating a first joint conditional guidance mechanism.

[0064] Specifically, compared to the language-image pre-training model for the CLIP model, the text encoder is an important component of the language-image pre-training model. After being trained on a large number of text-image pairs, it can learn the association between text and images and map the text description to a high-dimensional semantic space. Given a text description T, it is input into the text encoder of the language-image pre-training model. The text encoder processes the text description word by word and converts it into a series of word embeddings, which not only capture the semantic information of the words, but also take into account the contextual relationship of the words in the text. These word embeddings are further processed (such as by average pooling) to generate a fixed-length word embedding vector y t , embed the word into vector y t as the first code.

[0065] Furthermore, the contrastive language-image pre-training model has learned rich cross-modal association knowledge on a large text-image dataset through its unique pre-training mechanism, enabling accurate mapping and alignment of text descriptions and image features in a shared latent space. It allows users to precisely control the content and style of image generation through natural language instructions. Specifically, when a user inputs a detailed descriptive text such as "a dense forest in the Amazon rainforest, photographed in July 2023", the contrastive language-image pre-training model first uses a text encoder to convert the text into a high-dimensional semantic vector. This vector not only captures the key words and their meanings in the text, but also contains the overall semantic information of these words in context and attributes such as time and place in a specific situation.

[0066] Then, in the image generation stage, the CLIP model uses the complex associations between text and images learned during training and uses the above semantic vector as a guiding signal for the generative network. Based on this semantic vector, the generative network searches and constructs image features that match it in the latent space, thereby generating remote sensing images that meet the user's descriptive text requirements and have a high degree of realism and rich details.

[0067] Specifically, an image encoder is used to convert the content image C into a feature map. The feature map output from the image encoder usually has a high dimension. Therefore, before the feature map is input into the feedforward network, the feature map needs to be processed, such as global average pooling or global maximum pooling, to convert the feature map into a fixed-length feature vector. The feedforward network is also called a multi-layer perceptron (MLP) or a fully connected network (FCN). The feedforward network finally outputs a fixed-length encoding vector y c , the encoding vector y c As the second code.

[0068] Specifically, the metadata M contains a series of additional information about the dataset, such as labels, timestamps, etc. The metadata M is input into a linear layer, and after processing by the linear layer, the metadata M is converted into a fixed-length encoding vector y m , the encoding vector y m As the third code.

[0069] The expression of the first joint condition guidance mechanism is: y′=Concat(y t ,w c ·FFN(y c ),w m ·Encoder(y m )). Among them, y t is the first code, y c is the second code, y m For the third code, wc For adjusting FFN(y c )’s weight parameter, w m To adjust Encoder(y m ) is the weight parameter of the feature condition encoder. Encoder(y m ) indicates that the third encoding y is encoded using the feature conditional encoder m Encode. FFN is a feedforward network, FFN(y c ) indicates that the second encoding y is converted into c Mapped to a new feature space, the expression of FFN is: FFN(x) = W2·σ(W1·x+b1)+b2. Among them, W1 and W2 are weight matrices, b1 and b2 are bias terms, and σ is the activation function. Concat is the concatenation operation, and x is the input vector, which can be the original data.

[0070] S404: Introduce a cross-attention mechanism in the denoising process to enhance the cross-reference and adjustment between multimodal conditions.

[0071] Specifically, the expression of the cross attention mechanism is: Among them, Attention(Q,K,V) is a cross attention mechanism, QK T is the matrix multiplication between the query matrix Q and the transpose of the key matrix K, and its result reflects the similarity between the query and each key, d k is the dimension of the key, is the scaling factor, V is the value matrix, and softmax is used to convert Convert to attention weight matrix.

[0072] S405: Minimize the error between the predicted noise and the actual added noise, introduce a classifier-free guidance mechanism to adjust the guidance weight, and control the degree of dependence of the basic diffusion model on multimodal conditions.

[0073] Specifically, minimizing the error between the predicted noise and the actual added noise is taken as the training objective, and the expression of the training objective is: Among them, θ is the parameter of the basic diffusion model to be optimized, ∈ is the noise added in the forward process, and ∈ θ (z t ,c) represents the noise data z at a given time step t t and the prediction noise under condition c, L(θ) is the loss function, which measures the model prediction noise ∈ θ (z t ,t,c) and the difference between the actual noise ∈ added, E z,∈,t It represents the expectation operation on the joint distribution of z,∈,t, where z is the noisy data.

[0074] Furthermore, the expression of the classifier-free guidance mechanism (CFG) is: in, For the noise data z at time step t after the adjustment without classifier guidance, t and the prediction noise under the mixed condition c, ∈ θ (z t ) is the noise data z at only a given time step t t The prediction noise represents the generative capacity of the basic diffusion model under unconditional conditions, that is, the generative capacity that does not depend on any external conditions or guidance. ω is the guidance weight, which is used to control the balance between conditional generation and unconditional generation. A higher ω value will make the basic diffusion model more dependent on condition c for the multimodal condition, while a lower ω value will reduce the basic diffusion model's dependence on the multimodal condition. Among them, z t is the noise data at time step t, α t and σ t is a parameter related to time step t, which is used to control the amount of noise added. ∈ is the noise sampled from a certain noise distribution (such as the standard normal distribution). The classifier-free bootstrap mechanism flexibly controls the degree of dependence of the basic diffusion model on condition c during the generation process by adjusting the bootstrap weight ω. This mechanism enables the basic diffusion model to generate data that meets specific conditions and maintains a certain diversity as needed in different application scenarios. At the same time, by adjusting α t and σ t , it is also possible to control the speed at which noise is added and removed during the diffusion process, further affecting the quality and diversity of the generated data.

[0075] S103: construct a multi-condition control network in the basic diffusion model, train the multi-condition control network using the characteristic condition data set, and generate a final diffusion model.

[0076] Figure 6 A schematic diagram of the process of constructing a multi-condition control network provided in an embodiment of the present application, such as Figure 6 As shown in the figure, the encoder of the variational autoencoder (VAE) maps the diffusion process to the latent space, and the diffusion result is connected to the decoder to finally generate the output image. The noise image received at each step is the denoised image obtained in the previous step. The noise at the current time step is predicted through the UNet network to perform the denoising process. After the text conditions are encoded, they are uniformly injected through the attention layer to obtain a consistent image control effect.

[0077] A multi-condition control network is constructed in the basic diffusion model, and the multi-condition control network is trained using a feature condition data set to generate a final diffusion model, including: decoupling the geographic feature conditions in the remote sensing image into multiple feature condition data, and constructing a feature condition data set. The geographic feature conditions include segmentation masks and road maps. The second joint condition guidance mechanism is used to fuse multiple feature condition data to control the image generation process in the basic diffusion model.

[0078] It should be noted that the geographic feature conditions include a segmentation mask and a road map, and may also include a sketch.

[0079] Specifically, two core mechanisms are used when constructing a multi-condition control network: feature extraction and feature fusion. In feature extraction, a specialized feature extraction network (such as a stacked convolutional neural network) is used to decouple the geographic feature conditions in the remote sensing image into multiple feature condition data, and a feature condition dataset is constructed to extract key feature maps from the feature condition data. These feature maps capture the core information in the feature condition data, such as the region boundaries in the segmentation mask and the road direction in the road map.

[0080] Furthermore, stacked convolutional neural networks (CNNs) can extract useful feature representations. These features can capture key information in the image, such as boundaries, shapes, textures, etc. Take Sketch images as an example. Assume that the expression of the input Sketch image is: Where Cs is the number of channels of the image. For grayscale Sketch images, Cs = 1; for color images, Cs can be 3 or other values, Hs is the height of the image, and Ws is the width of the image. The expression of the feature map obtained by combining the Sketch image through the convolution layer and the pooling layer is set to Among them, C f is the number of channels of the output feature map, H f is the height of the output feature map, W f is the width of the input feature map. The expression of the Sketch image convolution process is: Among them, F S (i,j) is the output feature map F S The pixel value at position (i, j), S(i+m,j+n) is the pixel value of the input image S at position (i+m,j+n), K(m,n) is the weight value of the convolution kernel at position (m,n), M is the height of the convolution kernel, and N is the width of the convolution kernel.

[0081] Specifically, the second joint condition guidance mechanism is used to fuse multiple feature condition data, including: extracting key feature graphs from the feature condition data using a feature extraction network, and performing multi-scale fusion and feature weighting on the extracted key feature graphs using a feature fusion network.

[0082] Furthermore, the feature extraction network can be a stacked convolutional neural network, and the feature fusion network can be a ControlNet conditional control network, which is designed for multi-scale feature fusion and feature weighting. Through the stacked convolutional layers, the network can process and integrate feature maps from images with different feature conditions, perform feature interactions at multiple scales, and use feature weighting mechanisms to highlight important features and suppress irrelevant or redundant information. The fused features are embedded in the noise estimation of the basic diffusion model, and the image generation direction is guided by adjusting the noise distribution. After sufficient training and optimization, the final diffusion model is obtained. This model can generate remote sensing images that accurately meet the requirements based on a variety of geographic feature conditions provided by the user (such as segmentation masks, road maps, etc.). By adjusting the weights and combinations between different conditions, users can flexibly control the content and style of the generated images to meet diverse application needs.

[0083] S104: constructing a diffusion model and a remote sensing data set including condition information, wherein the condition information includes geographic feature conditions and multi-modal conditions.

[0084] S105: Performing initial training on the generated diffusion model using the remote sensing dataset, and performing final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model.

[0085] The generative diffusion model is initially trained using a remote sensing dataset, including: during the initial training process, taking the text encoding and the variable representation of the remote sensing image in the corresponding latent space as input, and optimizing the parameters of the generative diffusion model to minimize the difference between the generated image and the real image.

[0086] Specifically, a separate training strategy is adopted to initially train the generative diffusion model based on the weights of Stable Diffusion (SD) 1.5 using a remote sensing dataset. The remote sensing dataset provides rich training samples for the generative diffusion model, enabling it to learn the basic mapping relationship between text descriptions and remote sensing images.

[0087] The expression for the forward process of generating the diffusion model is: Among them, x t is the generated image at time step t, x t-1 is the data at time step t-1, β t is a parameter that controls the degree of diffusion. is a Gaussian distribution, used to describe xt The mean of this distribution is That is, a decay factor 1-βt is multiplied on the basis of the state of the previous time step to reduce the amplitude of the signal. The variance is βtI, where I is the identity matrix, which is used to represent the covariance matrix of the Gaussian distribution, indicating that the noise is independent in all dimensions and has a variance of βt.

[0088] Furthermore, the generated diffusion model is finally trained using a joint training strategy to obtain a controlled diffusion model, including: in the process of the final training, the value of the guide weight is adjusted to control the degree of dependence of the generated diffusion model on the conditional information, thereby obtaining the controlled diffusion model.

[0089] Specifically, the conditional omission strategy is used in the joint training strategy to randomly omit a condition (such as a keyword in the text description or a feature in the image) with a probability of 0.5, forcing the generative diffusion model to learn to generate reasonable images even when some information is missing. All conditions are omitted at the same time with a probability of 0.1, further challenging the generalization ability of the generative diffusion model, so that it can generate meaningful images even in the absence of any conditions.

[0090] The expression of the loss function without classifier guidance introduced in the final training process is: L CFG =E x,∈,c [ω·||∈-∈ θ (x+∈,c)|| 2 +(1-ω)·||∈-∈ θ (x+∈)|| 2 ]. Among them, L CFG is the overall representation of the loss function without classifier guidance, E x,∈,c is the expectation operator, ω is the guide weight, when ω is close to 0, the generative diffusion model relies more on noise ∈ to generate images, and considers less the conditional information. This will cause the generated images to be more random and less relevant to the given conditions. When ω is close to 1, the generative diffusion model relies more on conditional information and noise ∈ to generate images. This will make the generated images more strictly follow the input conditions, thereby improving the accuracy and controllability of the generated results. ∈ θ (x+∈,c) and ∈ θ (x+∈) represents the prediction of the noise ∈ by the generative diffusion model under the given condition c and without the given condition (i.e., relying only on noise and noisy data). ∈ θ is a function of the model (parameterized by θ) that attempts to predict the noise required to recover the original data x in the remote sensing image from the noisy data (x+∈).

[0091] Figure 7The schematic diagram of the embodiment of the present application using the joint training strategy to perform final training on the generated diffusion model to obtain the controlled diffusion model is as follows: Figure 7 As shown in the figure, a conditional control network is added to the original basic diffusion model architecture. The network structure and learnable parameters are the same as the downsampling part of the original backbone network. During the training process in this stage, the encoded image conditions are input into the conditional control network through the newly added conditional encoder, and the obtained image features in different dimensions are added to the noise prediction network through zero convolution to achieve joint control of multiple conditions.

[0092] S106: The final diffusion model and the controlled diffusion model are integrated to construct an initial multi-condition controllable diffusion model, which receives input of condition information and outputs a final remote sensing image.

[0093] Specifically, in order to generate a final remote sensing image that is both accurate and meets specific requirements, a multi-condition fusion input mechanism can be used to allow users to input multiple types of condition information according to specific requirements, including but not limited to text description conditions, image content conditions, and metadata conditions. These condition information undergoes the above processing flow within the model and together provide comprehensive guidance for generating high-quality remote sensing images.

[0094] When a text description condition is input, the text description condition is a direct expression of the user's intention. The text description condition is input in the form of natural language description and contains rich geographical and temporal information. This information is parsed by the model into a series of specific instructions or parameters to guide the presentation of geographical features and the setting of time background during the generation process. For example, "Generate remote sensing images of the area at 3 degrees south latitude and 60 degrees west longitude in July 2023" provides a clear time and space framework for the model. In addition to text descriptions, users can also provide reference images of specific areas as content image conditions. These reference images usually have high resolution and detailed geographical features and can be used as "templates" or "samples" for generating new images. By analyzing the texture, structure, color and other information of the reference image, the model can better understand and reproduce these features, thereby improving the authenticity and accuracy of the generated image. For example, a user may provide a high-resolution image showing a specific landform, vegetation or urban layout to guide the generation of a new image. When metadata conditions are input, the metadata conditions contain a series of quantitative and structured information, such as shooting time, geographical location, cloud cover, etc. This information exists in the form of precise data, providing the model with detailed guidance on the spatiotemporal characteristics of the generated image. By parsing these metadata conditions, the model can more accurately simulate the characteristics of remote sensing images in a specific spatiotemporal context. For example, specifying the shooting time as "July 2023" helps the initial multi-conditional controllable diffusion model to simulate the seasonal characteristics of the period; the geographical location "3 degrees south latitude, 60 degrees west longitude" guides the initial multi-conditional controllable diffusion model to generate the geographical characteristics of the corresponding area; and the cloud cover condition of "few clouds" helps the initial multi-conditional controllable diffusion model reduce the interference of clouds during the generation process. Within the initial multi-conditional controllable diffusion model, these three types of conditional information are effectively integrated. The text description condition provides the general framework and intention of generating images; the content image condition provides detailed geographical features and structural information; and the metadata condition provides precise spatiotemporal characteristics and quantitative data support. These three complement and verify each other, and together constitute a comprehensive guide for generating high-quality remote sensing images. Through the multi-conditional fusion input mechanism, the initial multi-conditional controllable diffusion model can generate remote sensing images that are both accurate and meet user needs.

[0095] Furthermore, the initial multi-condition controllable diffusion model receives the input of conditional information and outputs the final remote sensing image. The expression of the process is: T =x T-1 +σ T ∈ θ (x T-1 ,c), Among them, x T To generate the final remote sensing image at time step T, x T-1is the data at time step T-1, β T is the parameter that controls the degree of diffusion, σ T is the parameter that controls the amount of noise added in the Tth step, ∈ θ (x T-1 ,c)∈ θ is a parameterized function that takes x T and condition c as input and try to predict the addition to x in the forward process T-1 To generate x T In each iteration, condition c is used to guide the noise estimation so that the generated final remote sensing image gradually approaches the real situation.

[0096] S107: Using evaluation indicators to compare the output final remote sensing image with the real image, evaluate the performance of the initial multi-condition controllable diffusion model, and obtain evaluation results.

[0097] Specifically, after the initial multi-condition controllable diffusion model is constructed, in order to verify its performance, it is necessary to compare the final remote sensing image output by it with the real image. The evaluation indicators include FID (Fréchet perception distance), IS (initial score) and CLIP score.

[0098] Specifically, the expression of the FID evaluation index is: FID = |μ r -μ g | 2 +Tr(Σ r +Σ g -2ΣrΣg. Among them, μr is the mean vector of the real image set in the feature space, μg is the mean vector of the generated image set (generating the final remote sensing image set) in the feature space, |μ r -μ g | 2 is the square of the Euclidean distance between the mean vector of the real image set and the mean vector of the generated image set, Σ r is the covariance matrix of the real image set in the feature space, Σ g To generate the covariance matrix of the image set in the feature space, Tr represents the trace operation of the matrix,

[0099] The expression of IS evaluation index is: IS = exp(E x [D KL (p(y|x)|p(y))]). Where exp is the natural exponential function, which is used to convert the expected value of KL divergence into IS value, E x Denotes the expectation of the generated image (final remote sensing image) x, D KLRepresents KL divergence, which is used in IS to calculate the difference between the conditional probability distribution p(y|x) classified as y given a generated image and the category marginal distribution p(y).

[0100] S108: Iteratively train and optimize the initial multi-condition controllable diffusion model according to the evaluation results to generate a final multi-condition controllable diffusion model.

[0101] Specifically, after obtaining the performance evaluation results of the initial multi-conditional controllable diffusion model, the initial multi-conditional controllable diffusion model is iteratively trained and optimized according to the deficiencies and potential optimization space in the evaluation results. After multiple rounds of iterative training and optimization, the final generated multi-conditional controllable diffusion model will have stronger generation capabilities and higher performance, and can better meet the needs of remote sensing images in various application scenarios.

[0102] The present application embodiment also provides a remote sensing image generation device 800 based on a multi-condition controllable diffusion model, such as Figure 8 As shown, the device includes: a collection module 801, a first construction module 802, a second construction module 803, a third construction module 804, a training module 805, a fusion module 806, an evaluation module 807 and an optimization module 808.

[0103] The collection module 801 is used to collect remote sensing images, pre-process the remote sensing images, identify surface characteristic information in the remote sensing images and construct a data set.

[0104] The first construction module 802 is used to construct a basic diffusion model suitable for remote sensing image processing, and use the data set to train the basic diffusion model.

[0105] The second construction module 803 is used to construct a multi-condition control network in the basic diffusion model, train the multi-condition control network using the characteristic condition data set, and generate a final diffusion model.

[0106] The third construction module 804 is used to construct a diffusion model and a remote sensing data set including condition information, wherein the condition information includes geographic feature conditions and multi-modal conditions.

[0107] The training module 805 is used to perform initial training on the generated diffusion model using the remote sensing data set, and to perform final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model.

[0108] The fusion module 806 is used to fuse the final diffusion model and the controlled diffusion model to construct an initial multi-condition controllable diffusion model. The initial multi-condition controllable diffusion model receives input of condition information and outputs a final remote sensing image.

[0109] The evaluation module 807 is used to compare the output final remote sensing image with the real image using evaluation indicators, evaluate the performance of the initial multi-condition controllable diffusion model, and obtain evaluation results.

[0110] The optimization module 808 is used to iteratively train and optimize the initial multi-condition controllable diffusion model according to the evaluation results to generate a final multi-condition controllable diffusion model.

[0111] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0112] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0113] The methods, devices or modules described in this application can be implemented in a computer-readable program code mode. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor, a logic gate, a switch, an application-specific integrated circuit (English: Application Specific Integrated Circuit, referred to as: ASIC), a programmable logic controller and an embedded microcontroller. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer-readable program code mode, the controller can be completely implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0114] like Fig. 9 As shown, an embodiment of the present application also provides a remote sensing image generation server based on a multi-condition controllable diffusion model, including a memory 901 and a processor 902; the memory 901 is used to store computer executable instructions; the processor 902 is used to execute computer executable instructions to implement a remote sensing image generation method based on a multi-condition controllable diffusion model described above in an embodiment of the present application.

[0115] The embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the remote sensing image generation method based on a multi-condition controllable diffusion model described above in the embodiment of the present application.

[0116] It can be known from the description of the above implementation mode that a person skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0117] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0118] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A remote sensing image generation method based on a multi-condition controllable diffusion model, characterized in that: include: Collect remote sensing images, preprocess them, identify surface feature information in remote sensing images and construct data sets; Constructing a basic diffusion model suitable for remote sensing image processing, and using the data set to train the basic diffusion model; Constructing a multi-condition control network in the basic diffusion model, training the multi-condition control network using the characteristic condition data set, and generating a final diffusion model; Constructing a generative diffusion model and constructing a remote sensing data set including condition information; wherein the condition information includes geographic feature conditions and multimodal conditions; Performing initial training on the generated diffusion model using the remote sensing data set, and performing final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model; fusing the final diffusion model and the controlled diffusion model to construct an initial multi-condition controllable diffusion model, the initial multi-condition controllable diffusion model receiving an input of condition information and outputting a final remote sensing image; The output final remote sensing image is compared with the real image using evaluation indicators to evaluate the performance of the initial multi-condition controllable diffusion model and obtain evaluation results; The initial multi-condition controllable diffusion model is iteratively trained and optimized according to the evaluation results to generate a final multi-condition controllable diffusion model.

2. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1 is characterized in that: The preprocessing of the remote sensing image comprises: Use the trained deep learning denoising network to remove noise from remote sensing images; Correct the geometric distortion in remote sensing images to restore the true shape and position relationship of remote sensing images; Adjust the brightness and / or contrast of remote sensing images to make remote sensing images from different sources visually comparable; Increasing the amount and diversity of data in remote sensing images through data augmentation operations.

3. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1, characterized in that: The identifying of surface characteristic information in the remote sensing image and constructing a data set includes: The preprocessed remote sensing images are standardized, and the feature maps representing high-dimensional features in the remote sensing images are extracted using a pre-trained feature extraction network; The semantic segmentation network is used to process the feature map to identify and segment different semantic areas in the feature map; Through feature matching technology, the feature map is matched with the predefined condition template to identify the surface characteristic information in the remote sensing image; The preprocessed remote sensing images and the identified surface feature information are integrated to construct a data set.

4. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1, characterized in that: The method of constructing a basic diffusion model suitable for remote sensing image processing and training the basic diffusion model using a data set includes: The remote sensing image is converted into its variable representation in the corresponding latent space through a variational autoencoder; Minimize the reconstruction error loss function of the variational autoencoder to learn the distribution of feature representations formed after the remote sensing image is mapped to the latent space through the variational autoencoder; Creating a first joint condition guidance mechanism to integrate multimodal conditions into a unified representation; wherein the multimodal conditions include text description conditions, content image conditions, and metadata conditions; Introducing a cross-attention mechanism in the denoising process enhances cross-references and adjustments between multimodal conditions; The error between the predicted noise and the actual added noise is minimized, and a classifier-free guidance mechanism is introduced to adjust the guidance weights to control the degree of dependence of the basic diffusion model on multimodal conditions.

5. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 4 is characterized in that: The step of creating the first joint condition guidance mechanism includes: The text description is converted into a word embedding vector by comparing the text encoder of the language-image pre-training model, and the word embedding vector is used as the first encoding; The content image is converted into a feature map by an image encoder, and the feature map is processed by a feedforward network to obtain a second encoding; The metadata is passed through the linear layer and then gets the third encoding; The first encoding, the second encoding, and the third encoding are fused to create a unified representation, thereby creating a first joint conditional guidance mechanism.

6. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1, characterized in that: The method of constructing a multi-condition control network in the basic diffusion model, training the multi-condition control network using a characteristic condition data set, and generating a final diffusion model includes: Decoupling geographic feature conditions in remote sensing images into multiple feature condition data and constructing a feature condition data set; wherein the geographic feature conditions include segmentation masks and road maps; A second joint conditional guidance mechanism is used to fuse multiple feature conditional data to control the image generation process in the basic diffusion model.

7. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 6, characterized in that: The second joint condition guidance mechanism is used to fuse multiple feature condition data, including: Extract key feature maps from feature-conditioned data using a feature extraction network; The extracted key feature maps are fused at multiple scales and feature weighted using a feature fusion network.

8. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1, characterized in that: The using the remote sensing data set to initially train the generated diffusion model comprises: During the initial training process, the text encoding and the variable representation of the remote sensing image in the corresponding latent space are taken as input, and the parameters of the generative diffusion model are optimized to minimize the difference between the generated image and the real image.

9. The remote sensing image generation method based on a multi-condition controllable diffusion model according to claim 1, characterized in that: The method of using a joint training strategy to perform final training on the generated diffusion model to obtain a controlled diffusion model includes: In the final training process, the values ​​of the guided weights are adjusted to control the degree of dependence of the generated diffusion model on the conditional information, thereby obtaining a controlled diffusion model.

10. A remote sensing image generation device based on a multi-condition controllable diffusion model, characterized in that: include: The collection module is used to collect remote sensing images, pre-process the remote sensing images, identify the surface characteristic information in the remote sensing images and construct a data set; A first building module is used to build a basic diffusion model suitable for remote sensing image processing, and train the basic diffusion model using a data set; A second construction module is used to construct a multi-condition control network in the basic diffusion model, train the multi-condition control network using the characteristic condition data set, and generate a final diffusion model; The third construction module is used to construct a generation diffusion model and a remote sensing data set including condition information; wherein the condition information includes geographic feature conditions and multimodal conditions; A training module, used for performing initial training on the generated diffusion model using the remote sensing data set, and performing final training on the generated diffusion model using a joint training strategy to obtain a controlled diffusion model; A fusion module, used for fusing the final diffusion model and the controlled diffusion model to construct an initial multi-condition controllable diffusion model, the initial multi-condition controllable diffusion model receives an input of condition information and outputs a final remote sensing image; An evaluation module is used to compare the output final remote sensing image with the real image using evaluation indicators, evaluate the performance of the initial multi-condition controllable diffusion model, and obtain evaluation results; The optimization module is used to iteratively train and optimize the initial multi-condition controllable diffusion model according to the evaluation results to generate a final multi-condition controllable diffusion model.

Citation Information

Cited By

  • Non-training layout-to-image generation method based on diffusion model

    CN119169146A

  • Remote sensing image super-resolution method and product based on diffusion model and multi-modal large language model

    CN119722462A

  • Remote Sensing Image Super-Resolution Method and Product Based on Diffusion Model and Multimodal Large Language Model

    CN119722462B

  • Controllable condition clinical data synthesis method and device supporting logical relationship perception

    CN119851845A

  • Remote sensing image controllable generation method based on multi-modal information guidance

    CN119919824A