A remote sensing image generation method based on multi-condition guidance

Abstract

The present application belongs to the technical field of remote sensing image processing, deep learning image generation and remote sensing data enhancement, and particularly relates to a remote sensing image generation method based on multi-condition guidance. The method strengthens the spatial correspondence between the semantic label graph and the remote sensing image by introducing a semantic region joint attention module. Meanwhile, a multi-scale feature fusion method is used to enhance the image texture detail expressiveness, solving the problem of insufficient texture detail generation. In view of the multi-source differences existing in the imaging time, geographical area, sensor type, spatial resolution and illumination condition of the real remote sensing image, a multi-condition collaborative modeling mechanism is introduced to realize the joint regulation of the remote sensing image style. In addition, the present application constructs a shadow projection constraint mechanism based on the illumination angle, and integrates the remote sensing imaging physical law into the diffusion generation process, effectively improving the physical authenticity of the generated image.

Description

Technical Field

[0001] This invention belongs to the fields of remote sensing image processing, deep learning image generation, and remote sensing data enhancement, and specifically relates to a remote sensing image generation method based on multi-condition guidance. Background Technology

[0002] Remote sensing images are widely used in fields such as land resource monitoring, urban planning, ecological environment assessment, disaster emergency response, agricultural surveys, and military reconnaissance. With the development of deep learning technology, the demand for high-quality training samples for intelligent interpretation tasks based on remote sensing images, such as semantic segmentation, target detection, and change detection, is constantly increasing. However, the acquisition of remote sensing images is affected by factors such as satellite revisit cycles, imaging angles, weather conditions, sensor performance, and geographical regional differences, making it difficult to obtain large-scale samples covering multiple regions, time phases, and sensor conditions in a short period. Furthermore, pixel-level annotation of remote sensing images typically requires specialized knowledge, and manual annotation is costly and time-consuming, resulting in a relatively limited supply of high-quality labeled data suitable for model training.

[0003] To alleviate the problems of insufficient remote sensing image data and high annotation costs, researchers typically use data augmentation or image generation methods to expand training samples. Traditional data augmentation methods mainly include operations such as rotation, flipping, cropping, scaling, brightness perturbation, and noise injection. Although these methods can increase the number of samples, they are essentially local transformations based on the original samples, making it difficult to generate new land cover combinations, regional styles, and imaging conditions. Therefore, they have limited ability to improve the model's generalization ability across regions, time periods, and sensor scenarios.

[0004] In recent years, deep generative models have been increasingly applied to remote sensing image generation tasks. These methods can learn the data distribution of real remote sensing images and generate relatively realistic remote sensing images based on semantic label maps, text descriptions, or other conditions, providing a new technical approach for constructing low-cost training samples. However, existing remote sensing image generation methods still have the following shortcomings in complex scenarios:

[0005] (1) When existing methods are used to generate complex remote sensing scenes, the models often produce semantic confusion for land features with similar spectral responses (such as vegetation and farmland), and the generated images show that they cannot distinguish between different land feature categories.

[0006] (2) The appearance changes of real remote sensing images are affected by a variety of factors such as imaging time, geographical area, and sensor type. Most existing methods are limited to the level of single-condition control and lack a unified expression and collaborative modeling of multi-source conditions. The models can often only learn the partial control relationships in the training set and it is difficult to regulate the generated results in multiple dimensions.

[0007] (3) Existing remote sensing image generation methods usually imply changes in illumination in the distribution of training data, lack explicit modeling of the physical relationship between solar elevation angle, solar deflection angle and scene structure, which leads to problems such as inconsistent shadow direction, unreasonable shadow length, and mismatch between shadow and building or vegetation position in the generated image, thereby reducing the physical authenticity and actual usability of the generated image. Summary of the Invention

[0008] The purpose of this invention is to address the problems of poor structural consistency, insufficient controllability of multiple conditions, and lack of physical realism in existing remote sensing image generation technologies. It proposes a multi-source remote sensing image generation method based on multi-condition guidance, which achieves coordinated and controllable generation of remote sensing image structure, style, and shadow physical laws by jointly modeling semantic structural information, multi-source imaging parameters, and illumination constraints.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A remote sensing image generation method based on multi-condition guidance includes the following steps:

[0011] Step 1: Construct the training dataset: Obtain remote sensing image samples; perform pixel-level semantic annotation on the remote sensing image samples to obtain semantic label maps; extract multi-condition parameter information corresponding to each remote sensing image, including time information, geographic coordinate information, sensor type information, spatial resolution information, and illumination parameter information, including solar altitude angle and solar deflection angle; use each remote sensing image, its corresponding semantic label map, and multi-condition parameter information as a set of training data to form the training dataset;

[0012] Step 2: Construct a multi-condition guided generation network: The multi-condition guided generation network includes a perceptual compression encoder, a noise addition module, a noise prediction module, a noise reduction module, and a perceptual compression decoder;

[0013] The perceptual compression encoder is used to receive remote sensing images and compress and encode the remote sensing images into a latent representation;

[0014] The input of the noise-adding module is the latent noise representation, which is used to add Gaussian noise to the received latent representation during the training process to obtain the noisy latent representation corresponding to the t-th time step;

[0015] The noise prediction module includes a region semantic joint perception attention module, a shadow projection module, a U-Net encoder, a multi-parameter introduction module, a time-step sensitive multi-scale feature fusion module, and a U-Net decoder; wherein:

[0016] The input to the region semantic joint perception attention module is a semantic label map, which is used to perform region semantic joint perception processing on the semantic label map to obtain the region semantic condition feature output.

[0017] The input to the shadow projection module is the solar elevation angle, solar deflection angle, and semantic label map, which are used to perform shadow intensity estimation processing on the received signal to obtain shadow space prior feature output.

[0018] The first input of the U-Net encoder receives a noisy latent representation, the second input receives the region semantic conditional features output by the region semantic joint perception attention module, and the third input receives the shadow space prior features output by the shadow projection module. The U-Net encoder is used to concatenate the noisy latent representation, the region semantic conditional features, and the shadow space prior features in the channel dimension, and to perform multi-level downsampling encoding on the concatenated features to obtain an output including at least a shallow feature map, a middle feature map, and a deep feature map.

[0019] The multi-parameter input module receives time information, geographic coordinate information, sensor type information, and spatial resolution information as input; its outputs are connected to various levels of the U-Net encoder and U-Net decoder, respectively. It performs position encoding and feature mapping on the time, geographic coordinate, and spatial resolution information to obtain a first multi-parameter conditional feature; it performs text encoding and cross-attention processing on the sensor type information to obtain a second multi-parameter conditional feature; the first multi-parameter conditional feature is injected into the feature maps of each level of the U-Net encoder and U-Net decoder using additive modulation; and the second multi-parameter conditional feature is fused into the feature maps of each level of the U-Net encoder and U-Net decoder using a cross-attention mechanism.

[0020] The input of the time-step sensitive multi-scale feature fusion module is the shallow feature map, middle feature map, and deep feature map output by the U-Net encoder, and it also receives the current diffusion time step information. Based on the current diffusion time step information, it performs time-step modulation, weighted fusion, channel attention refinement, and spatial attention refinement on the encoded feature maps of the shallow feature map, middle feature map, and deep feature map to obtain a refined fused feature map. Then, through multiple parallel convolution processing, it obtains multiple refined sub-features.

[0021] The first input of the U-Net decoder receives multiple refined sub-features, which are respectively injected into the corresponding layers of the decoder; its second input receives features from the U-Net encoder; it is used to perform multi-level upsampling decoding processing on all received features and output the prediction noise of the current time step.

[0022] The noise removal module is input to the predicted noise and the noisy latent representation, and is used to obtain a noise-free latent representation by reverse denoising through time steps.

[0023] Step 3: Train the multi-condition guided generative network, which includes the following sub-steps:

[0024] Randomly sample a diffusion time step t from a preset set of time steps;

[0025] Add Gaussian noise of the corresponding noise level to the latent representation output by the perceptual compression encoder to obtain the noisy latent representation at the t-th time step;

[0026] The noisy latent representation, the region semantic conditional features output by the region semantic joint perception attention module, and the shadow space prior features output by the shadow projection module are input into the U-Net encoder; the multi-parameter conditional features output by the multi-parameter introduction module are input into each level of the U-Net encoder and the U-Net decoder; the encoded feature maps of multiple different levels output by the U-Net encoder are processed by the time-step sensitive multi-scale feature fusion module and then input into the U-Net decoder.

[0027] Calculate the loss between the predicted noise output by the U-Net decoder and the added Gaussian noise, and train a multi-condition guided generative network using the training dataset until the loss converges.

[0028] Step 4: Generate remote sensing images using the trained multi-condition guided generative network: Input the semantic label map corresponding to the image to be generated and the target multi-condition parameter information into the trained multi-condition guided generative network, and input the noiseless latent representation output by the noise prediction module into the perceptual compression decoder to obtain the final generated remote sensing image.

[0029] Furthermore, the specific steps of the region semantic joint perception attention module in performing region semantic joint perception processing on the semantic label map to obtain region semantic condition features include:

[0030] The semantic label map is converted into a one-hot encoded tensor and downsampled to the same scale as the noisy latent representation to obtain the region structure features;

[0031] The noisy latent representation is extended along the category dimension and then added element-wise to the region structure features to obtain the fused region features;

[0032] The fused regional features are used as query vectors, and the semantic embedding vectors corresponding to each land cover category in the semantic label map are used as key vectors and value vectors. The regional semantic conditional features are calculated through a cross-attention mechanism.

[0033] Furthermore, the specific steps of the shadow projection module in performing shadow intensity estimation processing on the received signal to obtain prior features of the shadow space include:

[0034] Construct the shadow direction field based on the solar deflection angle and solar altitude angle;

[0035] For each target pixel position in the semantic label map, a search path is determined along the shadow direction field within a preset maximum search distance, and uniform sampling is performed on the search path to obtain multiple candidate sampling points;

[0036] Determine the latent projector mask on the semantic label graph along the search path;

[0037] Based on the semantic label map, a pixel-level relative height map is obtained through a height prediction model, and the effective shadow projection length of each candidate sampling point is calculated by combining the solar elevation angle.

[0038] The shadow influence weight of each candidate sampling point on the target pixel position is calculated based on the effective shadow projection length, the distance between the candidate sampling point and the target pixel position, and the predefined attenuation coefficient.

[0039] The shadow influence weights of all candidate sampling points on the search path are weighted and aggregated to obtain the shadow intensity at the target pixel location, thereby obtaining the shadow space prior features.

[0040] Furthermore, the multi-parameter introduction module performs location encoding and feature mapping processing on time information, geographic coordinate information, and spatial resolution information to obtain a first multi-parameter conditional feature; it performs text encoding and cross-attention processing on sensor type information to obtain a second multi-parameter conditional feature; the first multi-parameter conditional feature is injected into the feature map of each level of the U-Net encoder and U-Net decoder using additive modulation; the specific steps of fusing the second multi-parameter conditional feature into the feature map of each level of the U-Net encoder and U-Net decoder using a cross-attention mechanism include:

[0041] The time information, geographic coordinate information, and spatial resolution information are respectively coded with sine and cosine position codes and then concatenated to obtain a numerical conditional vector.

[0042] The numerical conditional vector is mapped through a fully connected layer to the same channel dimension as the current layer feature map of the U-Net encoder or U-Net decoder to obtain the numerical modulation feature. The numerical modulation feature is used as the first multi-parameter feature and is additively modulated into the corresponding layer feature map of the U-Net encoder and U-Net decoder.

[0043] Sensor type information is constructed into a text description and encoded into a text embedding vector by a pre-trained text encoder, which is then used as the second multi-parameter conditional feature. The visual features of the current layer of the U-Net encoder or U-Net decoder are used as the query vector, and the text embedding vector is used as the key vector and value vector. The text embedding vector is then integrated into the corresponding layer feature maps of the U-Net encoder and U-Net decoder through a cross-attention mechanism.

[0044] Furthermore, the specific steps of the time-step sensitive multi-scale feature fusion module to perform time-step modulation, channel attention refinement, and spatial attention refinement on multiple different levels of encoded feature maps based on the current diffusion time-step information to obtain a single refined feature map include:

[0045] The current diffusion time step information is encoded by sine and cosine positions to obtain a time embedding vector. The time embedding vector is then passed through a linear layer to obtain a scale factor and a shift factor.

[0046] After unifying the channel dimension of the feature maps output from different levels of the U-Net encoder, affine modulation is performed using the scaling factor and the offset factor.

[0047] The temporal embedding vector is normalized by two linear transformations, a SiLU activation layer, and Softmax to generate a set of competitive weights. These competitive weights are then used to perform weighted fusion on the modulated feature maps at each level to obtain the fused feature map.

[0048] The fused feature map is refined by channel attention and spatial attention in sequence to obtain a refined feature map. The refined feature map is then concatenated with the corresponding layer feature map of the U-Net decoder through multiple parallel convolutional layers.

[0049] Furthermore, the perceptual compression encoder and perceptual compression decoder are the encoder and decoder parts of a pre-trained variational autoencoder.

[0050] Furthermore, the time information includes the year and month of imaging; the geographic coordinate information includes longitude and latitude.

[0051] By adopting the above technical solution, the present invention has the following beneficial effects:

[0052] (1) This invention uses a semantic region joint perception attention module to decouple and cross-attention the spatial structure information of the label map from the category semantic knowledge encoded by RemoteCLIP, and explicitly introduces high-level semantic priors of categories. This effectively solves the problems of category boundary drift and semantic confusion in traditional methods, and significantly improves the clarity of ground feature boundaries and the accuracy of category attribution in the generated image.

[0053] (2) The present invention uses a time-step sensitive multi-scale fusion module to dynamically adjust the fusion weights of features at different scales according to the phased rules of the diffusion denoising process. In the early stage, it prioritizes the rationality of the global structure and gradually strengthens the expression of local texture and details in the later stage, which effectively improves the problems of blurred texture and strong smearing in the images generated by the traditional diffusion model.

[0054] (3) The present invention uses a multi-parameter injection module to uniformly encode and collaboratively inject multi-source heterogeneous conditions such as time information, geographic coordinates, sensor type, spatial resolution and text style description, thereby realizing joint and refined control of the style at the ground object level and the style at the sensor level of remote sensing images. This solves the limitation of existing methods that can only perform single-condition control and are difficult to cover the diverse imaging conditions of real remote sensing.

[0055] (4) The present invention embeds a differentiable physical shadow model into a deep learning generation framework through a shadow projection module. By constructing a differentiable correlation between the sun geometry, scene structure and shadow distribution, the generated image conforms to the real physical imaging law, thereby improving the physical consistency and visual realism of the generated image. Attached Figure Description

[0056] Figure 1 This is a structural diagram of the multi-condition guided remote sensing image generation model of the present invention.

[0057] Figure 2 This is a schematic diagram of the processing flow of the semantic region joint perception attention module.

[0058] Figure 3 A schematic diagram of the processing flow for the multi-parameter injection module.

[0059] Figure 4 This is a schematic diagram of the processing flow of the shadow casting module.

[0060] Figure 5 This is a schematic diagram of the processing flow of the time-step sensitive multi-scale fusion module.

[0061] Figure 6 The figures are the experimental results, where (a) is the labeled image, (b) is the real image, (c) is the generated result of the existing diffusion model, and (d) is the generated result of the multi-condition guided generative network. Detailed Implementation

[0062] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0063] Example 1:

[0064] For ease of description, the relevant technical terms appearing in the specific implementation methods will be explained first:

[0065] Region semantic joint perception attention module (Label Attention, LA);

[0066] Time-step-sensitive multi-scale feature fusion module (TMFM);

[0067] Multi-Parameter Injection Module (MPIM);

[0068] Shadow Projection Module (SPM).

[0069] This embodiment provides a multi-source remote sensing image generation method based on multi-condition guidance, including the following steps:

[0070] Step 1: Construct the training dataset. This step includes the following sub-steps:

[0071] (1.1) Obtain remote sensing image samples; In this embodiment, remote sensing image samples that can reflect the style changes of remote sensing images are selected from the fMoW dataset; In other embodiments, other publicly available remote sensing datasets or self-built remote sensing image datasets can also be used.

[0072] (1.2) Use semantic segmentation and annotation tools to perform pixel-level semantic annotation of land cover categories in remote sensing images; and use different pixel values ​​to encode different land cover categories to obtain semantic label maps. In this embodiment, the land feature categories include seven types: buildings, vegetation, farmland, roads, water bodies, bare soil, and background, with corresponding pixel values ​​of 0, 1, 2, 3, 4, 5, and 6, respectively.

[0073] (1.3) Extract the multi-condition parameter information corresponding to each remote sensing image. The multi-condition parameter information is composed as follows:

[0074] (a) Time information, including the year and month of imaging;

[0075] (b) Geographic coordinate information, including longitude and latitude;

[0076] (c) Sensor type information;

[0077] (d) Spatial resolution information;

[0078] (e) Illumination parameters, including solar altitude angle and solar deflection angle.

[0079] (1.4) Each remote sensing image, along with its corresponding label image and multiple conditional parameters, is used as a set of training data to form a training dataset.

[0080] Step 2: Build a multi-condition guided generation network:

[0081] like Figure 1 As shown, the multi-condition guided generation network includes a perceptual compressed encoder, a noise prediction module, and a perceptual compressed decoder. The perceptual compressed encoder and the perceptual compressed decoder together constitute the perceptual compression module. The perceptual compression module is used to reduce the computational complexity of the diffusion generation process. The noise prediction module is used to predict the noise residual corresponding to the current time step during the diffusion denoising process, and to impose conditional constraints on the remote sensing image generation process based on the input semantic label map and multi-condition parameters. In this embodiment, the perceptual compressed encoder and the perceptual compressed decoder are the encoder and decoder parts of a pre-trained variational autoencoder.

[0082] The sensing compression module encoder E is used to process high-resolution remote sensing images in pixel space. Mapping to a low-dimensional latent representation It should be noted that the image size and potential representation size given in the embodiments are only one implementation method. In other implementation methods, other sizes can be set according to the input image size, compression ratio, and network structure.

[0083] The input of the noise-adding module is connected to the latent noise representation. Used to send the received potential representation Adding Gaussian noise yields the noisy latent representation at the t-th diffusion time step. .in, This represents the diffusion time step. .

[0084] The noise prediction module includes a Region Semantic Joint Awareness Attention Module (LA), a Shadow Projection Module (SPM), a U-Net encoder, a Multi-Parameter Introduction Module (MPIM), a Time-Step Sensitive Multi-Scale Feature Fusion Module (TMFM), and a U-Net decoder. The LA, MPIM, and SPM are used to encode features of different conditional information, and then input the encoded conditional features along with the latent noise representation during the diffusion process into the U-Net network. The TMFM, located between the encoder and decoder, dynamically modulates and fuses features at different scales according to the diffusion time step. The implementation details of each module are described below:

[0085] The input to the region semantic joint perception attention module is the semantic label map. Used for semantic label graphs Regional semantic joint sensing processing is performed to obtain regional semantic conditional feature output. The implementation process is as follows: Figure 2 As shown:

[0086] A semantic label graph containing 7 categories Convert to one-hot encoded tensor In this context, the category attribution of each pixel location is mapped to a one-hot vector, representing the land cover category to which that pixel belongs. For the ... There are 10 categories, among which The semantic description text for each category is constructed. For example, the building category corresponds to "building," and the vegetation category corresponds to "vegetation." The semantic description text of each category is input into the RemoteCLIP encoder and mapped to a 512-dimensional semantic embedding space to obtain semantic embedding vectors. Stacking the semantic vectors of all seven categories to form the semantic description matrix of the label image. Low-dimensional latent representation Extending along the category dimension to a noisy latent representation Then the one-hot encoded tensor The sample is fed into a downsampling process and then into a 3x3 convolution to obtain the noisy latent representation. Regional structural features of the same scale .

[0087] Regional structural features With noisy latent representation By adding elements one by one, we obtain the merged regional features. .

[0088] Merged regional features as query vector semantic embedding vector Using key vector K and value vector V, the semantic alignment features of the regions are calculated through a cross-attention mechanism. Finally, the homogeneous features will be... Input the encoder to obtain the regional semantic condition features used to implement the regional semantic constraints of the generation process.

[0089] The Multi-Parameter Input Module (MPIM) performs location encoding and feature mapping on the input time information, geographic coordinate information, and spatial resolution information, and performs text encoding and cross-attention processing on the sensor type information to obtain multi-parameter conditional features. Its implementation process is as follows: Figure 3 As shown:

[0090] Feature vectors are obtained by performing sine and cosine position encoding on time information, geographic coordinate information, and spatial resolution information respectively. , , And concatenate them to obtain a numerical conditional vector. ;

[0091] Numerical condition vector Numerical modulation features are obtained by mapping two fully connected layers to the same channel dimension as the current layer feature maps of the U-Net encoder and U-Net decoder. ; and numerical modulation features Numerical conditional embedding is achieved by adding features from each layer of the U-Ne encoder and U-Ne decoder.

[0092] Construct a text description from sensor type information. And encoded by a pre-trained text encoder Encoded as ;

[0093] Features output from the current layer of the U-Net encoder or U-Net decoder As the query vector Q, with text embedding vector As key vector K and value vector V, the text embedding vector is obtained through a cross-attention mechanism. The corresponding layer feature maps are integrated into the U-Net encoder and U-Net decoder.

[0094] The shadow projection module (SPM) receives solar elevation angle, solar deflection angle, and semantic label map as input. These are used to perform shadow intensity estimation on the received signal, resulting in the output of prior shadow space features. For its implementation process, please refer to [link / reference needed]. Figure 4 :

[0095] According to the angle of solar deflection and solar altitude angle Constructing the shadow direction field ;Shadow direction field It can be represented as:

[0096] ;

[0097] For semantic label graph The position of each target pixel in A search path is determined along the shadow direction field D within a preset maximum search distance L. The search path is then uniformly sampled to obtain K candidate sampling points. In this embodiment, L = 50 pixels and K = 64 candidate sampling points. The kth candidate sampling point... It can be represented as:

[0098] ;

[0099] in, This represents the position of the k-th candidate sampling point relative to the current pixel. The distance between them.

[0100] Along the search path In semantic label graph A latent projector mask is constructed. In this embodiment, if a candidate point's attribute in the semantic label graph is "building" or "plant," then it belongs to the latent projector. It should be noted that the set of potential projector categories can be expanded according to the specific remote sensing scenario, and may also include bridges, elevated roads, towers, mountains, or other land features with significant height differences.

[0101] To avoid the limitations of fixed shadow casting based solely on semantic categories, this embodiment further introduces a height prediction model. This height prediction model takes the semantic label map S as input and outputs a pixel-level relative height map. In one implementation, the height prediction model sequentially includes a 1×1 convolutional layer, a ReLU activation function, and another 1×1 convolutional layer to obtain a relative height map. .

[0102] According to the relative height map and solar altitude angle Calculate the effective shadow projection length of the candidate point. Effective shadow projection length It can be represented as:

[0103] ;

[0104] Based on the effective shadow projection length Distance between candidate sampling points and target pixel positions and predefined attenuation coefficient Calculate the k-th potential projector for the current pixel. The shadow affects the weight :

[0105] ;

[0106] in In this embodiment, the attenuation coefficient is represented. .

[0107] Finally, the search path was searched. The shadow influence weights of all candidate points are weighted and converged to obtain the target pixel position. shadow intensity Thus, prior features of the shadow space are obtained:

[0108] .

[0109] The U-Net encoder receives a noisy latent representation as its first input, regional semantic conditional features output by the region semantic joint perception attention module as its second input, and shadow space prior features output by the shadow projection module as its third input. The U-Net encoder concatenates the noisy latent representation, regional semantic conditional features, and shadow space prior features along the channel dimension, and performs multi-level downsampling encoding on the concatenated features to obtain at least shallow features. Mid-layer characteristics Deep features Output.

[0110] The time-step sensitive multi-scale feature fusion module (TMFM) receives the shallow, medium, and deep feature maps output by the U-Net encoder as input, and also receives the current diffusion time step information. Based on this information, it performs time-step modulation, weighted fusion, channel attention refinement, and spatial attention refinement on the encoded feature maps of the shallow, medium, and deep feature maps to obtain refined fused feature maps. These refined feature maps are then processed through three parallel convolutions to obtain the first, second, and third refined sub-features. The implementation process is as follows: Figure 5 As shown:

[0111] The information at the current diffusion time step t (t=1,2,...,1000) is encoded using sine and cosine position coding to obtain the time embedding vector. Temporal embedding vector The scaling factor is obtained after passing through a linear layer. and offset factor ;

[0112] The feature maps output from the three layers of the U-Net encoder, i.e., the shallow features Mid-layer characteristics Deep features After passing each layer through a 1x1 convolution, the channel dimensions are unified to obtain shallow features with unified dimensions. Mid-layer characteristics Deep features ; and then use the scaling factor and offset factor Affine modulation is performed to obtain a shallow feature map after affine modulation. Mid-layer feature map Deep feature map .

[0113] Temporal embedding vector After two linear transformations, a SiLU activation layer, and Softmax normalization, a set of competing weights is generated. , , The sum of the three is 1.

[0114] Use this set of competitive weights , , Assigned to the shallow feature maps after affine modulation Mid-layer feature map Deep feature map The fused feature map is obtained after weighted summation. .

[0115] For fused feature maps Global average pooling and global max pooling are performed separately to obtain two channel feature vectors with dimensions of 1×1×1536. , ;Will , The input is fed into a shared two-layer MLP, and the outputs of the two layers are summed and then activated by a sigmoid function to obtain the channel attention weights. Channel attention weights With fusion feature map Element-wise multiplication yields the channel refinement characteristics. ;right Average pooling and max pooling are performed along the channel dimension to obtain two 64×64×1 spatial feature maps. , The two feature maps are concatenated and then passed through a 7×7 convolutional layer with sigmoid activation to obtain spatial attention weights. Spatial attention weights With channel refining characteristics Element-by-element multiplication yields the final refined characteristics. .

[0116] The final refined feature map Refined features are obtained through three parallel 3x3 convolutions. , , These features are then concatenated with the features from the first, second, and third layers of the U-Net decoder, respectively.

[0117] The first input reception of the U-Net decoder , , The first, second, and third layers of the decoder are respectively injected into the second layer; the second input receives features from the U-Net encoder, and the third input receives multi-parameter conditional features that are attracted by the multi-parameter introduction module in an additive modulation manner; these features are used to perform multi-level upsampling decoding on all received features and output the prediction noise at the current time step.

[0118] The noise removal module takes as input the predicted noise and the noisy latent representation, and is used to obtain a noise-free latent representation by reverse denoising through time steps.

[0119] The input to the perceptual compression decoder is a noiseless latent representation, and the noiseless latent representation is decoded and reconstructed to obtain the final remote sensing image output.

[0120] Step 3: Train the multi-condition guided generative network, which includes the following sub-steps:

[0121] Read a set of training samples from the training dataset constructed in step 1 and input them into the multi-condition guided generation network; the training samples include remote sensing image x, semantic label map S, time information, geographic coordinate information, sensor type information, spatial resolution information, solar altitude angle and solar deflection angle.

[0122] The remote sensing image x is input into the encoder E of the sensing compression module to obtain the latent representation z; a diffusion time step t is randomly sampled from a preset set of time steps; Gaussian noise of the corresponding noise level is added to the latent representation z to obtain the noisy latent representation corresponding to the t-th time step. The semantic label map is input into the region semantic joint perception attention module (LA) to generate region semantic conditional features. The time information, geographic coordinate information, sensor type information, and spatial resolution information are input into the multi-parameter input module MPIM to generate multi-parameter conditional features. The solar elevation angle and solar deflection angle are input into the shadow projection module SPM to generate prior features for the shadow space. .

[0123] With noisy latent representation Regional semantic condition features Prior features of shadow space and multi-parameter conditional features Using the noise prediction module as input, the predicted noise is obtained. During this process:

[0124] Noisy latent representation Regional semantic condition features Prior features of shadow space After concatenation, the data is input to the U-Net encoder, and the time step t is simultaneously injected into each downsampling block of the encoder through adaptive normalization; multi-parameter conditional features. The input is synchronized to each level of the encoder and decoder through weighted modulation;

[0125] The encoded feature maps at multiple different levels output by the U-Net encoder are processed by a time-step-sensitive multi-scale feature fusion module and then input into the U-Net decoder.

[0126] Calculate the real noise With prediction noise The noise prediction loss between [variables]. In one implementation, the mean squared error loss is used as the training objective:

[0127] ;

[0128] Finally, the multi-condition guided generative network is trained using each set of training data. Training stops when the loss function converges, thus obtaining the trained multi-condition guided generative model.

[0129] Step 4: Generate remote sensing images using the trained multi-condition guided generative network:

[0130] The semantic label map S corresponding to the image to be generated, and the target multi-condition parameter information. Gaussian noise is randomly initialized in the latent space. .Will By inputting multiple conditions into a pre-trained noise prediction module, predicted noise is obtained. This noise is then continuously reversed through time steps to obtain a noise-free latent representation. Finally, the noise-free latent representation is... The decoder D of the input sensing compression module is used to obtain the final generated remote sensing image.

[0131] Figure 6 These are the experimental results, where (a) is the original image, (b) is the label image, (c) is the result generated by the existing diffusion model, and (d) is the result generated by the multi-condition guided generative network. A comparison shows that (d) has the following advantages over (c):

[0132] 1. The generation of ground feature boundaries is clearer, and the accuracy of generated categories is higher;

[0133] 2. The generated remote sensing images have an overall style closer to real images, with improved physical consistency and visual realism; 3. The generated images have clearer texture details. Overall, the multi-condition guided multi-source remote sensing image generation model significantly outperforms existing diffusion models.

[0134] By combining three controllable image generation methods based on diffusion models, the technical effectiveness of the multi-condition guided multi-source remote sensing image generation model in this embodiment is verified on the fMoW-SEG dataset. The three comparison methods are as follows:

[0135] Comparison Method 1 (ControlNet): A controllable image generation method proposed by Stanford University based on a pre-trained diffusion model. It achieves stable control over structural conditions such as edge maps and segmentation maps by adding a trainable copy encoder branch to a diffusion model with locked parameters. Its control information is injected by directly adding the channel dimensions, and the correspondence between the spatial layout of the label map and the land cover categories relies on implicit learning by the model.

[0136] Comparison Method 2 (GLIGEN): A controllable generation method based on a gated self-attention mechanism proposed by the University of Wisconsin-Madison. It achieves spatially controllable generation under open vocabulary conditions by fusing spatial localization information with text representation and injecting it into the self-attention layer. Its spatial localization relies on discretized bounding box coordinates, resulting in high parameter efficiency.

[0137] Comparison Method 3 (ControlNet-XS): A lightweight and controllable generation method proposed by the German Aerospace Center (DLR). It redesigns the control injection mechanism from a feedback control perspective, achieving efficient structural condition adaptation without increasing encoder copies. Compared to ControlNet, this method maintains image generation quality while having lower computational overhead.

[0138] The evaluation is conducted across three dimensions: image authenticity, label structure consistency, and semantic matching accuracy. The definitions of each evaluation metric are as follows:

[0139] FID (Fréchet Inception Distance): This measures the distance between the generated image and the real image in terms of feature distribution. The lower the value, the better the realism of the generated image.

[0140] LPIPS (Learned Perceptual Image Patch Similarity): This measures the difference between the generated image and the real image in the perceptual feature space. The lower the value, the higher the perceptual quality.

[0141] mIoU (mean Intersection over Union): This is the average intersection-over-union ratio for each category. First, a pre-trained semantic segmentation model is used to predict the generated image. Then, the intersection-over-union ratios of the predicted results and the ground truth label images are calculated and averaged. A higher value indicates better structural consistency between the generated image and the label image.

[0142] OA (Overall Accuracy): This represents the overall pixel accuracy, indicating the proportion of correctly classified pixels in the segmentation prediction results of the generated image. A higher value indicates a stronger ability to preserve label structure.

[0143] CLIP-Score: Used to measure the degree of matching between the generated image and the corresponding scene semantic description. The higher the value, the better the semantic consistency.

[0144] Table 1: Comparison of Generation Effects between This Example and the Comparative Example

[0145]

[0146] Compared with existing best-in-class comparison methods, this implementation achieved optimal results across all evaluation metrics. In terms of image realism, FID reached 187.74, a relative decrease of 3.2%; LPIPS reached 0.5292, a relative decrease of 0.7%, indicating higher image quality and realism in the generated images. Regarding semantic structure consistency, mIoU reached 0.3626, a relative improvement of 0.9%, and OA reached 0.6798, a relative improvement of 1.6%, indicating more accurate alignment between the generated images and semantic labels, and more accurate land cover category generation. In terms of semantic matching, CLIP-Score reached 27.45, a relative improvement of 9.3%, reflecting a significant enhancement in semantic consistency between the image and the guiding conditions.

[0147] In summary, the remote sensing image generation method of this invention strengthens the spatial correspondence between semantic label maps and remote sensing images by introducing a semantic region joint attention module; simultaneously, it enhances the expressiveness of image texture details by employing a multi-scale feature fusion method, thus solving the problem of poor structural consistency in the remote sensing image generation process. Addressing the multi-source differences in real remote sensing images across imaging time, geographical region, sensor type, spatial resolution, and illumination conditions, this invention introduces a multi-condition collaborative modeling mechanism to achieve joint control of remote sensing image style. Furthermore, this paper constructs a shadow projection constraint mechanism based on illumination angle, integrating the physical laws of remote sensing imaging into the diffusion generation process, effectively improving the physical realism of the generated images.

[0148] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A remote sensing image generation method based on multi-condition guidance, characterized in that, The method includes the following steps: Step 1: Construct the training dataset: Obtain remote sensing image samples, perform pixel-level semantic annotation on the remote sensing image samples to obtain semantic label maps; extract multi-condition parameter information corresponding to each remote sensing image, the multi-condition parameter information includes time information, geographic coordinate information, sensor type information, spatial resolution information, and illumination parameter information, the illumination parameter information includes solar altitude angle and solar deflection angle; use each remote sensing image and its corresponding semantic label map and multi-condition parameter information as a set of training data to form the training dataset; Step 2: Construct a multi-condition guided generation network: The multi-condition guided generation network includes a perceptual compression encoder, a noise addition module, a noise prediction module, a noise reduction module, and a perceptual compression decoder; The perceptual compression encoder receives remote sensing images and compresses and encodes the remote sensing images into a latent representation; The input of the noise-adding module is the latent noise representation, which is used to add Gaussian noise to the received latent representation during the training process to obtain the noisy latent representation corresponding to the t-th time step; The noise prediction module includes a region semantic joint perception attention module, a shadow projection module, a U-Net encoder, a multi-parameter introduction module, a time-step sensitive multi-scale feature fusion module, and a U-Net decoder; wherein: The input to the region semantic joint perception attention module is a semantic label map, which is used to perform region semantic joint perception processing on the semantic label map to obtain the region semantic condition feature output. The input to the shadow projection module is the solar elevation angle, solar deflection angle, and semantic label map, which are used to perform shadow intensity estimation processing on the received signal to obtain shadow space prior feature output. The first input of the U-Net encoder receives a noisy latent representation, the second input receives the region semantic conditional features output by the region semantic joint perception attention module, and the third input receives the shadow space prior features output by the shadow projection module. The U-Net encoder concatenates the noisy latent representation, the region semantic conditional features, and the shadow space prior features in the channel dimension, and performs multi-level downsampling encoding on the concatenated features to obtain an output including at least a shallow feature map, a middle feature map, and a deep feature map. The multi-parameter input module receives time information, geographic coordinate information, sensor type information, and spatial resolution information as input; its outputs are connected to various levels of the U-Net encoder and U-Net decoder, respectively. It performs position encoding and feature mapping on the time, geographic coordinate, and spatial resolution information to obtain a first multi-parameter conditional feature; it performs text encoding and cross-attention processing on the sensor type information to obtain a second multi-parameter conditional feature; the first multi-parameter conditional feature is injected into the feature maps of each level of the U-Net encoder and U-Net decoder using additive modulation; and the second multi-parameter conditional feature is fused into the feature maps of each level of the U-Net encoder and U-Net decoder using a cross-attention mechanism. The input of the time-step sensitive multi-scale feature fusion module is the shallow feature map, middle feature map, and deep feature map output by the U-Net encoder, and it also receives the current diffusion time step information. Based on the current diffusion time step information, it performs time-step modulation, weighted fusion, channel attention refinement, and spatial attention refinement on the encoded features of the shallow feature map, middle feature map, and deep feature map to obtain a refined fused feature map. Then, through multiple parallel convolution processing, multiple refined sub-features are obtained. The first input of the U-Net decoder receives multiple refined sub-features, which correspond to the corresponding levels injected into the decoder; its second input receives features from the U-Net encoder; it is used to perform multi-level upsampling decoding processing on all received features and output the prediction noise at the current time step. The noise removal module is input to the predicted noise and the noisy latent representation, and is used to obtain a noise-free latent representation by reverse denoising through time steps. Step 3: Train the multi-condition guided generative network, which includes the following sub-steps: Randomly sample a diffusion time step t from a preset set of time steps; Add Gaussian noise of the corresponding noise level to the latent representation output by the perceptual compression encoder to obtain the noisy latent representation at the t-th time step; The noisy latent representation, the region semantic conditional features output by the region semantic joint perception attention module, and the shadow space prior features output by the shadow projection module are input into the U-Net encoder; the multi-parameter conditional features output by the multi-parameter introduction module are input into each level of the U-Net encoder and the U-Net decoder; the encoded feature maps of multiple different levels output by the U-Net encoder are processed by the time-step sensitive multi-scale feature fusion module and then input into the U-Net decoder. Calculate the loss between the predicted noise output by the U-Net decoder and the added Gaussian noise, and train a multi-condition guided generative network using the training dataset until the loss converges. Step 4: Generate remote sensing images using the trained multi-condition guided generative network: Input the semantic label map corresponding to the remote sensing image to be generated and the target multi-condition parameter information into the trained multi-condition guided generative network, and input the noiseless latent representation output by the noise prediction module into the perceptual compression decoder to obtain the final generated remote sensing image.

2. The method according to claim 1, characterized in that, The specific steps of the region semantic joint perception attention module in performing region semantic joint perception processing on the semantic label map to obtain region semantic condition features include: The semantic label map is converted into a one-hot encoded tensor and downsampled to the same scale as the noisy latent representation to obtain the region structure features; The noisy latent representation is extended along the category dimension and then added element-wise to the region structure features to obtain the fused region features; The fused regional features are used as query vectors, and the semantic embedding vectors corresponding to each land cover category in the semantic label map are used as key vectors and value vectors. The regional semantic conditional features are calculated through a cross-attention mechanism.

3. The method according to claim 1, characterized in that, The specific steps by which the shadow projection module performs shadow intensity estimation processing on the received signal to obtain prior features of the shadow space include: Construct the shadow direction field based on the solar deflection angle and solar altitude angle; For each target pixel position in the semantic label map, a search path is determined along the shadow direction field within a preset maximum search distance, and uniform sampling is performed on the search path to obtain multiple candidate sampling points; Determine the latent projector mask on the semantic label graph along the search path; Based on the semantic label map, a pixel-level relative height map is obtained through a height prediction model, and the effective shadow projection length of each candidate sampling point is calculated by combining the solar elevation angle. The shadow influence weight of each candidate sampling point on the target pixel position is calculated based on the effective shadow projection length, the distance between the candidate sampling point and the target pixel position, and the predefined attenuation coefficient. The shadow influence weights of all candidate sampling points on the search path are weighted and aggregated to obtain the shadow intensity at the target pixel location, thereby obtaining the shadow space prior features.

4. The method according to claim 1, characterized in that, The multi-parameter introduction module performs location encoding and feature mapping processing on time information, geographic coordinate information and spatial resolution information to obtain the first multi-parameter conditional feature. The sensor type information is text-encoded and cross-attention processed to obtain the second multi-parameter conditional feature. The first multi-parameter conditional feature is then injected into the feature map of each level of the U-Net encoder and U-Net decoder in an additive modulation manner. The specific steps for fusing the second multi-parameter conditional features into the feature maps of each level of the U-Net encoder and U-Net decoder using a cross-attention mechanism include: The time information, geographic coordinate information, and spatial resolution information are respectively coded with sine and cosine position codes and then concatenated to obtain a numerical conditional vector. The numerical conditional vector is mapped through a fully connected layer to the same channel dimension as the current layer feature map of the U-Net encoder or U-Net decoder to obtain the numerical modulation feature. The numerical modulation feature is used as the first multi-parameter feature and is additively modulated into the corresponding layer feature map of the U-Net encoder and U-Net decoder. Sensor type information is constructed as a text description, encoded into a text embedding vector by a pre-trained text encoder, and used as a second multi-parameter conditional feature. The visual features of the current layer of the U-Net encoder or U-Net decoder are used as the query vector, and the text embedding vector is used as the key vector and value vector. The text embedding vector is integrated into the corresponding layer feature maps of the U-Net encoder and U-Net decoder through a cross-attention mechanism.

5. The method according to claim 1, characterized in that, The time-step sensitive multi-scale feature fusion module performs time-step modulation, channel attention refinement, and spatial attention refinement on multiple different levels of encoded feature maps based on the current diffusion time-step information to obtain a single refined feature map. The specific steps include: The current diffusion time step information is encoded by sine and cosine positions to obtain a time embedding vector. The time embedding vector is then passed through a linear layer to obtain a scale factor and a shift factor. After unifying the channel dimension of the feature maps output from different levels of the U-Net encoder, affine modulation is performed using the scaling factor and the offset factor. The temporal embedding vector is normalized by two linear transformations, a SiLU activation layer, and Softmax to generate a set of competitive weights. These competitive weights are then used to perform weighted fusion on the modulated feature maps at each level to obtain the fused feature map. Channel attention refinement is performed on the fused feature map to obtain the channel-refined feature map. The fused feature map is then refined by channel attention and spatial attention to obtain a refined feature map. The refined feature map is concatenated with the corresponding layer feature map of the U-Net decoder through multiple parallel convolutional layers.

6. The method according to claim 1, characterized in that, The perceptual compression encoder and perceptual compression decoder are the encoder and decoder parts of a pre-trained variational autoencoder.

7. The method according to claim 1, characterized in that, The time information includes the year and month of imaging; the geographic coordinate information includes longitude and latitude.

Images