Multi-modal semantic and physical law driven remote sensing image generation method

Abstract

The invention discloses a multi-modal semantic and physical law driven remote sensing image generation method, belongs to the technical field of computer vision and remote sensing image generation, and aims to solve the problems of insufficient cross-modal semantic alignment, low reliability of a generation result and insufficient physical mechanism fusion. The four-stage method comprises the following steps: firstly, rejecting low-quality samples from original data and unifying a spatial scale; then, extracting a multi-modal semantic vector by adopting a BLIP model and a CLIP model, and introducing a remote sensing physical rule to carry out vector optimization; then position coding and physical constraint conditions are embedded in the submerged space, and multi-source information joint modeling is achieved through a cross-modal encoder; and finally, by taking text description, physical priori knowledge and diffusion time steps as joint conditions, performing de-noising reasoning based on a Transform architecture, and completing back diffusion reconstruction by means of a trans-attention mechanism. According to the method, physical rationality and semantic consistency are improved, and a more reliable technical normal form is provided for remote sensing image generation in the fields of disaster monitoring, military simulation and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and remote sensing image generation, and particularly relates to a multi-modal semantic and physical law driven remote sensing image generation method. BACKGROUND

[0002] Optical remote sensing images have a basic position in remote sensing technology and application. They are mainly obtained by remote sensing satellites and unmanned aerial vehicles and other platforms, have characteristics such as various spatial resolutions, rich spatial and semantic information, and multi / hyperspectral imaging, and have been widely used in fields such as environmental monitoring, precision agriculture, urban planning, disaster warning and national security. At the same time, due to factors such as weather and light conditions, load and platform performance, optical remote sensing data acquisition still faces challenges such as long acquisition period, high storage and transmission cost, and complex annotation processing, which significantly increases the threshold of data acquisition and application.

[0003] In recent years, optical remote sensing image generation based on deep generative models has provided a new idea for the above-mentioned problems. On the one hand, the model can efficiently synthesize new samples by learning the distribution of existing data, which can be used to expand the high-quality data set, thereby accelerating the data acquisition and analysis process; on the other hand, the generative model can realize temporal and spatial dimension compensation and multi-modal image synthesis, and support continuous and stable monitoring in resource-poor or observation-limited areas.

[0004] In the generative model pedigree, generative adversarial networks and diffusion models have promoted the rapid improvement of image generation accuracy and efficiency. However, traditional GANs still have inherent problems such as mode collapse and unstable training. In contrast, diffusion models rely on gradual denoising mechanism and stable training characteristics, and have higher detail fidelity and better mode coverage in natural image generation, and have unique advantages in complex texture reconstruction and long-tail distribution modeling. Given the characteristics of complex ground object categories, various scales and structures, and long-tail texture statistics in remote sensing scenarios, diffusion models provide a more potential technical path for solving high-fidelity generation and controllable synthesis of optical remote sensing images.

[0005] Existing deep generative models mainly emphasize pixel-level similarity or statistical distribution fitting, while remote sensing images have significant semantic-spatial-spectral triadic correlation. On the one hand, the spatial distribution of geographical entities follows specific geographical laws and topological constraints; on the other hand, there is a close correspondence between spectral features and semantic categories. Therefore, a multi-modal semantic and physical law driven remote sensing image generation method can explicitly couple semantic priors, spatial topological constraints and spectral physical laws into the diffusion process in a unified latent space, use high-level semantic as controllable conditions, use spatial position encoding and adjacency constraints to describe geographical structure logic, and use spectral smoothness physical priors to guide reconstruction, while ensuring semantic accuracy, spatial structure rationality and spectral consistency. SUMMARY

[0006] The present application aims to provide a multi-modal semantic and physical law driven remote sensing image generation method. The method takes remote sensing imaging mechanism as a prior constraint, fuses physical rules such as spectral consistency and spatial topology, and realizes joint modeling and controllable generation of geometry-radiation-semantic in the forward noise adding and reverse denoising process of the diffusion model through cross-modal alignment of text high-level semantic and image latent representation. In view of the problems of long data acquisition period, frequent cloud and fog obstruction, label scarcity and multi-scale complex background in remote sensing scene, the method guides the generation result to simultaneously improve in semantic matching, spatial structure rationality and spectral physical consistency through the cooperation of conditionalized Transformer denoising network and position coding, so as to obtain high-fidelity, high-consistency multi / hyperspectral remote sensing images. The method significantly enhances the generation robustness and interpretability in complex scenes, and can be widely applied to remote sensing business fields such as environmental monitoring, agricultural remote sensing, urban planning, disaster assessment, land and ocean management, simulation evaluation and data enhancement.

[0007] To achieve the above-mentioned purpose, the present application provides a multi-modal semantic and physical law driven remote sensing image generation method, comprising the following steps:

[0008] S1, quality screening and scale unification processing are performed on the original remote sensing image dataset, low-quality remote sensing images with cloud cover, noise or distortion are removed, and all images are unified to a target resolution through down-sampling or super-resolution reconstruction, and then pixel value normalization and other preprocessing operations are performed;

[0009] S2, BLIP is used to automatically generate candidate text descriptions for the screened images, and spectral consistency and spatial distribution probability are introduced as physical constraints to optimize and screen the candidate texts, and high-quality semantic labels are obtained;

[0010] S3, a professional word segmenter in the remote sensing field is used to perform word segmentation processing on the semantic labels, ensuring the integrity and accuracy of the professional terms of ground objects;

[0011] S4, a pre-trained CLIP text encoder is used to convert the text into a high-dimensional semantic vector representation for cross-modal alignment;

[0012] S5, the processed remote sensing images are encoded into a latent space and embedded with two-dimensional spatial position information, and spectral and geometric physical constraints are introduced into the loss function to construct a joint latent representation that fuses semantic, geometric and radiation characteristics;

[0013] S6, performing a diffusion process on the joint latent representation: a forward process constructs training samples by adding noise step by step; a backward denoising process, with the text semantic vector, physical prior encoding and diffusion time step as joint conditions, uses a denoising network based on the Transformer architecture, realizes multi-modal condition fusion through cross-attention mechanism, iteratively predicts and removes noise, and reconstructs the discrete latent feature after denoising;

[0014] S7, mapping the denoised discrete latent feature back to the continuous space through inverse vector quantization, and reconstructing the final high-fidelity remote sensing image by using the upsampling decoder.

[0015] Preferably, the step of preprocessing the remote sensing image dataset in S1 is:

[0016] S1.1-1, according to the typical quality degradation factors of remote sensing images, the following quantitative screening standards are formulated:

[0017] Cloud coverage: remove images with cloud coverage area exceeding 10% of the total image area.

[0018] Strip noise: identify and remove images with strip noise by calculating the gradient consistency of the image in the row or column direction.

[0019] Geometric distortion: remove images with geometric distortion by checking the straightness of the image edge or verifying the control points in the case of geographic reference.

[0020] Apply the above standards to each image in the public image remote sensing dataset RSICD, and automatically or semi-automatically remove all images that do not meet the quality;

[0021] S1.2-1, images with resolution higher than the target resolution are down-sampled to the same target resolution , the specific way is as follows:

[0022] ;

[0023] ;

[0024] wherein, is the pixel value of the original high-resolution image in the local window, S is the scaling factor, is the number of pixels in the local window, x and y are the horizontal and vertical coordinate pixel indexes on the target resolution image, is the resolution higher than the target resolution image, and are the starting coordinates of x and y on the high-resolution image respectively;

[0025] S1.2-2, using ESRGAN network to super-resolution reconstruct the image with resolution less than the target resolution to make its resolution the same as the target resolution , improve the image resolution, and synthesize more abundant texture information to make up for the information loss of low-resolution images;

[0026] S1.3-1, linearly map the pixel value range of the image from the original range [0, 255] to the interval [-1, 1] to accelerate the convergence process of model training and improve numerical stability. Specifically as follows:

[0027] ;

[0028] wherein, is the normalized image, is the pixel value of each image after uniform resolution, , are the minimum and maximum pixel values of the image, respectively;

[0029] S2.1-1, input the remote sensing image preprocessed by S1 into the pre-trained BLIP model. The BLIP model extracts visual features through its image encoder and generates multiple natural language descriptions related to the semantic content of the image using its fused text decoder. The initial candidate text description set generated is ;

[0030] S2.1-2, calculate the physical score for each , the score comprehensively measures the consistency of the text and the image in terms of semantics and physics. The definition of the physical score function is as follows:

[0031] ;

[0032] wherein, text is the text description matched with the image to be evaluated, is the similarity score of the original CLIP model, is the weight of each physical constraint, is the degree of satisfaction of the mth physical constraint, is the global weight of the semantic matching component, is the global weight of the physical constraint component, and M is the number of physical constraint items included in the evaluation;

[0033] and introduce spectral consistency constraints and spatial distribution constraints , the specific method is as follows:

[0034] ;

[0035] ;​

[0036] wherein, is the number of ground object types, is the standard spectral feature of the th ground object type, is the actual spectral feature of the corresponding region of the image, is the probability of the ground object appearing at the current location, and DEM is a digital elevation model used to provide elevation information of the terrain;

[0037] S2.1-3, sort all candidate texts according to their physical scores in descending order. The top three texts with the highest scores enter the final screening pool for verification and modification. The verification criteria include:

[0038] Physical accuracy: describes whether it is consistent with the mechanism of remote sensing imaging;

[0039] Semantic fit: describes whether it accurately and unambiguously reflects the main ground objects, spatial relationships, and scene types in the image;

[0040] Language fluency and application value: describes whether it is smooth, whether it conforms to the field expression habits, and whether it has guiding significance for downstream tasks;

[0041] Preferably, the step of splitting the enhanced text using a professional remote sensing vocabulary splitter in S3 specifically includes:

[0042] S3.1-1, the preprocessing operation of the text entering the final screening pool specifically includes: removing all line breaks, tabs, and special symbols other than sentence boundary punctuation from the text; compressing multiple consecutive spaces in the text into a single space; unifying the units describing geographic scales or physical quantities in the text into a standard form; for complex long texts containing multiple independent sentences, use boundary punctuation such as periods and semicolons to split them into independent short sentences. This helps the subsequent tokenizer and encoder better capture local semantic dependencies;

[0043] S3.1-2, professional remote sensing vocabulary splitting of the preprocessed text segment, specifically including: splitting the input text into character units, constructing an initial vocabulary table wherein all unique characters; traverse the entire corpus and count the co-occurrence frequency of all adjacent word pairs. In each iteration, select the pair with the highest frequency merge into a new word , update the vocabulary table to Repeat steps S3.1-2 until the preset vocabulary size is reached or the number of merges reaches the limit. Finally, a self-built sub-vocabulary vocabulary learned through data-driven learning is obtained. The updated self-built sub-vocabulary vocabulary is matched with the remote sensing professional lexicon. Sub-vocabularies follow the remote sensing professional lexicon first. For words that do not appear in the lexicon, they are retained in the self-built sub-vocabulary vocabulary.

[0044] The above-mentioned professional word segmentation process can effectively solve the problem of segmenting long terms and compound words in the field of remote sensing, while also taking into account the flexibility of general characters. Finally, it generates a sub-word vocabulary that adapts to the semantic features of remote sensing text, providing high-quality semantic input units for subsequent text encoders and significantly improving the accuracy of cross-modal alignment.

[0045] Preferably, the step in S4, which uses a pre-trained CLIP text encoder to convert text into a high-dimensional semantic vector representation, specifically includes:

[0046] S4.1-1, Define a trainable embedding matrix ,in Let d be the vocabulary size and d be the embedding dimension for each subword. The embedding vector is obtained by looking up the table. Meanwhile, to infuse the model with sequence order information, sine-cosine position coding is introduced. The details are as follows:

[0047] ;

[0048] ;

[0049] Where pos represents the position of the word in the sequence, and i represents the dimension index. Finally, the embedding of the input sequence is represented as:

[0050] ;

[0051] S4.2-1. Add the word embedding vector and the position encoding vector element-wise to form the initial input of the Transformer encoder. The resulting word vector sequence after fusing positional information... The input is fed into the multi-layered Transformer structure of the CLIP text encoder. In the self-attention mechanism of each layer, the input is first processed... Linear projection onto the query, key, and value space. Specifically:

[0052]

[0053]

[0054] ;

[0055] S4.2-2. Based on the above projection, calculate the attention score and use scaled dot product attention to stabilize training. Simultaneously, add an attention mask, marking the filler characters of the sequence as True while preserving other positions, ensuring that each position can only focus on the positions preceding it and itself, preventing information leakage. Finally, normalize to attention weights using softmax, specifically:

[0056] ;

[0057] in, It is a mask matrix;

[0058] S4.2-3. The attention output, computed independently for each head, is expanded into a multi-head model and mapped back to the original model's dimensions through a linear transformation, preserving multi-view features, as detailed below:

[0059] ;

[0060] Where h is the number of attention heads, For the output of the i-th attention head, It is a learnable linear layer;

[0061] S4.2-4. The attention output is passed through residual connections and layer normalization to alleviate the gradient vanishing problem in deep networks, as detailed below:

[0062] ;

[0063] S4.2-5, will A feedforward neural network, typically consisting of two linear transformations and a nonlinear activation function, is used to enhance the model's nonlinearity. Residual connections and layer normalization are also employed. Specifically:

[0064] ;

[0065] S4.2-6. The Transformer modules are stacked in multiple layers: the shallow layer captures the grammatical structure, the middle layer models the relationships between phrases, and the deep layer integrates global semantics. Finally, the first position of the output of the last Transformer module or the average pooling of the output of all positions is taken as the global semantic representation of the entire text sequence.

[0066] ;

[0067] Preferably, step S5, which maps the processed remote sensing image to the latent space and fuses spatial location coding to generate a latent space semantic representation containing physical constraints through a cross-modal encoder, is as follows:

[0068] S5.1-1 Input Image Continuous latent variables are obtained through the encoder VQ-VAE. The encoder consists of a series of convolutional layers, residual blocks, and downsampling layers, used to extract high-level features of the image and reduce its spatial dimensionality.

[0069] S5.2-1. To preserve the spatial structure information of the input image and prevent the loss of positional information due to the translation invariance of the convolutional network, in continuous latent variables... Two-dimensional positional coding and two-dimensional sine-cosine positional coding are superimposed on top to preserve spatial geometry, as follows:

[0070] ;

[0071] ;

[0072] The enhanced latent variables are obtained by adding the positional codes element-wise to the continuous latent variables. The details are as follows:

[0073] ;

[0074] in, Spatial coordinates;

[0075] S5.2-2 After adding positional encoding, use vector quantization to look up and replace the nearest neighbor codebook vector. To obtain the discrete latent representation, as follows:

[0076] ;

[0077] in, It is the index of the quantized discrete code vector.

[0078] S5.3-1. To maintain the physical authenticity of remote sensing images, a spectral consistency constraint is added to the loss function during VQ-VAE training. and geometric consistency constraints This constitutes the joint loss function. The details are as follows:

[0079] ;

[0080] ;

[0081] ;

[0082] Where C represents the total number of bands in the image, To generate the pixel values ​​of the image in band c, It is a set of adjacent pixel pairs. The MSE loss is calculated for the input and reconstructed images. For VQ-VAE's commitment loss, , These are the weight parameters for MSE loss and VQ-VAE, respectively. , These are hyperparameters representing physical constraints.

[0083] Through this joint loss function, the model not only learns the visual reconstruction of images during training, but is also explicitly guided to comply with the physical laws of remote sensing imaging, thereby generating high-fidelity remote sensing images that combine visual similarity and physical rationality, providing reliable generated data support for subsequent applications such as land cover classification and target recognition.

[0084] S5.3-2. To improve stability and representational power during training, an exponential moving average is used to dynamically update the codebook vector, thereby improving training stability, as detailed below:

[0085] ;

[0086] ;

[0087] in, It is a code vector The allocation count, It is allocated in the current batch Number of times, It is the smoothing coefficient;

[0088] Preferably, the forward diffusion and backward propagation steps of the diffusion model in S6 are as follows:

[0089] S6.1-1 Initial Input Data Consistent with dataset distribution For each time step Independently sampled noise from a standard Gaussian distribution, wherein the sampled noise satisfies And a set of fixed variance coefficients is used. Controlling noise intensity, among which, Follow As the noise level increases, the proportion of noise gradually rises; this process is modeled as a Markov chain, and its forward diffusion process satisfies the following formula:

[0090] ;

[0091] The conditional transition distribution for each step is as follows: For any time transition distribution All can be based on and Obtain directly;

[0092] S6.1-2, Based on the fixed variance coefficient Obtain the retention coefficient This scales the original data, ensuring that key features of the original data are not completely obscured.

[0093] S6.1-3. Generate standard Gaussian noise of the same dimension as the original data, and control the noise amplitude through the noise figure;

[0094] S6.1-4, Take the sample from the previous time step by Scaling and adding the noise yields the noisy data for the current moment. The details are as follows:

[0095] ;

[0096] in, Standard Gaussian noise;

[0097] S6.1-5, Repeat S6.1-1 to S6.1-4 until the real data is transformed from clear to pure noise.

[0098] S6.2-1. The vector sequence encoded in text, the vector obtained from physical prior knowledge via MLP, and the time step vector are concatenated into a conditional input sequence. This sequence serves as the conditional input to the denoising network. Through cross-attention mechanisms and interaction with the latent image representation, the model explicitly integrates semantic associations, physical constraints, and diffusion progress information while learning noise prediction, ultimately generating a high-fidelity image that conforms to both visual realism and remote sensing physical laws. Specifically:

[0099] ;

[0100] in, A vector sequence encoded as text. The vector is obtained from the physical priors using MLP. This is the time step vector.

[0101] S6.3-1, Noisy latent tensor tie A vector is generated and projected onto the model dimension to obtain an image vector. This is used as the query vector Q. The unified conditional input sequence is treated as an external searchable memory, from which the semantic and mechanistic priors most relevant to the current spatial location are selectively retrieved. These priors are then combined with the diffusion time step context as the key vector K and value vector V. To reduce redundant overhead, the key-value representations of the text and physical priors can be cached at the sequence level, with only the time step component updated with time step t. Specifically:

[0102] ;

[0103] in, This is the vector representation of layer 0. The weight matrix is ​​the input projection. To broadcast the same bias to all N vectors, the residual structure is first pre-normalized, and then the diffusion time step and physical prior are injected in a channel-level gated manner through conditional affine modulation; specifically, for the ... Layer input Conditional modulation is performed to obtain Then, multi-head self-attention is performed to integrate spatial structure information and spectral correlation to form a robust context of the same mode. Specifically:

[0104] ;

[0105] ;

[0106] ;

[0107] in It comes from physical a priori knowledge. With time step joint mapping, For dimension-wise scaling.

[0108] This process maintains geometric-texture consistency and spectral domain structure while providing noise-robust same-modal priors for subsequent cross-modal fusion.

[0109] S6.3-2, Given a latent feature map According to the predicted noise Obtain the denoising limit map at this moment. The details are as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] in, It's a noise schedule. It is the retention coefficient at step t. It is the cumulative retention coefficient up to step t;

[0114] S6.3-3, Based on the above results Calculate the mean of the optimal retreat direction. Update the feature map of the next state. The details are as follows:

[0115] ;

[0116] ;

[0117] in, This step, using the sampling standard deviation, ensures convergence in the right direction while preserving natural diversity. It is standard Gaussian noise;

[0118] S6.3-4, repeat S6.1-1 to S6.3-3, from... Keep retreating until Thus, a clear discrete latent characterization is obtained. ;

[0119] Preferably, the specific steps of the S7 decoder in converting the latent pixel space to obtain a high-fidelity remote sensing image include:

[0120] S7.1-1, Denoising the discrete latent representation Consider it as a containing A set of feature vectors, each vector having dimension . First, it is reshaped into a two-dimensional feature map for subsequent processing.

[0121] ;

[0122] in, It is the reorganized two-dimensional latent feature map.

[0123] S7.1-2, Use a linear projection layer to reduce the number of channels of the latent features. Initial number of channels required to map to the decoder The details are as follows:

[0124] ;

[0125] in, It is the projection weight matrix. It is a bias term. These are the projected features. These are the latent features of the input;

[0126] S7.1-3. Based on the pre-trained vector codebook, a nearest neighbor search operation is performed on image features, mapping continuous feature representations to corresponding discrete symbols in the codebook. This inverse mapping process realizes the transformation from distributed representation to structured representation, effectively improving the distribution consistency and numerical stability of the decoder's input features.

[0127] S7.1-4 The decoder adopts a multi-stage topology, progressively expanding the spatial scale of the feature map through a hierarchical forward propagation path. Each processing stage sequentially performs two core operations: resolution enhancement and feature refinement. First, the feature map size is expanded using spatial interpolation. Then, a nonlinear transformation module composed of multiple convolutional kernels and normalization units is connected, significantly enhancing the continuous representation of ground object edges and the ability to preserve local texture details. At the same time, it effectively avoids spectral aliasing and geometric distortion caused by discrete sampling. This achieves a good balance between computational load and reconstruction quality. Through a progressive restoration mechanism from low-frequency contours to high-frequency details, it ensures that the generated image has both accurate spatial layout and geometric shape, while retaining rich surface texture features and radiometric accuracy, fully meeting the stringent image fidelity requirements of professional remote sensing applications.

[0128] Beneficial Effects: This invention proposes a remote sensing image generation method driven by multimodal semantics and physical laws. It organically couples the physical mechanisms of remote sensing imaging with high-level semantic priors within a unified latent space. Through a closed-loop process of "data consistency - semantic acquisition - latent representation - conditional denoising - physical correction - pixel domain reconstruction," it significantly improves the overall performance of generated images in terms of semantic alignment, spatial structure rationality, and spectral physical consistency. Controllable conditional cues are constructed based on BLIP and CLIP textual semantics and remote sensing-specific vocabularies. Combined with two-dimensional location coding and geographic topological constraints, this ensures that the generated results are more consistent with real geographic patterns in terms of feature layout. Secondly, a cross-attention mechanism is introduced into the conditional Transformer denoising network to achieve collaborative modulation of semantic cues, physical priors, and temporal step information, enabling fine-grained controllability of the backdiffusion process regarding "where to generate, what to generate, and to what extent to generate." Employing latent diffusion and multi-scale window attention, it balances long-range dependencies in large-format scenes with local detail reconstruction, obtaining high-resolution, high-fidelity images at a lower computational cost. This invention effectively avoids typical mismatches such as semantic discrepancies and spatial logic distortions, and maintains stronger robustness and generalization ability even under complex conditions such as cloud cover, lighting variations, differences in ground scale, and scarcity of training samples. Therefore, the generated remote sensing images can not only serve as data augmentation and domain-adaptive samples for downstream tasks such as segmentation, detection, and change monitoring, significantly reducing annotation costs and improving accuracy and stability in small-sample scenarios, but also be used for simulation evaluation and scheme derivation, serving operational scenarios such as environmental monitoring, agricultural and forestry surveys, urban planning, disaster assessment, and coastal zone and land resource management, demonstrating comprehensive advantages of high fidelity, high controllability, strong interpretability, and engineering applicability. Attached Figure Description

[0129] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0130] Figure 2This is a schematic diagram illustrating the process of filtering and processing images in a dataset;

[0131] Figure 3 This is a schematic diagram of the process for generating and optimizing enhanced text descriptions;

[0132] Figure 4 This is a schematic diagram of the optimal descriptive text preprocessing workflow;

[0133] Figure 5 This is a diagram illustrating the text encoding process;

[0134] Figure 6 This is a schematic diagram of the spatial location coding structure for image fusion;

[0135] Figure 7 This is a flowchart of the overall training process for the diffusion model;

[0136] Figure 8 This is a schematic diagram of the structure of a high-fidelity image with latent space transformation;

[0137] Figure 9 This is a schematic diagram of the first dataset image where candidate texts are optimized into final descriptive texts.

[0138] Figure 10 This is a schematic diagram of the second dataset images used to optimize candidate texts into final descriptive texts.

[0139] Figure 11 This is a real dataset image from a specific application example of the model;

[0140] Figure 12 This is a high-resolution remote sensing image generated in the first specific application embodiment of the model;

[0141] Figure 13 This is a specific application example of the model, using real dataset images;

[0142] Figure 14 This is a high-resolution remote sensing image generated in the second specific application example of the model. Detailed Implementation

[0143] The invention will now be further described with reference to the accompanying drawings.

[0144] like Figure 1 As shown, a remote sensing image generation method driven by multimodal semantics and physical laws includes the following steps:

[0145] S1. The original remote sensing image dataset is subjected to quality screening and scale unification processing to remove low-quality remote sensing images with cloud cover, noise, or distortion. All images are then unified to the target resolution through downsampling or super-resolution reconstruction, followed by preprocessing operations such as pixel value normalization. S2. BLIP is used to automatically generate candidate text descriptions for the screened images. Spectral consistency and spatial distribution probability are introduced as physical constraints to optimize and screen the candidate texts, obtaining high-quality semantic labels.

[0146] S3. Use a professional word segmenter in the field of remote sensing to segment the semantic tags to ensure the integrity and accuracy of the ground feature terminology;

[0147] S4. A pre-trained CLIP text encoder is used to convert the text into a high-dimensional semantic vector representation for cross-modal alignment.

[0148] S5. The processed remote sensing image is encoded into the latent space and embedded with two-dimensional spatial location information. At the same time, by introducing spectral and geometric physical constraints into the loss function, a joint latent representation integrating semantic, geometric and radiometric properties is constructed.

[0149] S6. Perform a diffusion process on the joint latent representation: the forward process constructs training samples by gradually adding noise; the reverse denoising process uses text semantic vectors, physical prior encoding and diffusion time steps as joint conditions, adopts a denoising network based on the Transformer architecture, realizes multimodal conditional fusion through cross-attention mechanism, iteratively predicts and removes noise, and reconstructs the denoised discrete latent features.

[0150] S7. The denoised discrete latent features are mapped back to the continuous space through inverse vector quantization, and the final high-fidelity remote sensing image is reconstructed using an upsampling decoder.

[0151] Furthermore, such as Figure 2 As shown, the steps of S1 for preprocessing the remote sensing image dataset are as follows:

[0152] S1.1-1. Based on typical quality degradation factors of remote sensing images, the following quantitative screening criteria are established:

[0153] Cloud cover: Images with cloud cover exceeding 10% of the total image area are removed.

[0154] Strip noise: Images with strip noise are identified and removed by calculating the gradient consistency of the image in the row or column direction.

[0155] Geometric distortion: Images with geometric distortion are removed by checking the straightness of image edges or verifying control points with georeference.

[0156] The above criteria are applied to each image in the publicly available remote sensing image dataset RSICD, automatically or semi-automatically removing all images that do not meet the quality requirements;

[0157] S1.2-1. Downsample an image with a resolution greater than the target resolution to bring its resolution to the same as the target resolution. The specific method is as follows:

[0158] ;

[0159] ;

[0160] in, S represents the pixel values ​​of the original high-resolution image within a local window, where S is the scaling factor. Here, x represents the number of pixels within the local window, and y represents the pixel indices of the horizontal and vertical coordinates on the target resolution image, respectively. For a resolution higher than the target resolution image, and These are the starting coordinates of x and y on the high-resolution image, respectively;

[0161] S1.2-2. Use the ESRGAN network to super-reconstruct images with a resolution lower than the target resolution, bringing their resolution back to the same as the target resolution. This improves image resolution and synthesizes richer texture information to compensate for the lack of information in low-resolution images.

[0162] S1.3-1. Linearly map the pixel value range of the image from the original range [0, 255] to the interval [-1, 1] to accelerate the convergence process of model training and improve numerical stability. Specifically:

[0163] ;

[0164] in, The image after normalization. These are the pixel values ​​of each image after achieving a unified resolution. , These are the minimum and maximum pixel values ​​of the image, respectively.

[0165] Furthermore, such as Figure 3 As shown, the steps in S2, which involve generating descriptive text based on the BLIP model and semantically matching the image content, and then optimizing and enhancing the text using remote sensing physical characteristics, specifically include:

[0166] S2.1-1. Input the remote sensing image processed in S1 into the pre-trained BLIP model. The BLIP model extracts visual features through its image encoder and uses its fused text decoder to generate multiple natural language descriptions related to the semantics of the image content. The resulting initial candidate text description set is as follows:

[0167] S2.1-2, For each calculated physics score The score comprehensively measures the semantic and physical consistency between text and image. The physical scoring function is defined as follows:

[0168] ;

[0169] Where text is the image to be evaluated. The accompanying text description, It is the similarity score of the original CLIP model. The weights of each physical constraint, Let m be the degree of satisfaction of the m-th physical constraint. The global weights for semantic matching components. , where M is the global weight of the physical constraint component and M is the number of physical constraint entries included in the evaluation;

[0170] And introduce spectral consistency constraints and spatial distribution constraints The specific method is as follows:

[0171] ;

[0172] ;

[0173] in, It is the number of land cover types. It is the first Standard spectral characteristics of each land cover type It is the actual spectral characteristics of the corresponding region in the image. Ground features The probability of occurrence at the current location; DEM, or Digital Elevation Model, is used to provide elevation information of the terrain.

[0174] S2.1-3. Sort all candidate texts in descending order of their physical scores. The top three texts with the highest scores enter the final screening pool for verification and modification. Specifically, remote sensing experts or trained annotators can manually verify and modify the descriptions of the top-ranked texts. The verification criteria include:

[0175] Physical accuracy: describes whether it conforms to the remote sensing imaging mechanism;

[0176] Semantic fit: Whether the description accurately and unambiguously reflects the main features, spatial relationships, and scene type in the image;

[0177] Language fluency and application value: Is the description fluent, does it conform to the domain's expression habits, and does it provide guidance for downstream tasks?

[0178] Furthermore, such as Figure 9 As shown, this is a satellite aerial view of an urban area, specifically as follows: On the left side of the image are rows of long, narrow industrial buildings; in the middle right is a school campus, with a green football field and a blue rectangular swimming pool visible; above are dense residential buildings; and along the right edge is a main north-south road, with internal roads and trees dotting the landscape. Overall, it presents a mixed urban landscape of industry, campus, and residence.

[0179] like Figure 10 As shown, this is a satellite view of a large oval stadium. A green lawn forms the center of the stadium, with a clearly visible red running track surrounding it. It is surrounded by tiered stands, with rainbow-colored sections on the outer edges. A large parking area connects to the east side of the stadium, which is surrounded by city roads and buildings. The overall layout is a standard multi-purpose stadium with a closed circular grandstand and a central pitch.

[0180] The top three text descriptions are generated using BLIP, as shown in the table below:

[0181] for Figure 9 The top three candidate texts had incomplete scene relationships and weak differentiation. After manual verification and modification, the football field on campus and the two sides of the long industrial plant were described simultaneously. This was both specific and accurate, and presented the spatial juxtaposition relationship. The information was complete and highly verifiable.

[0182] for Figure 10 The top three candidate texts lacked detail and neglected the core structure. After manual verification and modification, they simultaneously met the requirements of correct shape, complete core elements, sufficient detail, clear semantics, and physical verifiability.

[0183] Furthermore, such as Figure 4 As shown, the step in S3 of segmenting the enhanced text using a professional remote sensing word segmenter specifically includes:

[0184] S3.1-1. Preprocessing operations for text entering the final filtering pool specifically include: removing all line breaks, tabs, and special characters except sentence boundary punctuation; compressing multiple consecutive spaces into a single space; standardizing the units describing geographical scales or physical quantities in the text; and splitting complex long texts containing multiple independent clauses into independent short sentences using boundary punctuation such as periods and semicolons. This helps subsequent word segmentation and encoders better capture local semantic dependencies.

[0185] S3.1-2. Perform professional remote sensing vocabulary segmentation on the preprocessed text fragments, specifically including: splitting the input text into character units and constructing an initial vocabulary. ,in For all unique characters; iterate through the entire corpus, count the co-occurrence frequency of all adjacent word pairs, and in each iteration, select the pair with the highest frequency. Merge into a new word The vocabulary list has been updated to... Repeat steps S3.1-2 until the preset vocabulary size is reached or the maximum number of merges is reached, ultimately resulting in a self-built sub-vocabulary vocabulary learned through data-driven learning. Match the updated self-built sub-vocabulary vocabulary with a remote sensing professional thesaurus, prioritizing sub-vocabularies from the remote sensing professional thesaurus. For words not found in the thesaurus, retain them in the self-built sub-vocabulary vocabulary.

[0186] The above-mentioned professional word segmentation process can effectively solve the problem of segmenting long terms and compound words in the field of remote sensing, while also taking into account the flexibility of general characters. Finally, it generates a sub-word vocabulary that adapts to the semantic features of remote sensing text, providing high-quality semantic input units for subsequent text encoders and significantly improving the accuracy of cross-modal alignment.

[0187] Furthermore, such as Figure 5 As shown, the step in S4, which uses a pre-trained CLIP text encoder to convert text into a high-dimensional semantic vector representation, specifically includes:

[0188] S4.1-1, Define a trainable embedding matrix ,in Let d be the vocabulary size and d be the embedding dimension for each subword. The embedding vector is obtained by looking up the table. Meanwhile, to infuse the model with sequence order information, sine-cosine position coding is introduced. The details are as follows:

[0189] ;

[0190] ;

[0191] Where pos represents the position of the word in the sequence, and i represents the dimension index. Finally, the embedding of the input sequence is represented as:

[0192] ;

[0193] S4.2-1. Add the word embedding vector and the position encoding vector element-wise to form the initial input of the Transformer encoder. The resulting word vector sequence after fusing positional information... The input is fed into the multi-layered Transformer structure of the CLIP text encoder. In the self-attention mechanism of each layer, the input is first processed... Linear projection onto the query, key, and value space. Specifically:

[0194]

[0195]

[0196] ;

[0197] S4.2-2. Based on the above projection, calculate the attention score and use scaled dot product attention to stabilize training. Simultaneously, add an attention mask, marking the filler characters of the sequence as True while preserving other positions, ensuring that each position can only focus on the positions preceding it and itself, preventing information leakage. Finally, normalize to attention weights using softmax, specifically:

[0198] ;

[0199] in, It is a mask matrix;

[0200] S4.2-3. The attention output, computed independently for each head, is expanded into a multi-head model and mapped back to the original model's dimensions through a linear transformation, preserving multi-view features, as detailed below:

[0201] ;

[0202] Where h is the number of attention heads, For the output of the i-th attention head, It is a learnable linear layer;

[0203] S4.2-4. The attention output is passed through residual connections and layer normalization to alleviate the gradient vanishing problem in deep networks, as detailed below:

[0204] ;

[0205] S4.2-5, will A feedforward neural network, typically consisting of two linear transformations and a nonlinear activation function, is used to enhance the model's nonlinearity. Residual connections and layer normalization are also employed. Specifically:

[0206] ;

[0207] S4.2-6. The Transformer module is stacked in multiple layers: the shallow layer captures the syntactic structure, the middle layer models the relationships between phrases, and the deep layer integrates global semantics. Finally, the first position of the output of the last Transformer module or the average pooling of the output of all positions is taken as the global semantic representation of the entire text sequence.

[0208] ;

[0209] Furthermore, such as Figure 6 As shown, step S5 maps the processed remote sensing image to the latent space and fuses spatial location coding to generate a latent space semantic representation containing physical constraints through a cross-modal encoder.

[0210] S5.1-1 Input Image Continuous latent variables are obtained through the encoder VQ-VAE. The encoder consists of a series of convolutional layers, residual blocks, and downsampling layers, used to extract high-level features of the image and reduce its spatial dimensionality.

[0211] S5.2-1. To preserve the spatial structure information of the input image and prevent the loss of positional information due to the translation invariance of the convolutional network, in continuous latent variables... Two-dimensional positional coding and two-dimensional sine-cosine positional coding are superimposed on top to preserve spatial geometry, as follows:

[0212] ;

[0213] ;

[0214] The enhanced latent variables are obtained by adding the positional codes element-wise to the continuous latent variables. The details are as follows:

[0215] ;

[0216] in, Spatial coordinates;

[0217] S5.2-2 After adding positional encoding, use vector quantization to look up and replace the nearest neighbor codebook vector. To obtain the discrete latent representation, as follows:

[0218] ;

[0219] in, It is the index of the quantized discrete code vector.

[0220] S5.3-1. To maintain the physical authenticity of remote sensing images, a spectral consistency constraint is added to the loss function during encoder VQ-VAE training. and geometric consistency constraints This constitutes the joint loss function. The details are as follows:

[0221] ;

[0222] ;

[0223] ;

[0224] Where C represents the total number of bands in the image, To generate the pixel values ​​of the image in band c, It is a set of adjacent pixel pairs. The MSE loss is calculated for the input and reconstructed images. For VQ-VAE's commitment loss, , These are the weight parameters for MSE loss and VQ-VAE, respectively. , These are hyperparameters representing physical constraints.

[0225] Through this joint loss function, the model not only learns the visual reconstruction of images during training, but is also explicitly guided to comply with the physical laws of remote sensing imaging, thereby generating high-fidelity remote sensing images that combine visual similarity and physical rationality, providing reliable generated data support for subsequent applications such as land cover classification and target recognition.

[0226] S5.3-2. To improve stability and representational power during training, an exponential moving average is used to dynamically update the codebook vector, thereby improving training stability, as detailed below:

[0227] ;

[0228] ;

[0229] in, It is a code vector The allocation count, It is allocated in the current batch Number of times, It is the smoothing coefficient;

[0230] Furthermore, such as Figure 7 As shown, the steps of forward diffusion and backward propagation in the diffusion model in S6 are as follows:

[0231] S6.1-1 Initial Input Data Consistent with dataset distribution For each time step Independently sampled noise from a standard Gaussian distribution, wherein the sampled noise satisfies And a set of fixed variance coefficients is used. Controlling noise intensity, among which, The noise level increases with increasing t, leading to a gradual increase in the noise percentage. This process is modeled as a Markov chain, and its forward diffusion process satisfies the following formula:

[0232] ;

[0233] The conditional transition distribution for each step is as follows: For any time transition distribution All can be based on and Obtain directly;

[0234] S6.1-2, Based on the fixed variance coefficient Obtain the retention coefficient This scales the original data, ensuring that key features of the original data are not completely obscured.

[0235] S6.1-3. Generate standard Gaussian noise of the same dimension as the original data, and control the noise amplitude through the noise figure;

[0236] S6.1-4, Take the sample from the previous time step by Scaling and adding the noise yields the noisy data for the current moment. The details are as follows:

[0237] ;

[0238] in, Standard Gaussian noise;

[0239] S6.1-5, Repeat S6.1-1 to S6.1-4 until the real data is transformed from clear to pure noise.

[0240] S6.2-1. The vector sequence encoded in text, the vector obtained from physical prior knowledge via MLP, and the time step vector are concatenated into a conditional input sequence. This sequence serves as the conditional input to the denoising network. Through cross-attention mechanisms and interaction with the latent image representation, the model explicitly integrates semantic associations, physical constraints, and diffusion progress information while learning noise prediction, ultimately generating a high-fidelity image that conforms to both visual realism and remote sensing physical laws. Specifically:

[0241] ;

[0242] in, A vector sequence encoded as text. Vectors obtained from physical priors via MLP This is the time step vector.

[0243] S6.3-1, Noisy latent tensor tie A vector is generated and projected onto the model dimension to obtain an image vector. This is used as the query vector Q. The unified conditional input sequence is treated as an external searchable memory, from which the semantic and mechanistic priors most relevant to the current spatial location are selectively retrieved. These priors are then combined with the diffusion time step context as the key vector K and value vector V. To reduce redundant overhead, the key-value representations of the text and physical priors can be cached at the sequence level, with only the time step component updated with time step t. Specifically:

[0244] ;

[0245] in, This is the vector representation of layer 0. The weight matrix is ​​the input projection. To broadcast the same bias to all N vectors, the residual structure is first pre-normalized, and then the diffusion time step and physical prior are injected in a channel-level gated manner through conditional affine modulation; specifically, for the ... Layer input Conditional modulation is performed to obtain Then, multi-head self-attention is performed to integrate spatial structure information and spectral correlation to form a robust context of the same mode. Specifically:

[0246] ;

[0247] ;

[0248] ;

[0249] in It comes from physical a priori knowledge. With time step joint mapping, For dimension-wise scaling.

[0250] This process maintains geometric-texture consistency and spectral domain structure while providing noise-robust same-modal priors for subsequent cross-modal fusion.

[0251] S6.3-2, Given a latent feature map According to the predicted noise Obtain the denoising limit map at this moment. The details are as follows:

[0252] ;

[0253] ; ;

[0254] in, It's a noise schedule. It is the retention coefficient at step t. It is the cumulative retention coefficient up to step t;

[0255] S6.3-3, Based on the above results Calculate the mean of the optimal retreat direction. Update the feature map of the next state. The details are as follows:

[0256] ;

[0257] ;

[0258] in, This step, using the sampling standard deviation, ensures convergence in the right direction while preserving natural diversity. It is standard Gaussian noise;

[0259] S6.3-4, repeat S6.1-1 to S6.3-3, from... Keep retreating until Thus, a clear discrete latent characterization is obtained. ;

[0260] Furthermore, such as Figure 8 As shown, the specific steps of the S7 decoder in converting the latent pixel space to obtain a high-fidelity remote sensing image include:

[0261] S7.1-1, Denoising the discrete latent representation Consider it as a containing A set of feature vectors, each vector having dimension . First, it is reshaped into a two-dimensional feature map for subsequent processing.

[0262] ;

[0263] in, It is the reorganized two-dimensional latent feature map.

[0264] S7.1-2, Use a linear projection layer to reduce the number of channels of the latent features. Initial number of channels required to map to the decoder The details are as follows:

[0265] ;

[0266] in, It is the projection weight matrix. It is a bias term. These are the projected features. These are the latent features of the input;

[0267] S7.1-3. Based on the pre-trained vector codebook, a nearest neighbor search operation is performed on image features, mapping continuous feature representations to corresponding discrete symbols in the codebook. This inverse mapping process realizes the transformation from distributed representation to structured representation, effectively improving the distribution consistency and numerical stability of the decoder's input features.

[0268] S7.1-4 The decoder adopts a multi-stage topology, expanding the spatial scale of the feature map step by step through a hierarchical forward propagation path. Each processing stage performs two core operations in sequence: resolution enhancement and feature refinement. First, the feature map size is expanded through spatial interpolation. Then, a nonlinear transformation module composed of multiple convolutional kernels and normalization units is connected, which significantly enhances the continuous representation of ground object edges and the ability to preserve the details of local textures, while effectively avoiding spectral aliasing and geometric distortion caused by discrete sampling.

[0269] Furthermore, high-fidelity remote sensing images are generated based on the input descriptive text and compared with the original images in the dataset. Figure 11 , Figure 12 As shown in Example 1, Figure 13 , Figure 14 This is Example 2.

[0270] Example 1:

[0271] Enter text: "Several business jets are parked on the apron beside ahangar, with a taxiway running along the east side."

[0272] The actual image primarily shows multiple business jets parked on the tarmac outside the hangar, with their noses pointing in different directions. A taxiway runs vertically to the right of the image, with visible yellow guide lines. The hangar building and its shadows are clearly visible, and the ground features concrete slabs of varying shades.

[0273] The model first analyzes the key objects in the text: "business jets", "apron", "hangar", "taxiway". The model understands the spatial relationship between these objects. Based on this semantic information, the image generation module generates a logical scene layout: the foreground is the tarmac and business jets, one side is the hangar building, and the background or side is a straight taxiway.

[0274] The generated image was compared with the real image. The generated image correctly contained all the key elements and was laid out reasonably, conforming to the constraints of the physical world and the text description.

[0275] Example 2:

[0276] Enter text: "Two large white cylindrical storage tanks at an industrial facility, with pipe racks and a service road nearby."

[0277] The actual image primarily features two large, white, circular storage tanks within an industrial facility. Small pipelines / valve structures are visible in the upper left corner. The tanks are surrounded by utility tunnels, service roads, and open space. A railway track / road is visible in the lower right corner. The ground textures are diverse (bare ground, grass, concrete), with shadows oriented uniformly.

[0278] This method first analyzes the key objects and attributes in the text: "white cylindrical storage tanks," "industrial facility," "pipe racks," and "service road," and models their spatial relationships: two large storage tanks are adjacent to each other, pipelines extend from the edges of the tanks and connect to each other, and service roads are arranged along one side or edge of the tanks. Based on this, the image generation module generates a logical scene layout with semantic priors: the main subject of the image is two white cylindrical storage tanks, which are connected to each other and to the outer edge by pipe racks, with a service road paved along the edge, and the surrounding area consists of industrial land materials and supporting structures.

[0279] The generated image was compared with the real image. The generated result correctly contained all the key elements: two white storage tanks, pipe corridor, and service road. The relative positions were reasonable and the scale matched. The overall result conformed to the physical constraints and textual description of the industrial scene. Compared with the real image, the generated image was more regular and simplified in terms of surface texture and small components, but it did not affect the accurate expression of the main semantics and structure.

Claims

1. A multi-modal semantic and physical law driven remote sensing image generation method, characterized in that, The method comprises the following steps: S1, quality screening and scale unification of the original remote sensing image dataset, removing low-quality remote sensing images with cloud cover, noise or distortion, and unifying all images to the target resolution through downsampling or super-resolution reconstruction, followed by pixel value normalization preprocessing operation; S2, using BLIP to automatically generate candidate text descriptions for the screened remote sensing images, and introducing spectral consistency and spatial distribution probability as physical constraints to optimize and screen the candidate texts, and obtaining high-quality semantic labels; S3, using a word segmenter to process the semantic labels, ensuring the integrity and accuracy of the professional terms of the ground objects; S4, using a pre-trained CLIP text encoder to convert the text into a high-dimensional semantic vector representation for cross-modal alignment; S5, encoding the processed remote sensing image into a latent space and embedding two-dimensional spatial location information, while introducing spectral and geometric physical constraints in the loss function to construct a joint latent representation that integrates semantic, geometric and radiometric characteristics; S6, performing a diffusion process on the joint latent representation: the forward process constructs training samples by gradually adding noise; the reverse denoising process uses a denoising network based on the Transformer architecture, with the text semantic vector, physical prior encoding and diffusion time step as joint conditions, to iteratively predict and remove noise and reconstruct the discrete latent features after denoising; S7, mapping the denoised discrete latent features back to the continuous space through inverse vector quantization, and reconstructing the final high-fidelity remote sensing image using an upsampling decoder.

2. The multi-modal semantic and physical law driven remote sensing image generation method according to claim 1, characterized in that, The specific steps in S1 are: S1.1-1, according to the typical quality degradation factors of remote sensing images, the following quantitative screening criteria are developed: Cloud cover: remove images with cloud cover area exceeding 10% of the total image area; Strip noise: identify and remove images with strip noise by calculating the gradient consistency of the image in the row or column direction; Geometric distortion: remove images with geometric distortion by checking the straightness of the image edge or verifying the control points if there is a geographic reference; Apply the above criteria to each image in the public original remote sensing image dataset and remove all images that do not meet the quality; S1.2-1, downsample an image having a resolution greater than a target resolution to have the same target resolution In detail, the method is as follows: ； ； wherein, is the pixel value of the original high resolution image in the local window, S is a scaling factor, is the number of pixels in the local window, x and y are the horizontal and vertical coordinate pixel indices on the target resolution image, is the resolution higher than the target resolution image, and are the starting coordinates on the high resolution image that x and y correspond to, respectively; S1.2-2, using ESRGAN network super-resolution reconstruction of the image with resolution less than the target resolution to make its resolution the same as the target resolution , improve the image resolution, and synthesize more abundant texture information to make up for the information loss of the low-resolution image; S1.3-1, linearly map the pixel value range of the image from the original range [0, 255] to the interval [-1, 1] to accelerate the convergence process of model training and improve numerical stability; the specific steps are as follows: ； wherein, is the normalized image, is the pixel value of each image after uniform resolution, , are the minimum and maximum pixel values of the image, respectively.

3. The multi-modal semantic and physical law driven remote sensing image generation method of claim 2, wherein, The specific steps in S2 include: S2.1-1, input the remote sensing image processed in S1 into the pre-trained BLIP model, the BLIP model extracts visual features through its image encoder, and generates multiple natural language descriptions related to the semantic content of the image using its fused text decoder, and the initial candidate text description set generated is ; S2.1-2, compute a physical score for each The score is a comprehensive measure of the consistency of the text and image at both semantic and physical levels. The physical score function is defined as follows: ； wherein text is text associated with the image to be evaluated a text description of the outfit, is a similarity score of the original CLIP model, is a weight of each physical constraint, is a satisfaction degree of the mth physical constraint, is a global weight of the semantic matching component, is a global weight of the physical constraint component, and M is the number of physical constraint items included in the evaluation. and introducing spectral consistency constraints and spatial distribution constraints The specific method is as follows: ； ； wherein, is the number of ground object types, is the standard spectral feature of the is the actual spectral feature of the corresponding region of the image, is the actual spectral feature of the corresponding region of the image, is the probability of the ground object appearing at the current location, DEM is a digital elevation model for providing elevation information of the terrain; S2.1-3, sort all candidate texts in descending order according to their physical scores; the top three texts with the highest scores enter the final screening pool for verification and modification; the verification criteria include: Physical accuracy: whether the description is consistent with the remote sensing imaging mechanism; Semantic fit: whether the description accurately and unambiguously reflects the main ground objects, spatial relationships and scene types in the image; Language fluency and application value: whether the description is smooth, whether it conforms to the field expression habits, and whether it has guiding significance for downstream tasks.

4. The multi-modal semantic and physical law driven remote sensing image generation method of claim 3, wherein, The specific steps in S3 include: S3.1-1, the preprocessing operation of the text entering the final screening pool specifically includes: removing all line breaks, tabs and special symbols other than sentence boundary punctuation in the text; compressing multiple spaces in the text into a single space; unifying the units of geographical scales or physical quantities in the text into a standard form; for complex long texts containing multiple independent sentences, using periods, semicolons and boundary punctuation to split them into independent short sentences, which helps the subsequent word segmentation and encoder to better capture the local semantic dependence; S3.1-2, performing professional remote sensing vocabulary segmentation on the preprocessed text segment, specifically including: splitting the input text into character units, and constructing an initial vocabulary table wherein is all unique characters; traversing the entire corpus, counting the co-occurrence frequency of all adjacent sub-word pairs, and in each iteration, selecting the pair with the highest frequency merge into new words , update the vocabulary table to , and loop S3.1-2 until the preset vocabulary size is reached or the number of mergers reaches the upper limit, and finally obtain a self-built sub-word vocabulary table learned through data-driven learning; match the updated self-built sub-word vocabulary table with the remote sensing professional vocabulary, and the sub-word follows the remote sensing professional vocabulary first, and for the words not appearing in the vocabulary, keep them in the self-built sub-word vocabulary table.

5. The multi-modal semantic and physical law driven remote sensing image generation method of claim 4, wherein, The specific steps in S4 are: S4.1-1, Defining a Trainable Embedding Matrix where is the vocabulary size, d is the embedding dimension, and for each subword Looking up the embedding vector ; meanwhile, to give the model sequential information, we introduce sine-cosine positional encoding as follows: ； ； where pos denotes the position of the word in the sequence, denotes the dimension index; finally, the embedding of the input sequence is denoted as: ； S4.2-1, add the subword embedding vector and the position encoding vector element by element to form the initial input of the Transformer encoder; the subword vector sequence after fusing the position information In the multi-layer Transformer structure input into the CLIP text encoder, in the self-attention mechanism of each layer, first project the input into the query vector Q, the key vector K and the value vector V space; specifically as follows: ； ； ； S4.2-2, based on the above projection, the attention score is calculated, and the scaled dot-product attention is used to stabilize the training; at the same time, the attention mask is added, and the padding symbol of the sequence is marked as True, and the other positions are reserved, so that each position can only pay attention to the previous and itself positions, preventing information leakage; finally, through softmax normalization, the attention weight is obtained, specifically: ； wherein is a mask matrix; S4.2-3, the attention output calculated independently for each head is extended to multi-head, and is mapped back to the dimension of the original model through a linear transformation, retaining the multi-view features, specifically as follows: ； where h is the number of attention heads, is the output of the th attention head, is a learnable linear layer; S4.2-4, the attention output is connected through residual connection and layer normalization, which can alleviate the gradient vanishing problem of deep network, specifically as follows: ； S4.2-5, to Through a feed-forward neural network, which consists of two linear transformations and a non-linear activation function to increase the non-linear capabilities of the model; also, residual connections and layer normalization are used; in detail as follows: ； S4.2-6, the Transformer module is stacked through multiple layers, which can capture the syntax structure in the shallow layer, model the inter-phrase relationship in the middle layer, and integrate the global semantic in the deep layer. Finally, the first position of the output of the last Transformer module or the average pooling of the output of all positions is taken as the global semantic representation of the whole text sequence: 。 6. The multi-modal semantic and physical law driven remote sensing image generation method of claim 5, wherein, The specific steps of S5 are: S5.1-1, input image Continuous latent variables are obtained by an encoder VQ-VAE The encoder VQ-VAE is composed of multiple groups of convolutional layers, residual blocks and down-sampling layers, for extracting high-level features of the image and reducing the spatial dimension; S5.2-1. In order to preserve the spatial structure information of the input image, prevent the position information loss caused by the translation invariance of the convolutional network, the continuous latent variable The two-dimensional position encoding is superimposed on the two-dimensional sine and cosine position encoding to preserve the spatial geometric structure, specifically as follows: ； ； adding the position encoding element-wise to the continuous latent variable, resulting in an enhanced latent variable as follows: ； wherein for spatial coordinates; S5.2-2, after adding position coding, vector quantization operation, look-up table replacement to the nearest neighbor codebook vector to obtain a discrete latent representation, in particular as follows: ； wherein is the quantized discrete code vector index; S5.3-1. In order to maintain the physical reality of remote sensing images, a spectral consistency constraint is added in the loss function of the encoder VQ-VAE training and a geometric consistency constraint to form a joint loss function , as follows: ； ； ； where C denotes the total number of bands of the image, is the pixel value of the generated image at band c, is the set of neighboring pixel pairs, is the MSE loss of the input and reconstructed image, is the commitment loss of the encoder VQ-VAE, , are the weight parameters of the MSE loss and the encoder VQ-VAE, respectively, , is the hyperparameter of the physical constraint; S5.3-2, in order to improve the stability and representation ability in the training process, the exponential moving average is used to dynamically update the codebook vector, which can improve the stability of the training, specifically as follows: ； ； wherein, is the code vector is the allocation count, is the number of times is allocated to is a smoothing coefficient.

7. The multi-modal semantic and physics rule driven remote sensing image generation method of claim 6, wherein, The diffusion steps of the diffusion model in S6 include forward diffusion and backward propagation, and the specific steps are: S6.1-1, initial input data fitting data set distribution for each time step independently sample noise from a standard Gaussian distribution, where the sampled noise satisfies and employ a set of fixed variance coefficients control noise intensity, where, increases incrementally as increases, such that the noise proportion gradually increases; this process is modeled as a Markov chain whose forward diffusion process satisfies the following equation: ； where the conditional transition distribution for each step is For any time instant transition distribution can be obtained directly from and ; S6.1-2, according to the fixed variance coefficient obtaining the reserved coefficient , so as to scale the original data, and ensure that the key features of the original data will not be completely submerged; S6.1-3, generate standard Gaussian noise with the same dimension as the original data, and control the noise amplitude through the noise coefficient; S6.1-4, the sample at the previous time instant with scaling and adding noise to obtain the noisy data at the current time instant ; as follows: ； wherein is a standard Gaussian noise; S6.1-5, repeat S6.1-1 to S6.1-4 until the real data is transformed from clear to pure noise state; S6.2-1, fuse the text semantics, physical priori and diffusion time step information into a unified conditional sequence as the guidance signal of the denoising network, splice the three to form a unified conditional input sequence, specifically as follows: ； wherein, is a sequence of text encoding vectors, is a vector obtained from the MLP for the physical prior, is a vector obtained from the MLP for the time step, S6.3-1, noisy latent tensor flatten to a vector and project to model dimension to get image vector and take it as query vector Q, take the unified conditional input sequence as an external retrievable memory bank, selectively retrieve the most relevant semantic prior and mechanism prior from it according to the current spatial position, and combine the diffusion time step context as key vector K and value vector V; to reduce the redundancy overhead, the key-value representation of the text and physical prior can be cached at the sequence level, only the time step component is updated with t, specifically: ； wherein, is a vector representation of the 0th layer, is a weight matrix of the input projection, is a bias broadcasted to all N vectors; first pre-normalize the residual structure, and inject diffusion time steps with physical priors in a channel-wise gated manner through conditional affine modulation; specifically, for the input of the i-th layer is conditionally modulated to obtain , and multi-head self-attention is performed to integrate spatial structure information and spectral correlation to form a robust context of the same modality; specifically as follows: ； ； ； in, It comes from physical a priori With time step joint mapping, Dimensional scaling; This process maintains the geometric-texture consistency and spectral domain structure, and provides noise-robust same-modal priori for subsequent cross-modal fusion; S6.3-2, given potential feature map , according to the predicted noise get the denoising limit map for this moment , as follows: ； ； ； wherein is a noise schedule, is a reservation coefficient at step t, is a cumulative reservation coefficient up to step t; S6.3-3、according to the above obtained , the mean value of the optimal back-off direction is calculated , the feature map of the next state is updated , as follows: ； ； wherein, is the sample standard deviation of this step, which guarantees convergence in the right direction and preserves natural diversity, is a standard Gaussian noise; S6.3-4, S6.1-1 to S6.3-3 are cycled from Back off A clear discrete latent representation is obtained .

8. The multi-modal semantic and physics rule driven remote sensing image generation method of claim 7, wherein, The specific steps of S7 to obtain high-fidelity remote sensing images include: S7.1-1, the de-noised discrete latent representation is considered as a set of feature vectors, each vector has a dimension of ; first, it is reshaped into a two-dimensional feature map for subsequent processing; ； wherein, is the two-dimensional latent feature map after the reformulation; S7.1-2. Use a linear projection layer to map the number of channels of latent features to the initial number of channels required by the decoder , as follows:​ ； wherein, is a projection weight matrix, is a bias term, is a projected feature, is an input latent feature; S7.1-3, based on the pre-trained vector codebook, perform neighbor search operation on the image features, and map the continuous feature representation to the corresponding discrete symbol in the codebook; this inverse mapping process realizes the conversion from distributed representation to structured representation, effectively improving the distribution consistency and numerical stability of the decoder input features; S7.1-4, the decoder adopts a multi-stage topology structure, and gradually expands the spatial scale of feature mapping through a hierarchical forward propagation path; each processing stage sequentially implements two core operations of resolution promotion and feature refinement: first, the feature map size is expanded through a spatial interpolation method, and then a nonlinear transformation module composed of multiple convolution kernels and normalization units is connected, which significantly enhances the continuity expression of feature edges and the detail preservation ability of local textures, while effectively avoiding the spectral aliasing phenomenon and geometric distortion caused by discrete sampling.

Images