Remote sensing image generation method based on federal visual language model

Abstract

The invention discloses a remote sensing image generation method based on a federal visual language model, which belongs to the technical field of machine learning and specifically comprises the following steps: receiving text instruction description by each client; extracting a multi-scale feature map from private remote sensing image data through a visual encoder, and generating a semantic embedding vector by text instruction description through a language encoder; inputting the semantic embedding vector and the multi-scale feature map into a dynamic attention mask generator to generate pixel-level space weight distribution; carrying out weighted fusion operation on the multi-scale feature map, and generating visual feature representation of text conditionalization; generating a remote sensing image according with the description of the text instruction through an image decoder; the client uploads model parameter increments of the visual encoder, the language encoder and the dynamic attention mask generator to the central server; the central server aggregates the model parameter increments, and distributes the updated global model parameters to each client; according to the method, the flexibility and semantic consistency of remote sensing image generation are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a remote sensing image generation method based on a federated visual language model. Background Art

[0002] With the accelerated digital transformation of fields like geographic information systems (GIS), environmental monitoring, and urban planning, the demand for rapid generation of high-precision remote sensing imagery is becoming increasingly urgent. Traditional remote sensing imagery relies primarily on hardware equipment such as satellites and drones for acquisition. Limited by weather conditions, geographical environments, acquisition costs, and timeframes, it struggles to meet the demands of real-time and diverse scenarios. The development of deep learning technology has opened up new avenues for remote sensing image generation. Models such as generative adversarial networks (GANs) and variational autoencoders (VAEs) can simulate and generate images of specific scenarios by learning from the characteristics of massive amounts of remote sensing data, significantly improving the efficiency and flexibility of data acquisition.

[0003] Existing remote sensing image generation methods have made considerable progress in the areas of multimodal fusion and distributed training. Some approaches employ conditional generation mechanisms, guiding image generation through input text descriptions or spatial parameters, achieving a preliminary integration of semantic and visual features. Within the framework of federated learning, researchers are attempting to integrate multiple data sources for distributed collaborative training, improving model generalization while protecting data privacy. Furthermore, some technologies optimize feature processing through methods such as angular transformation and geometric adjustment to enhance the spatial accuracy of generated images.

[0004] At the multimodal interaction level, existing solutions often rely on preset angle parameters or geometric transformations to control the generation process. For example, the FAGD framework constrains the image generation direction through fixed angle parameters. This framework lacks a deep understanding of natural language semantics and flexible adaptation, making it difficult to meet the diverse needs of complex remote sensing scenarios. In federated learning applications, the collaborative control method among clients is single, lacking a unified semantic-level command interface, resulting in inefficient collaborative generation of multi-source data. Furthermore, traditional parameter aggregation strategies often employ coarse-grained averaging methods, which are unable to refine the characteristic differences between natural and artificial elements in remote sensing data, easily leading to model overfitting or semantic bias. Summary of the Invention

[0005] The purpose of this invention is to provide a remote sensing image generation method based on a federated visual language model to solve the following technical problems:

[0006] Existing solutions mostly rely on preset angle parameters or geometric transformations to control the generation process, lack a deep understanding of natural language semantics and flexible adaptation, and traditional parameter aggregation strategies mostly use coarse-grained averaging methods, which are unable to perform fine-grained processing of the characteristic differences between natural and artificial elements in remote sensing data.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A remote sensing image generation method based on a federated visual language model is based on a federated architecture including a central server and multiple clients, and includes the following steps:

[0009] S1. Each client receives a text instruction description, wherein the text instruction description includes a semantic feature description and a spatial relationship description of a target remote sensing scene;

[0010] S2: The client's locally stored private remote sensing image data is used to extract multi-scale feature maps through a visual encoder, while the text instruction description is used to generate a semantic embedding vector through a language encoder;

[0011] S3. Input the semantic embedding vector and multi-scale feature map into the dynamic attention mask generator to generate a pixel-level spatial weight distribution that matches the text instruction description;

[0012] S4. Perform weighted fusion operations on multi-scale feature maps based on pixel-level spatial weight distribution to generate text-conditioned visual feature representations;

[0013] S5. Text-conditioned visual feature representation generates remote sensing images that conform to the text instruction description through the image decoder;

[0014] S6, the client only uploads the model parameter increments of the visual encoder, language encoder and dynamic attention mask generator to the central server;

[0015] S7. The central server aggregates the model parameter increments uploaded by each client and updates the global model parameters, and distributes the updated global model parameters to each client.

[0016] As a further solution of the present invention: in S1, the construction process of the text instruction description is:

[0017] Perform semantic parsing on the target remote sensing scene to extract the physical objects in the target remote sensing scene and the topological relationships between the physical objects; classify the physical objects into deformable object categories and rigid structure categories, where the deformable object category corresponds to vegetation elements or water elements, and the rigid structure category corresponds to building elements or road elements;

[0018] A dynamic attribute descriptor is assigned to the deformable object category, and the dynamic attribute descriptor includes a description of the morphological change range and the texture randomness parameter. A geometric constraint descriptor is assigned to the rigid structure category, and the geometric constraint descriptor includes a description of the contour accuracy requirements and the spatial positioning coordinates. A structured text sequence is generated based on the dynamic attribute descriptor and the geometric constraint descriptor, and the structured text sequence serves as the input data of the language encoder.

[0019] As a further solution of the present invention: in S3, the working process of the dynamic attention mask generator is:

[0020] The semantic embedding vector is decomposed into a set of object-level semantic components, each of which corresponds to a physical object in the text instruction description; at each scale level of the multi-scale feature map, the correlation score matrix between each pixel position and the set of object-level semantic components is calculated;

[0021] An initial attention heat map is generated according to the correlation score matrix, and the initial attention heat map is smoothed and optimized through a spatial continuity constraint function. A binary threshold segmentation operation is performed on the smoothed and optimized attention heat map to generate a binary mask containing the object activation area and the background suppression area. The binary mask is element-wise multiplied with the original multi-scale feature map to achieve selective enhancement and selective weakening of visual features.

[0022] As a further solution of the present invention: the process of performing smooth optimization operation on the initial attention heat map through the spatial continuity constraint function is:

[0023] A morphological closing operation is applied to the initial attention heat map to fill the gaps between adjacent high-response areas. An anisotropic diffusion equation is used to perform gradient-guided smoothing on the edge areas of the heat map, preserving sharp boundary features while eliminating discrete noise point features.

[0024] The topological relationship matrix between entity objects is introduced as a regularization constraint to ensure that the mutually related entity objects maintain spatial consistency in attention distribution; the heat map after spatial optimization is normalized to make the numerical range of the heat map adapt to the requirements of the subsequent binary threshold segmentation operation.

[0025] As a further solution of the present invention: in S4, the process of generating the text-conditioned visual feature representation is:

[0026] The feature map after the weighted fusion operation is compressed in the channel dimension to generate a low-dimensional dense feature vector. The semantic embedding vector and the low-dimensional dense feature vector are concatenated to form a mixed feature vector, which is input into the multi-layer perceptron to perform cross-modal feature alignment. The aligned mixed feature vector is split into a content encoding vector and a style encoding vector.

[0027] The content coding vector is input into the residual block sequence of the image decoder to perform scene structure reconstruction operations; the style coding vector is injected into the residual block sequence through the adaptive instance normalization layer to perform texture detail generation control operations.

[0028] As a further solution of the present invention, the control operation of the style coding vector is specifically as follows:

[0029] Parse the dynamic attribute descriptors in the text instruction description to extract texture complexity index data and spectral variation coefficient data; map the texture complexity index data to the amplitude modulation coefficient of the style coding vector; and map the spectral variation coefficient data to the phase rotation angle of the style coding vector;

[0030] In the adaptive instance normalization layer, the variance component of the feature map is scaled by the amplitude modulation coefficient; the mean component of the feature map is offset by the phase rotation angle to achieve the joint regulation operation of texture attributes and spectral attributes.

[0031] As a further solution of the present invention: In S5, the specific process of the image decoder generating the remote sensing image that conforms to the text instruction description is as follows:

[0032] A dual-branch decoding architecture is constructed, which includes a coarse-grained generation path and a fine-grained generation path. The coarse-grained generation path generates low-resolution image skeleton data, which includes the main features of the ground object outline and position relationship. The fine-grained generation path receives the low-resolution image skeleton data as input and performs resolution enhancement through a sub-pixel convolution layer.

[0033] During the resolution enhancement process, the style encoding vector is injected into the convolution kernel weight data of different levels in stages; the outputs of the coarse-grained generation path and the fine-grained generation path are subjected to a pixel-level weighted fusion operation, and the fusion operation weight is determined by the detail emphasis level data in the text instruction description.

[0034] As a further solution of the present invention: S5 further includes verifying the generated remote sensing image, the process is as follows:

[0035] The generated remote sensing image is input into a pre-trained visual description model to automatically generate scene description text data; the semantic similarity score between the automatically generated scene description text data and the original text instruction description is calculated; when the semantic similarity score is lower than a preset threshold, the parameter deviation data of the language encoder and the dynamic attention mask generator are located;

[0036] Adversarial text instruction descriptions are generated based on parameter deviation data and added to the training dataset as difficult samples; difficult samples are used to perform enhanced training operations on the language encoder and dynamic attention mask generator.

[0037] As a further solution of the present invention: in S7, the process of the central server aggregating the model parameter increments uploaded by each client is as follows:

[0038] The central server identifies the dynamic attention mask component parameters in the model parameter increments uploaded by each client; performs object category clustering on the dynamic attention mask component parameters, dividing them into a set of natural element-related parameters and a set of artificial element-related parameters; and calculates the centroid vectors of the natural element-related parameter set and the artificial element-related parameter set respectively;

[0039] The centroid vector is matched with the original parameters of each client using cosine similarity matching calculation, and the client parameters with similarity values ​​higher than the preset threshold are filtered to form a filtered parameter subset; the filtered parameter subset is weightedly fused according to the proportion of client data volume to generate the global model update parameters.

[0040] Beneficial effects of the present invention:

[0041] This invention effectively addresses the limitations of traditional remote sensing image generation technology through a federated visual language fusion architecture and dynamic attention mask technology. To address the problems of existing methods relying on preset parameters and lacking semantic understanding, this solution pioneered the use of text instructions as a unified cross-client control interface, utilizing a language encoder to convert natural language into semantic constraints. This frees each client from fixed parameter restrictions and enables collaborative generation of remote sensing images based on complex text instructions, significantly improving the flexibility and adaptability of multimodal interaction. In a federated learning scenario, semantic-level collaborative control is achieved through text instructions, avoiding the inefficiency caused by traditional single collaborative methods.

[0042] In terms of feature processing and parameter aggregation, this paper uses dynamic attention masking technology to achieve pixel-level feature reconstruction through binary spatial masks. It precisely controls the spatial weight distribution of feature maps based on text instructions. Compared with traditional geometric transformation methods, it can more finely process the semantics and spatial relationships of different objects in remote sensing scenes. At the same time, when aggregating parameters, the central server clusters the parameters of the dynamic attention mask components into object categories, distinguishes parameters related to natural and artificial elements, and performs weighted fusion based on data features. This overcomes the model overfitting and semantic bias problems caused by traditional coarse-grained averaging strategies, ensures consistency in object semantics and spatial distribution of images generated by multiple clients, and significantly improves the accuracy and authenticity of remote sensing image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below with reference to the accompanying drawings.

[0044] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] See also Figure 1 As shown, the present invention is a remote sensing image generation method based on a federated visual language model, based on a federated architecture including a central server and multiple clients, comprising the following steps:

[0047] S1. In the execution process, each client first receives a text instruction containing the semantic features and spatial relationship description of the target remote sensing scene. The instruction content covers key information such as entity object category, topological relationship, morphological attributes, etc., such as "three high-rise buildings in the city center are intertwined with a ring road, and green spaces and rivers are distributed around them."

[0048] S2. The client then uses a visual encoder with a hierarchical convolutional architecture based on locally stored private remote sensing image data to extract multi-scale feature maps containing low-level texture details, mid-level structural features, and high-level semantic information. Simultaneously, a language encoder performs word segmentation and semantic parsing on the text instructions, generating an embedding vector containing scene semantic information through a Transformer architecture, thus transforming text semantics into a feature space.

[0049] S3. During the feature fusion stage, the dynamic attention mask generator decomposes the semantic embedding vector into object-level semantic components. At different levels of the multi-scale feature map, the correlation score between each pixel position and the semantic component is calculated using cosine similarity to form an initial attention heatmap. Next, the heatmap is smoothed and optimized using morphological closing operations and the anisotropic diffusion equation. Entity topology is used as a regularization constraint to ensure the spatial consistency of attention distribution. Binarization is then performed to generate a pixel-level spatial weight distribution mask, achieving selective enhancement and suppression of visual features.

[0050] S4. Based on the above mask, the system performs a weighted fusion of multi-scale feature maps, compresses the channel dimensions, and concatenates them with the semantic embedding vector. Cross-modal feature alignment is achieved through a multi-layer perceptron. The separated content encoding vector and style encoding vector are used to reconstruct scene structure and generate texture details, respectively. The style encoding vector uses the dynamic attribute description in the text instructions to adjust the variance and mean of the feature map through an adaptive instance normalization layer, achieving precise control of texture and spectral properties.

[0051] S5. The image decoder adopts a dual-branch architecture. The coarse-grained path generates a low-resolution image skeleton, and the fine-grained path improves the resolution based on the skeleton through sub-pixel convolution. The style encoding vector is gradually injected in the process. Finally, the dual-path output is fused according to the detailed requirements of the text instruction to generate a remote sensing image that meets the semantic description.

[0052] S6. The client only uploads the model parameter increments of the visual encoder, language encoder, and dynamic attention mask generator to the central server.

[0053] S7. The central server clusters, filters, and weightedly fuses the parameters of the dynamic attention mask components, updates the global model, and distributes it to the client, achieving collaborative optimization under the federated learning framework and ensuring high precision and semantic consistency of the generated images.

[0054] In a preferred embodiment of the present invention, in S1, the construction process described by the text instruction is:

[0055] The construction process of textual instruction descriptions utilizes refined semantic parsing and structured encoding strategies to achieve precise conversion from natural language to machine-understandable instructions. The system first performs deep semantic analysis of the target remote sensing scene, applying named entity recognition (NER) and relationship extraction algorithms, combined with a pre-trained remote sensing domain language model, to accurately extract the scene's entity objects and their topological relationships. During the entity recognition phase, the model employs a multi-label classifier to classify common entities in remote sensing scenes, including but not limited to buildings (residential, commercial, industrial facilities), roads (freeways, urban roads, rural trails), vegetation (forests, grasslands, crops), water bodies (rivers, lakes, reservoirs), and unique landforms (bridges, substations, sports fields). Furthermore, through dependency parsing and spatial relationship reasoning, the system analyzes topological relationships between entities, such as spatial associations like "the building is located east of the road" and "the river runs through the city park," thereby constructing a semantic network diagram of the scene.

[0056] Based on this, the system divides the extracted entities into two categories: deformable objects and rigid structures. This classification strategy fully considers the physical characteristics of different features in remote sensing scenes: the deformable object category includes vegetation and water features whose forms are easily affected by environmental factors (such as wind, water levels, and seasonal changes); the rigid structure category includes artificial facilities with fixed geometric forms, such as buildings and roads. For different entity categories, the system adopts a differentiated attribute description strategy to achieve refined modeling of remote sensing scenes.

[0057] The dynamic attribute descriptors assigned to deformable object categories include descriptions of the range of morphological variation and texture randomness parameters. Regarding the range of morphological variation, the system establishes dynamic change models for different types of vegetation and water bodies through statistical analysis of historical remote sensing data. For example, for forest vegetation, descriptive parameters include canopy cover (e.g., 60%-85%) and leaf area index fluctuation range (2.5-4.0); for rivers, descriptive parameters include width variation range (50m in the dry season to 150m in the wet season) and curvature coefficient (1.2-1.8). Texture randomness parameters quantify the uncertainty of the surface features of deformable objects, such as the vegetation texture roughness index (0.3-0.7) and the water surface reflectivity fluctuation range (0.05-0.2). These parameters are probabilistically distributed using a Gaussian mixture model.

[0058] The geometric constraint descriptor configured for the rigid structure category includes a description of the required outline accuracy and spatial positioning coordinates. The outline accuracy requirement sets different accuracy thresholds based on the building or road's purpose and importance. For example, the outline accuracy requirement for high-rise buildings is ±0.5m, while for ordinary residential buildings it is ±1.0m; the width error threshold for urban arterial roads is ±0.3m, while for rural roads it is ±1.0m. Spatial positioning coordinates are expressed in absolute coordinates (WGS84) or relative coordinates (a local coordinate system with the scene center as the origin), along with confidence intervals to indicate positioning uncertainty, such as "Building A (116.3942±0.0005, 39.9018±0.0005)." For rigid structures with complex geometries, the system also supplements the coordinates of key control points and geometric transformation parameters, such as rotation angle and scaling.

[0059] Ultimately, the system integrates dynamic attribute descriptors and geometric constraint descriptors into a structured text sequence. This sequence uses a hierarchical nested format, with the entity object at its core, and sequentially contains category labels, attribute parameters, and topological relationship descriptions. For example, "[Building A (rigid structure, coordinates: 116.3942, 39.9018; contour accuracy: ±0.5m), Road B (rigid structure, width: 15±0.3m; curvature: 0.02±0.005), Vegetation C (deformable object, cover density: 75±5%; texture roughness: 0.45±0.05), topological relationship: [Building A is located 50m north of Road B, and Vegetation C surrounds Building A]]." This structured text sequence serves as standard input data for a language encoder, preserving the semantic richness of natural language while also possessing a machine-parseable formal structure.

[0060] In another preferred embodiment of the present invention, in S3, the working process of the dynamic attention mask generator is:

[0061] First, the semantic embedding vector output by the language encoder is decomposed into a set of object-level semantic components through a self-attention mechanism, with each component corresponding to an entity object in the text instruction. To enhance the granularity of the semantic representation, the system employs a multi-head attention mechanism, projecting the semantic information of each entity object into multiple subspaces to form a multidimensional semantic representation. For example, for the "building" entity, the semantic components can be decomposed into multiple dimensions, such as "shape features," "material features," and "functional features," each of which is processed by a separate attention head.

[0062] The system adopts a pyramid feature matching strategy on the multi-scale feature maps output by the visual encoder. The visual encoder extracts feature maps of different scales through a residual network or Transformer architecture, including low-resolution global semantic features (such as scene type), medium-resolution structural features (such as building layout), and high-resolution detail features (such as road texture). At each scale level, the system calculates the correlation score matrix between each pixel position and the set of object-level semantic components. In specific implementation, cosine similarity or dot product operations are used to measure the similarity between the feature vector and the semantic vector, and the score is normalized into a probability distribution through the Softmax function to form a confidence heat map reflecting that each pixel belongs to a specific entity object.

[0063] In a preferred embodiment of the present invention, the process of performing a smooth optimization operation on the initial attention heat map using the spatial continuity constraint function is as follows:

[0064] First, a morphological closing operation is applied to fill the gaps between high-response areas in the heat map through dilation and erosion operations, eliminating the problem of object region fragmentation caused by discontinuous feature extraction. The dilation operation uses an adaptive structural element, whose size is dynamically adjusted according to the scale of the currently processed features to ensure effective connection while preserving details. The heat map is then smoothed using the anisotropic diffusion equation. This equation is guided by the gradient direction of the heat map, retaining details in areas with large gradients (such as object edges) and smoothing in areas with small gradients (such as the interior of the object), effectively suppressing noise while preserving the sharpness of object edges.

[0065] To ensure the accuracy of the topological relationship between entities, the system introduces a topological relationship matrix as a regularization constraint. This matrix is ​​constructed based on the topological relationship between entities parsed in the text instructions, and defines the constraints on the spatial distribution of different entities. For example, if the text instruction describes "a river passing through a city park", when generating the attention mask, the system will force the river area to maintain spatial connectivity with the park area. In specific implementation, the topological relationship is converted into an energy function, and the attention distribution is aligned by minimizing the energy function. For example, for adjacent buildings and roads, the system will constrain the attention scores of the boundary areas of the two to maintain continuity to avoid unreasonable spatial gaps.

[0066] Finally, the heatmap is normalized to [0, 1], and binary segmentation is performed using the Otsu algorithm or adaptive thresholding to generate a binary mask containing the object activation area and the background suppression area. To further optimize the mask quality, the system introduces a multi-scale fusion mechanism to weightedly fuse the masks generated at different levels, with low-level masks providing detail information and high-level masks providing semantic constraints. The resulting binary mask is element-wise multiplied with the original feature map to enhance the target object features and weaken the background features. This pixel-level feature selection mechanism enables the subsequent image generation process to focus on key areas related to text instructions, significantly improving the semantic accuracy and detail quality of the generated image.

[0067] In another preferred embodiment of the present invention, the process of generating the text-conditioned visual feature representation in S4 is:

[0068] First, the system uses global average pooling combined with 1×1 convolution to compress the channel dimension of the weighted fused feature map output by the dynamic attention mask generator. Global average pooling calculates the average value of each feature map channel, compressing spatial dimension information into channel-dimensional statistics to reduce data redundancy. The 1×1 convolution layer further transforms the high-dimensional feature map into a low-dimensional dense feature vector by learning linear combinations between channels, thus preserving key feature information while reducing computational complexity.

[0069] The system then concatenates the semantic embedding vector generated by the language encoder with the compressed low-dimensional dense feature vector to form a hybrid feature vector containing both visual and semantic information. To effectively align cross-modal features, the hybrid feature vector is input into a multi-layer perceptron (MLP) for nonlinear transformation. The MLP learns the mapping relationship between features of different modalities through multiple fully connected layers. Using activation functions (such as ReLU), it introduces nonlinearity, enabling semantic association and numerical matching of semantic and visual features in the same feature space, eliminating the inconsistent feature distribution caused by modal differences.

[0070] The aligned mixed feature vector undergoes a dimension splitting operation, separating it into a content encoding vector and a style encoding vector. The content encoding vector primarily carries scene structural information, such as the geometric outlines of buildings and the topological orientation of roads, and serves as the basic input for the image decoder to reconstruct the scene structure. The style encoding vector is responsible for controlling the image's texture details and spectral properties. It is injected into the residual block sequence of the image decoder through the adaptive instance normalization (AdaIN) layer, achieving fine-grained control of the visual style of the generated image.

[0071] In a preferred embodiment of this invention, the control operation of the style coding vector is specifically as follows:

[0072] The dynamic attribute descriptors in the text instruction description are deeply analyzed to extract data on texture complexity index and spectral variation coefficient. Texture complexity index is quantified by calculating characteristics such as density, directionality, and repeatability of texture elements. For example, the complexity of vegetation texture can be evaluated based on the density of leaf distribution and the regularity of leaf vein orientation. The spectral variation coefficient is determined by analyzing the fluctuation of the spectral reflectance curve of the ground object and the difference in the response of the characteristic band. For example, the reflectance of water in the near-infrared band is significantly lower than that of vegetation, and its spectral variation coefficient has a unique distribution.

[0073] The system maps the extracted texture complexity index data to the amplitude modulation coefficient of the style encoding vector, and maps the spectral variation coefficient data to the phase rotation angle. This mapping relationship is achieved through a pre-trained regression model, which learns the correspondence between texture, spectral properties, and encoding vector parameters on a large amount of labeled data. In the adaptive instance normalization layer, the amplitude modulation coefficient is used to scale the variance component of the feature map to control the richness of texture details. Larger amplitude coefficients can enhance the detail variation of textures, making the generated vegetation leaf texture more delicate and the building wall material more realistic. The phase rotation angle is used to offset the mean component of the feature map and adjust the spectral properties. For example, it can make the water body appear more in line with the blue-green hue described by the instruction, or make the green saturation of vegetation leaves match the spectral reflectance.

[0074] Through the combined operation of amplitude modulation and phase rotation, the style coding vector achieves coordinated regulation of texture and spectral properties. In the residual block sequence of the image decoder, the AdaIN layer normalizes the input feature map channel by channel and adjusts the normalized mean and variance according to the style coding vector. This mechanism ensures that the texture details and spectral characteristics of each pixel in the image generation process accurately match the requirements of the text instructions, effectively avoiding problems such as texture blurring and spectral distortion in traditional methods. Combined with the stable reconstruction of the scene structure by the content coding vector, the resulting remote sensing image achieves a high degree of consistency in structural accuracy, texture authenticity, and spectral consistency, meeting the high-precision requirements for complex remote sensing scene generation.

[0075] In another preferred embodiment of the present invention, in S5, the specific process of the image decoder generating the remote sensing image that conforms to the text instruction description is as follows:

[0076] The image decoder utilizes an innovative dual-branch decoding architecture and dynamic fusion strategy to generate high-precision remote sensing imagery consistent with textual descriptions. This dual-branch decoding architecture consists of a coarse-grained generation path and a fine-grained generation path, which work together to balance the stability of scene structure with the authenticity of detail. The coarse-grained generation path utilizes a lightweight convolutional neural network, taking the content encoding vector from the text-conditioned visual feature representation as input. Through a series of downsampling and upsampling operations, it generates low-resolution image skeleton data. This skeleton data contains the outline features and positional relationships of key features, such as the general shape of buildings and the layout of roads, providing the underlying structural framework for the image.

[0077] The fine-grained generation path focuses on the refinement of image details. After receiving the low-resolution image skeleton data output by the coarse-grained generation path, it uses the sub-pixel convolution layer to gradually improve the image resolution. The sub-pixel convolution layer effectively increases the number of image pixels without introducing additional parameters by rearranging the pixel values ​​in the feature map, thus avoiding the checkerboard effect that may be produced by traditional deconvolution operations. During the resolution enhancement process, the style coding vector is injected into the convolution kernel weight data of different levels in stages. Specifically, the style coding vector is injected into the shallow convolution layer to preliminarily adjust the overall texture tone and spectral characteristics of the image; as the network layer deepens, the style coding vector is injected again into the convolution layer close to the output layer to fine-tune the image details to ensure that the texture details and spectral properties of the generated image accurately match the text instruction requirements.

[0078] After completing image generation for both paths, the system performs a pixel-level weighted fusion operation on the outputs of the coarse-grained and fine-grained generation paths. The weights of this fusion operation are determined by the detail emphasis level data in the textual instruction description. This data is set based on the requirements of different application scenarios. For example, urban planning scenarios focus more on building details and have a higher detail emphasis level; while large-scale regional monitoring scenarios focus more on the overall layout and have a lower detail emphasis level. By adaptively adjusting the fusion weights, the system can flexibly control the proportion of structure and detail in the generated image, generating remote sensing images that meet the needs of diverse applications.

[0079] In a preferred embodiment of the present invention, the step S5 further includes verifying the generated remote sensing image, and the process is as follows:

[0080] First, the generated remote sensing image is fed into a pre-trained visual description model. This model, based on the Transformer architecture, automatically extracts visual features from the image and generates textual descriptions of the scene. To calculate the semantic similarity between the generated text and the original textual instructions, the system uses a cosine similarity algorithm based on word vectors, combined with pre-trained language models such as BERT, to obtain the contextual semantic representation of the text. When the semantic similarity score falls below a preset threshold, it indicates a semantic deviation between the generated image and the textual instructions. At this point, the system uses gradient backpropagation to locate parameter deviations between the language encoder and the dynamic attention mask generator, identifying the key parameters that cause the semantic inconsistency.

[0081] In response to the identified parameter deviations, the system generates adversarial textual descriptions of instructions. These adversarial instructions contain challenging semantic scenarios and complex spatial relationships and are added to the training dataset as difficult examples. By using these difficult examples to perform enhanced training on the language encoder and dynamic attention mask generator, the model learns to handle complex semantics and feature alignment tasks, effectively improving image generation capabilities in challenging scenarios and forming a complete closed loop from image generation and verification to model optimization.

[0082] In another preferred embodiment of the present invention, in S7, the process of the central server aggregating the model parameter increments uploaded by each client is as follows:

[0083] The central server adopts an innovative hierarchical clustering and weighted fusion strategy to improve the optimization efficiency and generalization ability of the global model. The central server first identifies the dynamic attention mask component parameters in the model parameter increments uploaded by each client. This component is the key to achieving precise control of visual features by text instructions. Subsequently, based on the response characteristics of the parameters to different object categories, the system performs object category clustering operations on the dynamic attention mask component parameters, dividing them into a set of parameters related to natural elements (such as vegetation and water bodies) and a set of parameters related to artificial elements (such as buildings and roads). By calculating the centroid vectors of the two sets respectively, the core feature representation of each type of parameter is obtained.

[0084] Next, the system performs a cosine similarity matching calculation on the centroid vector and the original parameters of each client, and selects the client parameters with similarity values ​​higher than the preset threshold to form a subset of filtered parameters. This screening mechanism ensures that the aggregated parameters retain both common characteristics and the uniqueness of the client data. Finally, a weighted fusion operation is performed on the subset of filtered parameters according to the proportion of the client data volume. The client parameters with a larger data volume account for a higher proportion in the fusion process, so that the global model update parameters can better reflect the overall data distribution characteristics. This parameter aggregation strategy of hierarchical clustering and weighted fusion avoids the problems of model overfitting and insufficient generalization ability that may be caused by traditional average aggregation methods. While protecting the privacy of client data, it effectively improves the collaborative optimization effect and generation quality of the remote sensing image generation model under the federated learning framework.

[0085] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A remote sensing image generation method based on a federated visual language model, based on a federated architecture comprising a central server and multiple clients, characterized in that: The following steps are involved: S1. Each client receives a text instruction description, wherein the text instruction description includes a semantic feature description and a spatial relationship description of a target remote sensing scene; S2: The client's locally stored private remote sensing image data is used to extract multi-scale feature maps through a visual encoder, while the text instruction description is used to generate a semantic embedding vector through a language encoder; S3. Input the semantic embedding vector and multi-scale feature map into the dynamic attention mask generator to generate a pixel-level spatial weight distribution that matches the text instruction description; S4. Perform weighted fusion operations on multi-scale feature maps based on pixel-level spatial weight distribution to generate text-conditioned visual feature representations; S5. Text-conditioned visual feature representation generates remote sensing images that conform to the text instruction description through the image decoder; S6, the client only uploads the model parameter increments of the visual encoder, language encoder and dynamic attention mask generator to the central server; S7. The central server aggregates the model parameter increments uploaded by each client and updates the global model parameters, and distributes the updated global model parameters to each client.

2. The remote sensing image generation method based on the federated visual language model according to claim 1, characterized in that: In S1, the construction process described by the text instruction is: Perform semantic parsing on the target remote sensing scene to extract the physical objects in the target remote sensing scene and the topological relationships between the physical objects; classify the physical objects into deformable object categories and rigid structure categories, where the deformable object category corresponds to vegetation elements or water elements, and the rigid structure category corresponds to building elements or road elements; A dynamic attribute descriptor is assigned to the deformable object category, and the dynamic attribute descriptor includes a description of the morphological change range and the texture randomness parameter. A geometric constraint descriptor is assigned to the rigid structure category, and the geometric constraint descriptor includes a description of the contour accuracy requirements and the spatial positioning coordinates. A structured text sequence is generated based on the dynamic attribute descriptor and the geometric constraint descriptor, and the structured text sequence serves as the input data of the language encoder.

3. The remote sensing image generation method based on the federated visual language model according to claim 1, characterized in that: In S3, the working process of the dynamic attention mask generator is as follows: The semantic embedding vector is decomposed into a set of object-level semantic components, each of which corresponds to a physical object in the text instruction description; at each scale level of the multi-scale feature map, the correlation score matrix between each pixel position and the set of object-level semantic components is calculated; Generate an initial attention heat map based on the correlation score matrix, and perform a smooth optimization operation on the initial attention heat map through a spatial continuity constraint function; perform a binary threshold segmentation operation on the smoothed and optimized attention heat map to generate a binary mask containing the object activation area and the background suppression area; The binary mask is element-wise multiplied with the original multi-scale feature map to achieve selective enhancement or weakening of visual features.

4. The remote sensing image generation method based on the federated visual language model according to claim 3 is characterized in that: The process of smoothing and optimizing the initial attention heat map through the spatial continuity constraint function is as follows: A morphological closing operation is applied to the initial attention heat map to fill the gaps between adjacent high-response areas. An anisotropic diffusion equation is used to perform gradient-guided smoothing on the edge areas of the heat map, preserving sharp boundary features while eliminating discrete noise point features. The topological relationship matrix between entity objects is introduced as a regularization constraint to ensure that the mutually related entity objects maintain spatial consistency in attention distribution; the heat map after spatial optimization is normalized to make the numerical range of the heat map adapt to the requirements of the subsequent binary threshold segmentation operation.

5. The remote sensing image generation method based on the federated visual language model according to claim 1, characterized in that: In S4, the process of generating text-conditioned visual feature representation is as follows: The feature map after the weighted fusion operation is compressed in the channel dimension to generate a low-dimensional dense feature vector. The semantic embedding vector and the low-dimensional dense feature vector are concatenated to form a mixed feature vector, which is input into the multi-layer perceptron to perform cross-modal feature alignment. The aligned mixed feature vector is split into a content encoding vector and a style encoding vector. The content coding vector is input into the residual block sequence of the image decoder to perform scene structure reconstruction operations; the style coding vector is injected into the residual block sequence through the adaptive instance normalization layer to perform texture detail generation control operations.

6. The remote sensing image generation method based on the federated visual language model according to claim 5, characterized in that: The control operations of the style encoding vector are as follows: Parse the dynamic attribute descriptors in the text instruction description to extract texture complexity index data and spectral variation coefficient data; map the texture complexity index data into the amplitude modulation coefficient of the style coding vector; Mapping the spectral variation coefficient data to the phase rotation angle of the style encoding vector; In the adaptive instance normalization layer, the variance component of the feature map is scaled by the amplitude modulation coefficient; the mean component of the feature map is offset by the phase rotation angle to achieve the joint regulation operation of texture attributes and spectral attributes.

7. The remote sensing image generation method based on the federated visual language model according to claim 5, characterized in that: In S5, the specific process of the image decoder generating the remote sensing image that conforms to the text instruction description is as follows: A dual-branch decoding architecture is constructed, which includes a coarse-grained generation path and a fine-grained generation path. The coarse-grained generation path generates low-resolution image skeleton data, which includes the main features of the ground object outline and position relationship. The fine-grained generation path receives the low-resolution image skeleton data as input and performs resolution enhancement through a sub-pixel convolution layer. During the resolution enhancement process, the style encoding vector is injected into the convolution kernel weight data of different levels in stages; the outputs of the coarse-grained generation path and the fine-grained generation path are subjected to a pixel-level weighted fusion operation, and the fusion operation weight is determined by the detail emphasis level data in the text instruction description.

8. The remote sensing image generation method based on the federated visual language model according to claim 7, characterized in that: The S5 also includes verifying the generated remote sensing image, and the process is as follows: The generated remote sensing image is input into a pre-trained visual description model to automatically generate scene description text data; the semantic similarity score between the automatically generated scene description text data and the original text instruction description is calculated; when the semantic similarity score is lower than a preset threshold, the parameter deviation data of the language encoder and the dynamic attention mask generator are located; Adversarial text instruction descriptions are generated based on parameter deviation data and added to the training dataset as difficult samples; difficult samples are used to perform enhanced training operations on the language encoder and dynamic attention mask generator.

9. The remote sensing image generation method based on the federated visual language model according to claim 1, characterized in that: In S7, the process of the central server aggregating the model parameter increments uploaded by each client is as follows: The central server identifies the dynamic attention mask component parameters in the model parameter increment uploaded by each client; Performing object category clustering on the dynamic attention mask component parameters, dividing them into a natural element related parameter set and an artificial element related parameter set; calculating the centroid vectors of the natural element related parameter set and the artificial element related parameter set respectively; The centroid vector is matched with the original parameters of each client using cosine similarity matching calculation, and the client parameters with similarity values ​​higher than the preset threshold are filtered to form a filtered parameter subset; the filtered parameter subset is weightedly fused according to the proportion of client data volume to generate the global model update parameters.

Images