A controllable remote sensing image generation method based on a scene graph and related devices

Abstract

The application provides a controllable remote sensing image generation method based on a scene graph and related devices, and belongs to the technical field of controllable remote sensing image generation.The semantic information of nodes and the semantic information of directed edges and global text description are encoded by using relationship flow to obtain node relationship features and global text embedding;based on the node relationship features, visual alignment node features are obtained, and a region decoupling mask attention mechanism is constructed based on the visual alignment node features;using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on an input scene graph;using the background region mask and the object region mask, an output feature map after region decoupling is obtained;the output feature map after region decoupling, the global text embedding and a geometric condition matrix are jointly input into a diffusion model for diffusion generation to obtain a controllable remote sensing image.The application solves the problem of low accuracy of controllable remote sensing image generation.

Description

Technical Field

[0001] This invention belongs to the field of controllable remote sensing image generation technology, specifically relating to a controllable remote sensing image generation method and related apparatus based on scene graphs. Background Technology

[0002] Remote sensing image generation is a crucial technology in remote sensing data augmentation, remote sensing scene simulation, remote sensing target detection-assisted training, and geospatial intelligent interpretation. Remote sensing data acquisition typically relies on satellite, airborne, or UAV platforms, and is often limited by weather, time, location, sensor conditions, and annotation costs, resulting in a shortage of high-quality samples for specific categories, layouts, or complex scenes. Generative models can synthesize images with target annotations and scene semantics based on given conditions, thus alleviating the data scarcity problem in remote sensing supervised learning tasks. In recent years, diffusion models have demonstrated strong high-fidelity image synthesis capabilities in both natural and remote sensing image generation, providing a new technological foundation for controllable remote sensing image generation.

[0003] Controllable generation of remote sensing images differs from the generation of general natural images. First, remote sensing images are mostly from a top-down perspective, and targets can be distributed in any direction. Therefore, using only horizontal bounding boxes or coarse-grained layouts is insufficient to fully express the geometric attributes of the targets. Second, there is a strong semantic coupling between targets and the background in remote sensing scenes. For example, ships are usually associated with ports, berths, or rivers; airplanes are usually associated with runways, airports, or hangar areas; and bridges usually cross rivers or roads. Third, the practical value of remote sensing images depends not only on the clarity of individual targets but also on whether the number, location, orientation, background type, and logical relationships of targets are all valid. Therefore, controllable remote sensing image generation needs to simultaneously meet the requirements of controllable spatial geometry, consistent semantic relationships, and reasonable background context.

[0004] Currently available remote sensing image generation methods include the following categories: 1. Remote sensing generation method based on text to image This type of method takes natural language descriptions as conditional input, obtains global semantic conditions through a text encoder, and then guides a diffusion model or other generative model to synthesize images. Its advantages include simple input, flexible expression, and the ability to describe overall scene categories and a small number of object relationships.

[0005] The problems with this type of method are that text prompts are difficult to clearly constrain the precise position, rotation direction and spatial topology of each target, especially in multi-target remote sensing scenarios, which can easily lead to errors in the number of targets, positional deviations and ambiguous relationships.

[0006] 2. Remote sensing generation method based on layout to image This type of method uses category labels and bounding boxes as conditions, and can explicitly specify the location of the foreground target, thus outperforming plain text methods in target spatial localization.

[0007] The problem with this type of method is that the bounding box itself only expresses geometric occupancy, not the semantic and functional relationships between targets, and cannot fully control the background context. Therefore, the model may generate remote sensing scenes that meet the layout requirements geometrically but are semantically unreasonable.

[0008] 3. A general generation method based on scene graph to image This type of method uses a graph structure to represent object nodes and the edges between objects. Typically, the scene graph is encoded into conditional features using a graph neural network before being input into the generator. While this type of method can describe object relationships, it tends to aggregate the graph representation into a global conditional vector, which weakens the connection between a single node and its corresponding region in the image, making it difficult to guarantee node-level spatial control and region-level semantic alignment.

[0009] 4. A structured and controllable generation method based on a diffusion model These methods use cross-attention, conditional injection, mask constraints, or region control strategies to input text, layout, or graph structure conditions into the diffusion model.

[0010] The problem with this type of method is that without a dedicated decoupling mechanism for scene graph nodes and image regions, the attention layer in the diffusion model may still mix multiple node conditions, causing semantic information of different objects or backgrounds to interfere with each other.

[0011] 5. Generation methods based on remote sensing data augmentation These methods focus on the gains of the generated images for downstream object detection or classification tasks, typically emphasizing visual realism and object detectability. However, if the generation process lacks control over scene relationships and background context, the generated images, while visually realistic, may not possess sufficient semantic consistency and training usability. Summary of the Invention

[0012] The purpose of this invention is to provide a method and related apparatus for generating controllable remote sensing images based on scene graphs, which solves the problem in the prior art that text prompts are difficult to clearly constrain the precise position, rotation direction and spatial topology of each target, resulting in low accuracy of controllable remote sensing image generation.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for generating controllable remote sensing images based on scene graphs, comprising the following steps: Obtain the input scene graph and represent it as a directed graph. The directed graph includes nodes, directed edges, and global text description. The nodes include semantic information and geometric information, and the directed edges include semantic information. Construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The geometric information of the nodes is encoded using geometric flow to obtain the node geometric features, and the geometric condition matrix is ​​obtained based on the node geometric features; By encoding the semantic information of nodes and directed edges, as well as the global text description, using relation flow, node relation features and global text embeddings are obtained. Based on node relationship features, visual alignment node features are obtained, and a region decoupling mask attention mechanism is constructed based on the visual alignment node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The output feature map after region decoupling, global text embedding, and geometric condition matrix are input together into the diffusion model for diffusion generation to obtain a controllable remote sensing image.

[0014] A further improvement of this invention is that the calculation formula for the directed graph is:

[0015] in, It is a directed graph. It is a set of several nodes. Let be the set of directed edges. This is a global text description.

[0016] A further improvement of the present invention lies in the set of the plurality of nodes. The calculation formula is:

[0017] in, For the first i 1 node This represents the total number of nodes. No. i Nodes The calculation formula is:

[0018] in, As a category, For a rotated bounding box; Rotate bounding box The calculation formula is:

[0019] in, The coordinates of the four vertices, It is in the format of 8 real numbers.

[0020] A further improvement of the present invention is that the set of directed edges... The calculation formula is:

[0021] in, For nodes To the node The representation of the edges.

[0022] A further improvement of this invention is that the calculation formula for the visual alignment node features is as follows:

[0023] in, For visual alignment of node features, For scaling parameters, For offset parameters, Features of node relationships This is element-wise multiplication.

[0024] A further improvement of this invention is that the calculation formula for the output feature map after region decoupling is as follows:

[0025] in, This is the output feature map after region decoupling. For feature map The background area in the middle, For element-wise multiplication, For cross-attention operators, The input is the number of foreground objects in a single image within a scene graph. For the first image in the input scene graph i Characteristics of a prospective goal.

[0026] A further improvement of this invention is that the calculation formula for the controllable remote sensing image is:

[0027] in, For controllable remote sensing images, This is the output feature map after region decoupling. For global text embedding, The geometric condition matrix, This is the influence coefficient, used to balance the effects of global text embedding and the geometric condition matrix. This is the cross-attention operator.

[0028] Secondly, the present invention provides a controllable remote sensing image generation system based on a scene graph, comprising: The data acquisition module is used to acquire the input scene graph and represent the input scene graph as a directed graph. The directed graph includes nodes, directed edges, and global text descriptions. The nodes include semantic information and geometric information, and the directed edges include semantic information. An encoder construction module is used to construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The first encoding module is used to encode the geometric information of the nodes using the geometric flow, obtain the geometric features of the nodes, and obtain the geometric condition matrix based on the geometric features of the nodes. The second encoding module is used to encode the semantic information of nodes and directed edges, as well as the global text description, using the relation flow to obtain node relation features and global text embeddings. The output feature map acquisition module is used to obtain visually aligned node features based on node relationship features, and to construct a region decoupling mask attention mechanism based on the visually aligned node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The diffusion generation module is used to input the output feature map after region decoupling, global text embedding, and geometric condition matrix into the diffusion model to generate a controllable remote sensing image.

[0029] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the controllable remote sensing image generation method based on scene graph described above.

[0030] Fourthly, the present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the controllable remote sensing image generation method based on scene graph described above.

[0031] Compared with the prior art, the present invention has the following beneficial effects: The proposed method for generating controllable remote sensing images based on scene graphs involves two aspects. First, a relation-geometric scene graph encoder is constructed, comprising a geometric flow and a relational flow. This separate modeling of the geometric and relational flows enhances the global consistency and local controllability of the generated controllable remote sensing images. Second, the relational flow is used to encode the semantic information of nodes, the semantic information of directed edges, and global text descriptions, yielding node relationship features and global text embeddings. This reduces mismatches between target and scene relationships. Furthermore, the decoupled output feature map, global text embeddings, and geometric condition matrices are input into a diffusion model for diffusion generation, resulting in a controllable remote sensing image. This demonstrates that the geometric condition matrix is ​​considered during diffusion generation, enabling the diffusion model to better maintain consistency between the foreground target and the layout of the input scene graph. These techniques effectively address the problem of low accuracy in generating controllable remote sensing images due to the difficulty in explicitly constraining the precise position, rotation direction, and spatial topology of each target through text prompts. Attached Figure Description

[0032] Figure 1 This is a flowchart of the controllable remote sensing image generation method based on scene graphs according to the present invention; Figure 2 This is a schematic diagram of the controllable remote sensing image generation system based on scene graphs according to the present invention; Figure 3 This is an overall framework diagram of the controllable remote sensing image generation method based on scene graph in Embodiment 4 of the present invention; Figure 4 This is a schematic diagram of the structure of the instance perception visual alignment module in Embodiment 4 of the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0033] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0034] Example 1: The flowchart of the controllable remote sensing image generation method based on scene graph of this invention is as follows: Figure 1 As shown, the controllable remote sensing image generation method based on scene graph of the present invention includes the following steps: S1. Obtain the input scene graph and represent the input scene graph as a directed graph. The directed graph includes nodes, directed edges, and global text description. The nodes include semantic information and geometric information, and the directed edges include semantic information. S2. Construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; S3. Encode the geometric information of the nodes using geometric flow to obtain the node geometric features, and obtain the geometric condition matrix based on the node geometric features; S4. Encode the semantic information of nodes and directed edges, as well as the global text description, using relation flow to obtain node relation features and global text embedding; S5. Based on the node relationship features, visual alignment node features are obtained, and a region decoupling mask attention mechanism is constructed based on the visual alignment node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. S6. Input the output feature map after region decoupling, global text embedding, and geometric condition matrix into the diffusion model for diffusion generation to obtain a controllable remote sensing image.

[0035] Example 2: A schematic diagram of the controllable remote sensing image generation system based on scene graph of the present invention is shown below. Figure 2 As shown, the controllable remote sensing image generation system based on scene graphs of the present invention includes: The data acquisition module is used to acquire the input scene graph and represent the input scene graph as a directed graph. The directed graph includes nodes, directed edges, and global text descriptions. The nodes include semantic information and geometric information, and the directed edges include semantic information. An encoder construction module is used to construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The first encoding module is used to encode the geometric information of the nodes using the geometric flow, obtain the geometric features of the nodes, and obtain the geometric condition matrix based on the geometric features of the nodes. The second encoding module is used to encode the semantic information of nodes and directed edges, as well as the global text description, using the relation flow to obtain node relation features and global text embeddings. The output feature map acquisition module is used to obtain visually aligned node features based on node relationship features, and to construct a region decoupling mask attention mechanism based on the visually aligned node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The diffusion generation module is used to input the output feature map after region decoupling, global text embedding, and geometric condition matrix into the diffusion model to generate a controllable remote sensing image.

[0036] Example 3: The controllable remote sensing image generation method based on scene graph of the present invention includes the following steps: S1. Obtain the input scene graph and represent it as a directed graph. The directed graph includes nodes, directed edges, and global text descriptions. The nodes include semantic information and geometric information, and the directed edges include semantic information.

[0037] The formula for calculating the directed graph in this step is:

[0038] in, It is a directed graph. It is a set of several nodes. Let be the set of directed edges. This is a global text description.

[0039] This step involves a set of several nodes. The calculation formula is:

[0040] in, For the first i 1 node This represents the total number of nodes.

[0041] No. i Nodes The calculation formula is:

[0042] in, As a category, For rotating bounding boxes.

[0043] Rotate bounding box The calculation formula is:

[0044] in, The coordinates of the four vertices, It is in the format of 8 real numbers.

[0045] This step involves a set of directed edges. The calculation formula is:

[0046] in, For nodes To the node The representation of the edges.

[0047] S2. Construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow.

[0048] S3. Encode the geometric information of the nodes using geometric flow to obtain the node geometric features, and obtain the geometric condition matrix based on the node geometric features.

[0049] S4. Encode the semantic information of nodes and directed edges, as well as the global text description, using relation flow to obtain node relation features and global text embeddings.

[0050] S5. Based on the node relationship features, obtain the visual alignment node features, and construct a region decoupling mask attention mechanism based on the visual alignment node features. Using the region decoupling mask attention mechanism, generate the background region mask and the object region mask based on the input scene map. Using the background region mask and the object region mask, obtain the output feature map after region decoupling.

[0051] The formula for calculating the visual alignment node features in this step is:

[0052] in, For visual alignment of node features, For scaling parameters, For offset parameters, Features of node relationships This is element-wise multiplication.

[0053] The formula for calculating the output feature map after region decoupling in this step is:

[0054] in, This is the output feature map after region decoupling. For feature map The background area in the middle, For element-wise multiplication, For cross-attention operators, The input is the number of foreground objects in a single image within a scene graph. For the first image in the input scene graph i Characteristics of a prospective goal.

[0055] S6. Input the output feature map after region decoupling, global text embedding, and geometric condition matrix into the diffusion model for diffusion generation to obtain a controllable remote sensing image.

[0056] The formula for calculating the controllable remote sensing image in this step is:

[0057] in, For controllable remote sensing images, This is the output feature map after region decoupling. For global text embedding, The geometric condition matrix, This is the influence coefficient, used to balance the effects of global text embedding and the geometric condition matrix. This is the cross-attention operator.

[0058] Example 4: The method of the present invention will be described in detail below. The overall framework diagram of the controllable remote sensing image generation method based on scene graph of the present invention is shown below. Figure 3 As shown, the controllable remote sensing image generation method based on scene graph of the present invention includes the following steps: Step 1: Obtain the input scene image.

[0059] Step 1 will be explained in detail below: Step 101: Obtain the input scene graph and represent it as a directed graph to describe objects, relationships, and global semantics in the remote sensing scene. The formula for calculating the directed graph is:

[0060] in, This represents a directed graph. express A set of nodes, Represents the set of directed edges. For nodes To the node The representation of the edges, Represents a global text description. Each node... Includes categories and rotating bounding box ,Right now:

[0061] Step 102: Rotate the bounding box Represented by the coordinates of four vertices, its form is:

[0062] in, The coordinates of the four vertices, It is in the format of 8 real numbers.

[0063] The nodes in the input scene graph include both foreground target nodes and background nodes; directed edges are used to represent the relationship types between nodes.

[0064] Step 2: Build a relation-geometry scene graph encoder.

[0065] Step 2 will be explained in detail below: Step 201: The relation-geometry scene graph encoder consists of a geometry stream and a relation stream. The geometry stream encodes the absolute spatial location and category semantics of each node, while the relation stream encodes the relational semantics between nodes and the graph topology. This dual-stream structure separates the processing of geometric layout and relational semantics, avoiding excessive entanglement between different types of features.

[0066] Step 3: Encode node geometry information using geometry flow.

[0067] Step 3 will be explained in detail below: Step 301: For node Extracting categories using a pre-trained text encoder The category semantic embedding is used, and the Fourier position encoding function is utilized. For the revolved bounding box Perform continuous coordinate encoding.

[0068] Step 302: Concatenate the coordinate encoding results and the category semantic embedding, and obtain the node geometric embedding (also known as node geometric features) through a learnable linear mapping:

[0069] in, This indicates a feature splicing operation.

[0070] Step 4: Encode relational semantic information through relational streams.

[0071] Step 4 will be explained below: Step 401: Input the node category text and edge relationship text into the pre-trained text encoder to obtain the node category embedding. Edge Relationship Embedding and global text embedding h t : h t =

[0072] Step 402: Map the above embeddings to a common hidden dimension using a node encoder and an edge encoder to obtain the initial node representation. Relationship between edges :

[0073] Step 403: Aggregate node relationship information using a graph neural network. In the... In the layer, nodes The next level representation is obtained by aggregating the source node pointing to that node and the corresponding edge relationship:

[0074] in, Indicates having a pointer to a node The set of source nodes of the incoming edges, Indicates the first Layer learnable parameters This represents the activation function. After the last layer, the node relationship embeddings (also known as node relationship features) are obtained. .

[0075] Step 5: Instance-aware visual alignment.

[0076] Step 5 will be explained in detail below: Step 501: Node relation embedding obtained from relation flow Located in the semantic relation space, the diffusion model generation process requires conditional features in the visual latent space. Therefore, this embodiment designs an instance-aware visual alignment module, the structure of which is shown in the schematic diagram below. Figure 4 As shown, the instance-aware visual alignment module includes a prior stream and an alignment stream.

[0077] Step 502: The prior stream modulates the relation embedding through a zero-initialization feature modulation layer to inject visual attributes while preserving structural priors. Intermediate visual features. From multilayer perceptron Obtain and use it to generate scaling parameters. and offset parameters The visual alignment node embedding (also known as the visual alignment node feature) is represented as follows:

[0078] in, For visual alignment of node features, For scaling parameters, For offset parameters, Features of node relationships This is element-wise multiplication.

[0079] Step 503: Alignment Stream extracts visual features of the target region using a frozen image encoder. And projected and Applying cosine similarity constraints, the loss function expression is as follows:

[0080] Step 6: Construct region decoupling mask attention.

[0081] Step 6 will be explained in detail below: Step 601: Embed the visual alignment node Divided into foreground feature set and background features The matrix is ​​obtained by stacking foreground features. .

[0082] Step 602: Define the cross-attention operator:

[0083] in, For cross-attention operators, For query vector, For key vectors, It is a value vector.

[0084] Step 603: Contextually enhance the background features using all foreground features to obtain background-aware features:

[0085] Step 604: Generate a binary background mask based on the scene layout And update the feature map using the enhanced background features. Background area in the middle:

[0086] Step 605: For each foreground node, generate a non-overlapping binary foreground mask based on its position. The corresponding foreground node features are then used to update the region. The final output feature map is:

[0087] in, This is the output feature map after region decoupling. For feature map The background area in the middle, For element-wise multiplication, For non-overlapping binary foreground masks, For cross-attention operators, The input is the number of foreground objects in a single image within a scene graph. For the first image in the input scene graph i Characteristics of a prospective goal.

[0088] Through the above processing, background node features and foreground node features are injected into the corresponding image regions respectively, realizing node-level control with region decoupling.

[0089] Step 7: Diffusion generation.

[0090] Step 7 will be explained in detail below: Step 701: Decouple the output feature map of the region As a regionalization condition in the diffusion generation process, the denoising network in the diffusion model further introduces global text embedding through cross-attention joint input. and geometric condition matrix The formula for calculating controllable remote sensing images is:

[0091] in, For controllable remote sensing images, This is the output feature map after region decoupling. For global text embedding, The geometric condition matrix, , This is the influence coefficient, used to balance the effects of global text embedding and the geometric condition matrix. This is the cross-attention operator.

[0092] Step 702: The training objective consists of diffusion denoising loss and instance-aware visual alignment loss:

[0093] in, Indicates the first Step noise latent variable, This represents the denoising model. , This represents the alignment loss weight.

[0094] To verify the effectiveness of the method of this invention, this embodiment was validated on the DIOR-SG, STAR, and COCO-Stuff datasets. Experimental results show that on the DIOR-SG dataset, the method of this invention achieves an FID of 25.46, a YOLOScore mAP50 of 54.15, and a global CLIPScore of 25.58. On the STAR dataset, the method of this invention achieves an FID of 27.08, a YOLOScore mAP50 of 42.56, and a global CLIPScore of 25.12. On the COCO-Stuff dataset, the method of this invention achieves an FID of 27.00 and an IS of 29.14. In downstream object detection experiments, after augmentation using data generated by the method of this invention, the detection model mAP improved from 50.12 to 52.19.

[0095] Compared with the prior art, the present invention has the following advantages: A. More precise control over spatial layout This invention performs Fourier position encoding on the rotated bounding box using geometric flow and introduces a geometric condition matrix H^{geo} during the diffusion generation process, enabling the diffusion model to better maintain the consistency between the foreground target and the layout of the input scene graph. In experiments, this invention achieves YOLOScore mAP_{50} 54.15 and mAP_{50-95} 34.66 on the DIOR-SG dataset, which are higher than the comparison methods, indicating its stronger object-level spatial alignment capability.

[0096] B. Stronger semantic consistency The relation flow of this invention propagates relations through node and edge relation embedding and graph neural networks, enabling the diffusion model to utilize relational information such as "docked at," "located in," "providing scene background," and "in the same river." Compared to methods that generate images solely based on layout, this invention reduces instances of mismatch between target and scene relationships.

[0097] C. Stronger ability to control background context This invention treats the background as an independent node and updates the background region first, then the foreground region, in RDMA (Region-Decoupled Masked Attention). This allows background generation to be constrained by both the foreground target and the relational context. Qualitative results show that in scenarios such as airport runways, ports, and rivers, this invention can generate backgrounds that better match the foreground target.

[0098] D. Fewer feature entanglements between geometric information and relational semantics. Existing methods that encode scene graphs into a single global vector are prone to mixing different types of conditional information. This invention models the geometric flow and relational flow separately using RGSGE (Relation-Geometry Scene Graph Encoder), which helps to enhance the global consistency and local controllability of generating controllable remote sensing images.

[0099] E. Node-level region alignment is more accurate. RDMA ensures that each foreground node feature is applied only to its corresponding mask region, while background node features are applied only to the background region, avoiding semantic leakage caused by the mixing of different node conditions in standard cross-attention. Ablation experiments show that after introducing RDMA, the model's YOLOS score improved from 33.32 / 21.03 to 53.07 / 33.76, indicating that RDMA is the core module for achieving precise spatial control.

[0100] F. The overall quality of the generated images is higher. On the DIOR-SG dataset, this invention achieves an FID of 25.46 and a global CLIPScore of 25.58; on the STAR dataset, it achieves an FID of 27.08 and a global CLIPScore of 25.12; and on the COCO-Stuff dataset, it achieves an FID of 27.00, an IS score of 29.14, and a global CLIPScore of 20.08. These results demonstrate the advantages of this invention in terms of image fidelity, semantic consistency, and cross-dataset generalization.

[0101] G. The effectiveness of each key module is supported by ablation experiments. After fully incorporating RGSGE, RDMA, IVA (Instance-Aware Visual Alignment), and background nodes, this invention achieves optimal or near-optimal results on FID (Fréchet Inception Distance), YOLOScore, and CLIPScore. The relation flow performs better than random initialization or joint training strategies when using a pre-trained and frozen strategy, indicating that the stability of the structural prior plays an important role in the generation quality.

[0102] H. Downstream tasks have higher practical value. When the images generated by this invention are used for object detection data augmentation, the mAP of the Faster R-CNN detection model increases from the baseline of 50.12 to 52.19, mAP_{50} increases from 75.81 to 77.26, and mAP_{75} increases from 54.33 to 58.31. These results demonstrate that the remote sensing images generated by this invention not only have high visual quality but also good training usability.

[0103] I. Supports fine-grained, combinable control By editing the scene graph, vehicle nodes can be added or deleted, or the background can be replaced from a river to a city road, while simultaneously generating a correspondingly modified remote sensing image. This capability facilitates the targeted synthesis of remote sensing samples with specific structures according to mission requirements.

[0104] Example 5: Please see Figure 5 As shown, the present invention also provides an electronic device 100 for a controllable remote sensing image generation method based on scene graphs; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0105] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the controllable remote sensing image generation method based on scene map described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0106] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0107] The memory 101 in the electronic device 100 stores multiple instructions to implement a controllable remote sensing image generation method based on a scene map, and the processor 102 can execute the multiple instructions to achieve the following: Obtain the input scene graph and represent it as a directed graph. The directed graph includes nodes, directed edges, and global text description. The nodes include semantic information and geometric information, and the directed edges include semantic information. Construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The geometric information of the nodes is encoded using geometric flow to obtain the node geometric features, and the geometric condition matrix is ​​obtained based on the node geometric features; By encoding the semantic information of nodes and directed edges, as well as the global text description, using relation flow, node relation features and global text embeddings are obtained. Based on node relationship features, visual alignment node features are obtained, and a region decoupling mask attention mechanism is constructed based on the visual alignment node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The output feature map after region decoupling, global text embedding, and geometric condition matrix are input together into the diffusion model for diffusion generation to obtain a controllable remote sensing image.

[0108] Example 6: If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A controllable remote sensing image generation method based on scene graphs, characterized in that, Includes the following steps: Obtain the input scene graph and represent it as a directed graph. The directed graph includes nodes, directed edges, and global text description. The nodes include semantic information and geometric information, and the directed edges include semantic information. Construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The geometric information of the nodes is encoded using geometric flow to obtain the node geometric features, and the geometric condition matrix is ​​obtained based on the node geometric features; By encoding the semantic information of nodes and directed edges, as well as the global text description, using relation flow, node relation features and global text embeddings are obtained. Based on node relationship features, visual alignment node features are obtained, and a region decoupling mask attention mechanism is constructed based on the visual alignment node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The output feature map after region decoupling, global text embedding, and geometric condition matrix are input together into the diffusion model for diffusion generation to obtain a controllable remote sensing image.

2. The controllable remote sensing image generation method based on scene graphs according to claim 1, characterized in that, The formula for calculating the directed graph is: in, It is a directed graph. It is a set of several nodes. Let be the set of directed edges. This is a global text description.

3. The controllable remote sensing image generation method based on scene graphs according to claim 2, characterized in that, The set of several nodes The calculation formula is: in, For the first i 1 node This represents the total number of nodes. No. i Nodes The calculation formula is: in, As a category, For a rotated bounding box; Rotate bounding box The calculation formula is: in, The coordinates of the four vertices, It is in the format of 8 real numbers.

4. The controllable remote sensing image generation method based on scene graph according to claim 2, characterized in that, The set of directed edges The calculation formula is: in, For nodes To the node The representation of the edges.

5. The controllable remote sensing image generation method based on scene graph according to claim 1, characterized in that, The formula for calculating the visual alignment node features is as follows: in, For visual alignment of node features, For scaling parameters, For offset parameters, Features of node relationships This is element-wise multiplication.

6. The controllable remote sensing image generation method based on scene graph according to claim 1, characterized in that, The formula for calculating the output feature map after decoupling the region is: in, This is the output feature map after region decoupling. For feature map The background area in the middle, For element-wise multiplication, For non-overlapping binary foreground masks, For cross-attention operators, The input is the number of foreground objects in a single image within a scene graph. For the first image in the input scene graph i Characteristics of a prospective goal.

7. The controllable remote sensing image generation method based on scene graph according to claim 1, characterized in that, The formula for calculating the controllable remote sensing image is: in, For controllable remote sensing images, This is the output feature map after region decoupling. For global text embedding, The geometric condition matrix, This is the influence coefficient, used to balance the effects of global text embedding and the geometric condition matrix. This is the cross-attention operator.

8. A controllable remote sensing image generation system based on scene graphs, characterized in that, include: The data acquisition module is used to acquire the input scene graph and represent the input scene graph as a directed graph. The directed graph includes nodes, directed edges, and global text descriptions. The nodes include semantic information and geometric information, and the directed edges include semantic information. An encoder construction module is used to construct a relation-geometry scene graph encoder, which includes a geometry flow and a relation flow; The first encoding module is used to encode the geometric information of the nodes using the geometric flow, obtain the geometric features of the nodes, and obtain the geometric condition matrix based on the geometric features of the nodes. The second encoding module is used to encode the semantic information of nodes and directed edges, as well as the global text description, using the relation flow to obtain node relation features and global text embeddings. The output feature map acquisition module is used to obtain visually aligned node features based on node relationship features, and to construct a region decoupling mask attention mechanism based on the visually aligned node features. Using the region decoupling mask attention mechanism, a background region mask and an object region mask are generated based on the input scene map. Using the background region mask and the object region mask, the output feature map after region decoupling is obtained. The diffusion generation module is used to input the output feature map after region decoupling, global text embedding, and geometric condition matrix into the diffusion model to generate a controllable remote sensing image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the controllable remote sensing image generation method based on scene graph as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the controllable remote sensing image generation method based on scene graph as described in any one of claims 1 to 7.

Images