Combined controllable image generation method and system based on graph structure prior

Abstract

The invention discloses a combined controllable image generation method and system based on graph structure priori, belongs to the technical field of artificial intelligence, and aims to solve the problem that an existing image generation technology is insufficient in space relation control. The method comprises the following steps of: coding a plurality of visual components into feature embedding vectors, and constructing a graph adjacency relation matrix according to a spatial or semantic relation of the feature embedding vectors; then, a double-layer structure prior graph composed of super nodes representing visual components and child nodes representing feature embedding vectors is constructed; then, on the double-layer structure prior graph, through message passing of a hierarchical graph neural network and structural consistency loss constraint, a graph perception feature embedding vector fusing structural information is learned; and finally, taking the image perception feature embedded vector as a condition to be injected into an image generation model, and generating a target image with high structural consistency, physical coherence and controllability through a visual decoder. The method can be applied to image generation tasks in the fields of product design, conceptual art, indoor layout and the like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a controllable image generation technology in computer vision and deep learning, particularly a combined controllable image generation method and system based on graph structure priors. Background Technology

[0002] With the development of deep learning technology, Artificial Intelligence Generated Content (AIGC) has made significant progress, especially in the field of image generation. Existing technologies allow users to generate high-quality images from text descriptions or reference images. Some models allow users to provide fragmented image parts and expect the model to fuse these parts into a semantically complete and logically consistent new conceptual image (e.g., the PiT model disclosed in the academic paper "Piece it Together: Part-Based Concepting with lP-Priors").

[0003] PiT models typically employ an encoder-generator framework. First, they encode individual visual parts into a unified latent space using the internal representation space of an image encoder (such as the IP-Adapter+ representation space). Then, a lightweight generative prior model (such as IP-Prior, a flow matching or diffusion model) is trained, learning to map the feature vectors of individual parts to the feature vectors of the complete concept. Finally, a large diffusion model such as SDXL is used for high-quality image rendering. PiT models have achieved significant success in realizing "part-to-whole" concept generation. Their core contribution lies in systematically solving this problem in the IP+ representation space and providing a complete data processing and training pipeline.

[0004] However, the inventors' research revealed that PiT models and similar technologies typically treat input visual components as a disordered set, ignoring the crucial structural relationships that should exist between them, such as spatial adjacency, directional relationships (up, down, left, right), or functional connections. When the generation task has high structural requirements, such as furniture layout, character pose assembly, or mechanical design, existing methods struggle to guarantee the structural accuracy of the generated images. The generated results often exhibit significant randomness, combining elements that contradict physical principles or even contain logical errors, failing to accurately reflect the user's intention regarding the spatial relationships of the components. This severely limits their application value in scenarios requiring high-precision structural layout (such as industrial product design, architectural visualization, and character concept art).

[0005] Therefore, how to introduce the ability to perceive and utilize the structural relationships between visual components into image generation models based on visual components in order to achieve image generation with higher controllability and structural consistency is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The main objective of this invention is to address the technical shortcomings of existing technologies that treat visual components as disordered sets, resulting in unreasonable image structures and uncontrollable layouts. This invention aims to provide an image generation method and system capable of perceiving and utilizing the structural relationships between components, thereby significantly improving the structural accuracy and controllability of the generated images.

[0007] To achieve this objective, the present invention proposes the following technical solution: a combinatorial controllable image generation method and system based on graph structure priors. In addition to acquiring the features of image components, this method introduces a graph structure describing the spatial relationships between visual components as prior knowledge. The core steps of the combinatorial controllable image generation method based on graph structure priors include:

[0008] a. Input multiple visual components, and the images of the visual components are processed by a visual encoder to obtain a set of feature embedding vectors of the visual components; input a graph adjacency relation describing the preset spatial relationship between the visual components or a semantic relationship based on the visual components, and construct a graph adjacency relation matrix.

[0009] b. Based on the feature embedding vector of the visual component and the graph adjacency matrix, construct a two-layer prior graph consisting of super-nodes (representing the visual component) and sub-nodes (representing the feature embedding vector within the image of the visual component).

[0010] c. Message passing of a hierarchical graph neural network (HGNN) is performed on the two-layer prior graph, including graph attention (GAT) update of the super node layer, graph convolution (GCN) update of the child node layer, and top-down / bottom-up attention interaction. At the same time, the structural consistency is constrained by the loss function Laplacian smoothing of the structural consistency loss term and the edge reconstruction loss, and a graph-aware feature embedding vector that integrates structural information is learned.

[0011] d. The graph-aware feature embedding vector is used as a condition and injected into an image generation model through a cross-attention mechanism to generate a structure-aware global visual component feature embedding vector.

[0012] e. The global component feature embedding vector is input into a high-quality visual decoder to generate the target image.

[0013] Furthermore, in the aforementioned combined controllable image generation method and system based on graph structure prior, the core components of the combined controllable image generation system based on graph structure prior include: a visual encoding module, a graph prior construction module, a graph feature learning module, a conditional generation module, and a visual decoding module.

[0014] The visual encoding module is used to encode the images of the plurality of visual components using a first visual encoder, thereby obtaining an initial feature embedding vector corresponding to each visual component.

[0015] The graph prior construction module constructs a two-layer prior graph containing the supernodes and the child nodes based on the feature embedding vectors of the multiple visual components and the graph adjacency matrix describing the preset spatial relationship between the visual components. Each visual component corresponds to a supernode in the two-layer prior graph, and each dimension or token of its feature embedding vector corresponds to a child node in the two-layer prior graph.

[0016] The graph feature learning module is configured with a hierarchical graph neural network (HGNN). This network employs a carefully designed top-down and bottom-up information transfer mechanism for bidirectional information transfer between supernodes (component level) and child nodes (feature level within components). This enables the feature representation of each supernode to perceive the contextual information of its neighboring supernodes and the predefined relationships between them, thus learning the graph-aware feature embedding vector.

[0017] The condition generation module, the component generation module is configured with the image generation model, which is used to inject the graph-aware feature vector, which is processed by the hierarchical graph neural network (HGNN) and rich in structural information, into an image generation model as a condition to generate a global visual component feature embedding vector.

[0018] The visual decoding module is used to generate the final target image based on the global visual component feature embedding vector using a first visual decoder;

[0019] The modules work collaboratively according to the steps of the graph-structured prior-based combined controllable image generation method.

[0020] Furthermore, in the aforementioned combinatorial controllable image generation method based on graph structure priors, the step of constructing the graph adjacency relation matrix in step a includes:

[0021] The graph adjacency matrix can be obtained from a user-interactive input describing the preset spatial relationships between the visual components;

[0022] The graph adjacency matrix can be automatically calculated based on the semantic relationships between the multiple visual components;

[0023] During the supervised training phase, the multiple visual components are aligned by an affine transformation using a machine learning algorithm to restore their relative positions and sizes in the target image. Based on the intersection-over-union ratio (IoU) or centroid distance between the aligned bounding boxes of the visual components, the graph adjacency matrix is ​​automatically calculated and generated.

[0024] Furthermore, in the aforementioned combinatorial controllable image generation method based on graph structure priors, the step of constructing the two-layer structure graph prior in step b includes:

[0025] Each visual component is considered as a super node, and the attribute features of the super node are initialized with the mean, maximum or 0 value of the feature embedding vector of the visual component. The attribute feature dimension of the super node is 1*D.

[0026] The supernodes are connected according to the graph adjacency matrix;

[0027] The feature embedding vector of each visual component is decomposed into multiple dimensions or tokens, and each dimension or token is regarded as a child node. The attribute features of the child node are the mean of the feature embedding vector of each dimension or token.

[0028] The child node is connected to its corresponding super node.

[0029] Furthermore, in the aforementioned combinatorial controllable image generation method based on graph structure priors, the message passing of the hierarchical graph neural network (HGNN) in step c includes:

[0030] On the two-layer prior graph, a graph attention network (GAT) is applied between the supernodes to aggregate neighbor visual component information;

[0031] On the two-layer prior graph, a graph convolutional network (GCN) is applied between the child nodes inside each of the supernodes to smooth the features;

[0032] A top-down attention mechanism for injecting the aggregated information of the supernode into the attribute features of the child nodes inside it;

[0033] A bottom-up attention mechanism is used to aggregate update information from the child nodes to the attribute features of their respective supernodes.

[0034] Furthermore, in the aforementioned combinatorial controllable image generation method based on graph structure priors, the method further includes employing a loss function with at least one of the aforementioned structural consistency loss terms during the training of the hierarchical graph neural network (HGNN), wherein the loss function is selected from:

[0035] A Laplacian smoothness loss is used to constrain adjacent supernodes on the two-layer prior graph to remain close in the feature space.

[0036] An edge reconstruction loss is provided for training a classifier to reconstruct the adjacency relationships of the two-layer prior graph based on the node features.

[0037] Furthermore, in the aforementioned combinatorial controllable image generation method based on graph structure priors, step d, which uses the graph-aware feature embedding vector as a condition, includes:

[0038] In each denoising or generation step of the image generation model, the graph-aware feature embedding vector is conditionally injected through a cross-attention mechanism to generate a structure-aware global visual component feature embedding vector.

[0039] The image generation model is either a diffusion model or a flow matching model.

[0040] Furthermore, the aforementioned combinatorial controllable image generation method based on graph structure priors also allows users to interactively modify the graph adjacency matrix, including adding, deleting, or modifying the connection relationships between visual components, and applying the user input to guide the generation when constructing the adjacency matrix.

[0041] Furthermore, in the aforementioned combined controllable image generation system based on graph structure priors, the graph feature learning module also uses a loss calculation unit during the training process. This unit calculates at least one of the structure regularization losses, selected from the Laplacian smoothing loss or the edge reconstruction loss.

[0042] Compared with existing technologies, the advantages of this invention are as follows: By explicitly introducing graph priors between visual components and designing a hierarchical graph neural network (HGNN) for information aggregation, structural controllability is enhanced. The integration of structural information reduces the ambiguity of the generation process, enabling the model to generate images with more logical consistency and more harmonious visual effects, thus improving generation quality. The graph construction method supports both automatic generation from images and interactive modification by users, balancing automation efficiency with user-defined flexibility. The graph prior module can be integrated as a plug-in into existing "part-to-whole" generation frameworks such as PiT, exhibiting good compatibility and scalability. This invention enables the generative model to understand and follow the spatial layout between components, outperforming existing technologies in tasks requiring high-precision structural control (such as jigsaw puzzles and layout design). Attached Figure Description

[0043] To more clearly illustrate the embodiments of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings.

[0044] Figure 1 This is a schematic diagram of the overall architecture of the combined controllable image generation method and system based on graph structure priors according to an embodiment of the present invention.

[0045] Figure 2 This is an example diagram illustrating the construction and representation of a two-layer prior diagram according to an embodiment of the present invention.

[0046] Figure 3 This is a flowchart of the internal information transmission mechanism of the hierarchical graph neural network (HGNN) according to an embodiment of the present invention.

[0047] Figure 4 This is a complete flowchart of the combined controllable image generation method and system based on graph structure priors according to an embodiment of the present invention.

[0048] Figure 5 These are examples of comparative experimental results from embodiments of the present invention (qualitative comparison of Graph-PiT results with other models). Detailed Implementation

[0049] The following will be combined with the appendix Figures 1 to 5 The preferred embodiments of the present invention will be described in detail to enable those skilled in the art to better understand the invention. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0050] The core idea of ​​this invention is to encode the spatial relationships between components specified by the user or automatically extracted into a graph structure, and to use a hierarchical graph neural network (HGNN) to learn feature representations that incorporate this structural information, thereby guiding the subsequent image generation process and ensuring that the generated result structurally conforms to expectations.

[0051] It should be noted that although functional modules are divided in the method and system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the method and system diagram or the order in the flowchart.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0053] Embodiment 1 of the present invention: A combinatorial controllable image generation method based on graph structure priors

[0054] This first embodiment details the image generation method according to claim 1, and its process is as follows: Figure 4 As shown.

[0055] Step 1: Input to obtain visual features and construct graph adjacency relationships (corresponding) Figure 1 Module 1001 in the middle, Figure 4 Step 4001 in the process.

[0056] In this embodiment, the method of the present invention first receives N visual component images provided by the user. For each visual component image, the method of the present invention uses a pre-trained visual encoder (e.g., the image encoder used by the IP-Adapter+) to encode it, thereby extracting an initial feature embedding vector x_i for each component i (i = 1, ..., N).

[0057] In this embodiment, the method of the present invention also requires the construction of a graph adjacency matrix A describing the spatial relationships between the N visual components. The construction of the graph adjacency matrix is ​​flexible and can support the various approaches described in claims 3 and 8.

[0058] 1. User Interactive Input: Allows users to directly specify the connection relationships between visual components through a graphical interface or configuration files, for example, explicitly stating "Component A is to the left of Component B" or "Component C is connected to Component D". These instructions are parsed as edges in the adjacency matrix A. This method provides users with the highest level of control freedom.

[0059] 2. Automatic generation based on semantic relationships: For visual components with clear semantics (such as "front", "body", "wheel"), the necessary connection relationships between them can be automatically inferred using prior knowledge bases or large language models.

[0060] 3. Automatic Calculation Based on Visual Information (Supervised Training Phase): During the training phase, when the visual components originate from the same complete target image, the relative position and size of each visual component in the target image can be recovered through an affine transformation alignment algorithm (such as SIFT feature matching combined with RANSAC). Based on the aligned bounding boxes of the visual components, adjacency relationships are generated by calculating the intersection-over-union ratio (IoU) or centroid distance. For example, if the IoU of two components is greater than a preset threshold T_iou, or their centroid distance is less than a threshold T_dist, then the corresponding element in the adjacency matrix A is set to 1, indicating that they are adjacent.

[0061] Step 2: Construction of the two-layer prior diagram (corresponding to...) Figure 1 Module 1002 in the middle, Figure 2 , Figure 4 Step 4002 in the middle.

[0062] In this embodiment, based on the feature embedding vector {x_i} of the visual component obtained in the previous step and the graph adjacency matrix A, the system constructs a two-layer prior graph G, which corresponds to the step described in claim 4:

[0063] Super-node Level: The N visual components are mapped to N super-nodes {S_1, S_2, ..., S_N} in graph G. The connection relationships between the super-nodes are defined by the graph adjacency matrix A. The initial attribute feature h_super_i^(0) of each super-node S_i can be initialized according to its corresponding visual component feature embedding vector x_i. For example, it can use the mean pooling or max pooling result of x_i, or directly use the zero vector for initialization, with a dimension of 1*D.

[0064] Sub-node Level: The feature embedding vector x_i of each visual component (typically d*D, e.g., 16x2048) contains rich internal details. To model these details, each feature embedding vector x_i is decomposed into d tokens, each token being considered a sub-node s_ij (j = 1, ..., d). The initial attribute feature h_sub_ij^(0) of each sub-node s_ij is the feature embedding vector (typically 1*D) of its corresponding token.

[0065] Inter-layer connections: Each supernode S_i establishes connections with all its subordinate child nodes {s_i1, s_i2, ..., s_id}, forming a star topology. Initially, the child nodes may not be connected, or they may be set to fully connected.

[0066] Step 3: Information transfer in a hierarchical graphical neural network (HGNN) (corresponding to...) Figure 1 Module 1003 in the middle, Figure 3 , Figure 4 Step 4003 in the process.

[0067] In this embodiment, to enable the feature representation of each visual component to perceive its contextual structure information, the method of the present invention designs a hierarchical graph neural network (HGNN) to perform information transfer and feature fusion on the two-layer prior graph G. This process corresponds to the mechanism described in claim 5, and the information transfer process within each layer of the HGNN (assuming a total of L layers) is as follows: Figure 3 As shown, it specifically includes:

[0068] 1. Inter-supernode information transfer (GAT): At the supernode layer, a graph attention network (GAT) is applied to aggregate information from neighboring supernodes. This allows each visual component to selectively focus on and absorb contextual information from its neighbors based on its relationship with neighboring visual components.

[0069] 2. Intra-node Messaging (GCN): Within the child node layers of each supernode S_i, a Graph Convolutional Network (GCN) is applied to smooth and integrate the features within the visual component. This helps to form a more robust representation of the content of the visual component itself.

[0070] 3. Top-down attention: The aggregated information of the supernode (the context of the visual component level) is injected into each of its subordinate child nodes. This enables the internal features of the visual component (the child nodes) to perceive the role that visual component plays in the overall structure.

[0071] 4. Bottom-up attention: The updated child node information is aggregated and used to update the supernode to which it belongs. This ensures that the visual component-level representation reflects changes in its internal details.

[0072] In this embodiment, by iterating the above process L times (e.g., L=2), and combining residual connections and layer normalization, the HGNN finally outputs a set of graph-aware feature embedding vectors {h_sub_ij^(L)} that integrate rich structural information.

[0073] Step 4: Conditional Generation and Training (corresponding) Figure 1 Module 1004 in the middle, Figure 4 Step 4004 in the middle.

[0074] In this embodiment, the graph-aware feature embedding vector {h_sub_ij^(L)} output by the HGNN is used as a structuring condition and injected into an image generation model to generate a structure-aware global visual component feature embedding vector. This process corresponds to the steps described in claim 7:

[0075] Conditional Injection: The image generation model is preferably a diffusion model or a flow matching model, such as the DiT (Diffusion Transformer) architecture. In each Transformer block of the model, the graph-aware feature embedding vector {h_sub^(L)} is used as the key and value through a cross-attention mechanism, while the latent image representation of the current denoising step is used as the query. This ensures that each step of the generation process strictly follows the provided graph structure prior.

[0076] Training Objective and Structural Consistency Loss: To ensure that the graph-aware feature embedding vectors learned by the HGNN accurately reflect the graph structure, according to claim 6, a structural consistency loss term is introduced during training. The total loss function L includes the standard generation loss Lgen (such as flow matching loss or diffusion loss) and at least one structural regularization loss.

[0077] Laplacian Smoothness Loss (Lsmooth): This loss constrains the adjacent supernodes in the bilayer prior graph G, whose feature representations should also remain close in the latent space. The formula is L_smooth=∑A_ij||h′_i-h′_j||^2, where h′_i is the final feature of the supernode i.

[0078] Edge Reconstruction Loss (Lrecon): This loss works by training a classifier that can predict whether an edge exists between a pair of supernodes based on their features h′_i and h′_j (i.e., reconstruct the adjacency matrix A). This forces the node features to encode enough information to reflect the graph's topology.

[0079] The total loss is L = L_gen + λ_s * L_smooth + λ_r * L_recon, where λ_s and λ_r are hyperparameters.

[0080] Step 5: High-quality visual decoding (corresponding to) Figure 1 Module 1005 in the middle, Figure 4 Step 4005 in the process.

[0081] In this embodiment, the trained image generation model receives the graph-aware feature vector as input during inference and generates a final global visual component feature embedding vector z_final. This vector z_final is then fed into a high-quality visual decoder (such as the decoder part of SDXL) to render the final target image. Because the generation process is strictly constrained by the graph structure, the final target image exhibits high consistency and rationality in the layout of the visual components and the overall structure.

[0082] In this embodiment, to verify the effectiveness of the proposed graph-structured prior-based combinatorial controllable image generation method, the Graph-PiT model proposed in this invention was compared with four representative baseline models: PiT, IP-Adapter+, λ-ECLIPSE, and OmniGen, which represent state-of-the-art approaches in the fields of controllable generation and visual infrastructure construction. Testing was conducted on four datasets (people, products, interior layouts, and puzzles) to demonstrate that the graph prior proposed in this invention can improve generation based on controlled components. These benchmarks provide an effective testing platform for structured generation, covering a wide range of compositional styles.

[0083] In this embodiment, Figure 5 This paper presents a qualitative comparison example of the Graph-PiT model results from the present invention with other models. The Graph-PiT model proposed in this invention consistently generates more coherent, structurally faithful, and semantically consistent images across all domains. Graph-PiT effectively captures spatial dependencies between parts, which are often ignored or mishandled in previous models (e.g., maintaining the integrity of human poses, product assembly, or layout). Graph-PiT uses explicit part-level topology to ensure the realism of compositions, while PiT and lP-Adapter+ tend to treat parts as isolated inputs without modeling pairwise relationships, and λ-ECLIPSE focuses on potential consistency but lacks structure awareness. Although OmniGen is flexible, it performs poorly in situations requiring precise part coordination and lacks interpretable structural priors.

[0084] In this embodiment, Figure 5 Comparative studies highlight the importance of adding graph-based prior constraints to combinatorial controllable image generation models. Graph-PiT's explicit relational reasoning approach better follows user intent, provides greater control, and generalizes more effectively to unseen combinations.

[0085] Embodiment 2 of the present invention: A combined controllable image generation system based on graph structure prior.

[0086] This embodiment describes a system for implementing the above method, the system corresponding to the configuration described in claims 2 and 9. The system includes:

[0087] The visual encoding module, corresponding to the first half of step one in Embodiment 1, is responsible for encoding the input visual component image into an initial feature embedding vector.

[0088] The graph prior construction module, corresponding to the latter half of step one and step two of embodiment one, is responsible for constructing the graph adjacency matrix A and building the two-layer prior graph G.

[0089] The graph feature learning module, corresponding to step three of embodiment one, is centered on the hierarchical graph neural network (HGNN), which is responsible for learning the graph-aware feature vector. During training, this module also includes a loss calculation unit for calculating the structural regularization losses (L_smooth and L_recon) as described in claim 9.

[0090] The conditional generation module corresponds to step four of embodiment one. Its core is the conditional image generation model (such as DiT), which is responsible for generating the global visual component feature embedding vector based on the graph-aware feature vector.

[0091] The visual decoding module, corresponding to step five of embodiment one, is responsible for embedding the global visual component features into a vector and rendering it into the final target image.

[0092] In this embodiment, these modules work together and are implemented in the form of software, hardware, or a combination of software and hardware, together constituting the combined controllable image generation method and system based on graph structure prior of the present invention.

[0093] In summary, the above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Any simple modifications, equivalent transformations, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A combinatorial controllable image generation method and system based on graph structure priors, characterized in that, The combinatorial controllable image generation method based on graph structure priors includes the following steps: a. Input multiple visual components, and the images of the visual components are processed by a visual encoder to obtain a set of feature embedding vectors of the visual components; input a graph adjacency relation describing the preset spatial relationship between the visual components or a semantic relationship based on the visual components, and construct a graph adjacency relation matrix. b. Based on the feature embedding vector of the visual component and the graph adjacency matrix, construct a two-layer prior graph consisting of super-nodes (representing the visual component) and sub-nodes (representing the feature embedding vector within the image of the visual component); c. Message passing of a hierarchical graph neural network (HGNN) is performed on the two-layer prior graph, including the graph attention network (GAT) update of the super node layer, the graph convolution network (GCN) update of the child node layer, and top-down / bottom-up attention interaction. At the same time, the structural consistency is constrained by the loss function Laplacian smoothing of the structural consistency loss term and the edge reconstruction loss, and a graph-aware feature embedding vector that integrates structural information is learned. d. The graph-aware feature embedding vector is used as a condition and input into an image generation model through a cross-attention mechanism to generate a structure-aware global visual component feature embedding vector; e. The global component feature embedding vector is input into a high-quality visual decoder to generate the target image.

2. A combinatorial controllable image generation method and system based on graph structure priors, characterized in that, The graph-structure-based prior-based combinatorial controllable image generation system is characterized by comprising: A visual encoding module is used to encode the images of the plurality of visual components using a first visual encoder to obtain an initial feature embedding vector corresponding to each visual component. A graph prior construction module constructs a two-layer prior graph containing the supernodes and the child nodes based on the feature embedding vectors of the multiple visual components and the graph adjacency matrix describing the preset spatial relationship between the visual components. Each visual component corresponds to a supernode in the two-layer prior graph, and each dimension or token of its feature embedding vector corresponds to a child node in the two-layer prior graph. A graph feature learning module is configured with a hierarchical graph neural network (HGNN). In this network, a carefully designed top-down and bottom-up information transmission mechanism is used to transmit information bidirectionally between the supernodes (component level) and the child nodes (component internal feature level). This enables the feature representation of each supernode to perceive the contextual information of its neighboring supernodes and the preset relationships between them, and learn the graph-aware feature embedding vector. A condition generation module is provided, wherein the component generation module is configured with the image generation model, which is used to inject the graph-aware feature vector, which is processed by the hierarchical graph neural network (HGNN) and rich in structural information, into an image generation model as a condition to generate a global visual component feature embedding vector. A visual decoding module is used to generate the final target image based on the global visual component feature embedding vector using a first visual decoder; The modules work together in accordance with the steps of the method described in claim 1.

3. The invention according to claim 1 or 2, characterized in that, The step of constructing the graph adjacency matrix in step a includes: The graph adjacency matrix can be obtained from a user-interactive input describing the preset spatial relationships between the visual components; The graph adjacency matrix can be automatically calculated based on the semantic relationships between the multiple visual components; During the supervised training phase, the multiple visual components are aligned by an affine transformation using a machine learning algorithm to restore their relative positions and sizes in the target image. Based on the intersection-over-union ratio (IoU) or centroid distance between the aligned bounding boxes of the visual components, the graph adjacency matrix is ​​automatically calculated and generated.

4. The invention according to claim 1 or 2, characterized in that, The steps in step b for constructing the prior of the two-layer structure diagram include: Each visual component is considered as a super node, and the attribute features of the super node are initialized with the mean, maximum or 0 value of the feature embedding vector of the visual component. The attribute feature dimension of the super node is 1*D. The supernodes are connected according to the graph adjacency matrix; The feature embedding vector of each visual component is decomposed into multiple dimensions or tokens, and each dimension or token is regarded as a child node. The attribute features of the child node are the mean of the feature embedding vector of each dimension or token. The child node is connected to its corresponding super node.

5. The invention according to claim 1 or 2, characterized in that, The message passing in step c of the hierarchical graph neural network (HGNN) includes: On the two-layer prior graph, a graph attention network (GAT) is applied between the supernodes to aggregate neighbor visual component information; On the two-layer prior graph, a graph convolutional network (GCN) is applied between the child nodes inside each of the supernodes to smooth the features; A top-down attention mechanism for injecting the aggregated information of the supernode into the attribute features of its internal child nodes; A bottom-up attention mechanism is used to aggregate update information from the child nodes to the attribute features of their respective supernodes.

6. The invention according to claim 1 or 2, characterized in that, It also includes, during the training of the hierarchical graph neural network (HGNN), employing a loss function with at least one of the structural consistency loss terms, the loss function being selected from: A Laplacian smoothness loss is used to constrain adjacent supernodes on the two-layer prior graph to remain close in the feature space. An edge reconstruction loss is provided for training a classifier to reconstruct the adjacency relationships of the two-layer prior graph based on the node features.

7. The invention according to claim 1 or 2, characterized in that, The step d, which uses the graph-aware feature embedding vector as a condition, includes: In each denoising or generation step of the image generation model, the graph-aware feature embedding vector is conditionally injected through a cross-attention mechanism to generate a structure-aware global visual component feature embedding vector. The image generation model is either a diffusion model or a flow matching model.

8. The invention according to claim 1 or 2, characterized in that, The method also allows users to interactively modify the graph adjacency matrix, including adding, deleting, or modifying connections between visual components, and applies the user input to guide the generation when constructing the adjacency matrix.

9. The invention according to claim 1 or 2, characterized in that, The aforementioned combined controllable image generation system based on graph structure prior is characterized in that, during the training process, the graph feature learning module also uses a loss calculation unit, which calculates at least one of the structure regularization losses, selected from the Laplacian smoothing loss or the edge reconstruction loss.

Images