Diffusion model image synthesis method and system based on structural perception and fine-grained semantic alignment

Abstract

This invention discloses an image synthesis method and system based on a diffusion model with structure awareness and fine-grained semantic alignment. The method includes the following steps: modeling the spatial layout structured knowledge of the input text using a spatial relationship enhancement module, and integrating this knowledge into the embedded representation of the input text as a guiding condition for the diffusion model; constructing positive sample pairs and fine-grained negative sample pairs between objects and attributes for the input text using an attribute enhancement module; and constructing a contrastive cross-attention loss based on the positive and negative sample pairs to constrain the denoising process of the diffusion model based on the guiding conditions, thereby generating a high-quality image. This invention solves the problems of missing objects, attribute mismatches, and incorrect spatial relationships in the generated image by explicitly modeling the spatial topology and fine-grained attribute constraints in the text, without significantly increasing the model parameter overhead.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and computer vision, specifically relating to an image synthesis method and system based on a diffusion model with structure perception and fine-grained semantic alignment. Background Technology

[0002] In recent years, text-to-image generation technology based on diffusion models has made significant progress, enabling the generation of high-quality images from natural language descriptions. Mainstream models typically employ the U-Net architecture combined with a cross-attention mechanism, using text embeddings as conditions to guide the noise prediction process. However, existing diffusion models often fail to faithfully reflect the details specified in the text when dealing with complex combined text prompts, mainly exhibiting the following three typical errors: (1) Object missing error: The generated image is missing objects explicitly mentioned in the text, resulting in incomplete scene content. (2) Attribute mismatch error: The visual attributes (such as color, size, material, etc.) of the generated objects are inconsistent with the text description. (3) Spatial relationship error: The relative spatial relationships between objects do not match the text description.

[0003] The main reason is that the cross-attention mechanism in the diffusion model has inherent limitations when dealing with fine-grained semantic alignment: the text embedding lacks explicit structural information guidance, making it difficult to capture both global layout constraints and local attribute features at the same time, resulting in insufficient semantic fidelity of the model in complex combination scenarios.

[0004] To address the aforementioned problems, existing technologies have proposed several improvement schemes from different perspectives, but all of them suffer from varying degrees of technical shortcomings: (1) Improved methods for object existence and attribute alignment: The Attend-and-Excite method (Chefer et al., 2023) aims to improve the alignment of object existence and attributes by iteratively updating the latent representation to ensure the activation of key tokens in the cross-attention graph. However, this method relies on a predefined set of tokens for attention guidance and cannot dynamically adapt to complex and varied spatial relationship descriptions, resulting in limited effectiveness when dealing with scenes involving multiple object relative position constraints. The Structured Diffusion method (Feng et al., 2023) enhances semantic alignment in the cross-attention graph by extracting noun phrases from the prompt and embedding them separately. However, this method mainly focuses on the independent processing of semantic blocks and does not explicitly model the spatial topology between objects, making it difficult to effectively handle composite prompts containing complex spatial relationship descriptions.

[0005] (2) Improved methods for spatial relationship modeling: The Diffusion-SpaceTime-Attn method (Wu et al., 2023a) uses a layout predictor to estimate object regions and applies dynamic spatiotemporal attention control to solve the object misalignment problem. These methods usually introduce additional network branches to predict bounding boxes and apply spatial constraints during the denoising process. However, their technical limitations are: they usually only focus on the optimization of spatial layout and ignore the synchronous enhancement of attribute binding, resulting in the existence of attribute mismatch or object omission problems; at the same time, the introduced layout prediction module significantly increases the computational burden and parameter overhead of the model.

[0006] (3) Improved methods for attribute binding alignment: The SynGen method (Rassin et al., 2023) identifies entity-modifier pairs by parsing the syntactic dependency graph and introduces a contrastive loss function to force the alignment of cross-attention graphs, making the attention distribution of object tokens and their attribute tokens more consistent. However, this method has obvious shortcomings in the negative sample construction strategy: all mismatched tokens are regarded as negative samples, resulting in an excessively large negative sample set and a high proportion of noise, which causes the contrastive loss value to inflate, affecting the stability and convergence efficiency of the training process; in addition, this method does not consider the potential impact of spatial relation constraints on attribute binding, and it is difficult to handle complex composite prompts containing both spatial and attribute constraints at the same time.

[0007] Based on the above analysis, existing technologies have the following common shortcomings in solving the semantic fidelity problem of text-to-image generation: (1) Task optimization isolation problem: Existing methods usually adopt a single task optimization strategy, either dealing with spatial misalignment problems alone or attribute mismatch problems alone, ignoring the inherent coupling dependency between spatial relationships and attribute binding in visual scene construction. In practical application scenarios, text prompts often contain complex combinations of object attributes and spatial relationships (such as "purple camera on a red chair"). Independent optimization of a single task is difficult to meet the generation requirements of complex semantic constraints, and spatial errors and attribute errors often overlap and amplify each other, resulting in a significant decrease in the final generation quality.

[0008] (2) Problem of coarse negative sample construction strategy: In the contrastive learning framework, the quality of negative sample construction directly affects the learning effect and generalization ability of the model. Existing methods mostly adopt random sampling or full negative sample strategy, lacking careful design of fine-grained difficult negative samples that are "semantically similar but do not match attributes / relationships", resulting in weak discrimination of supervision signals, low training efficiency, and difficulty in achieving accurate fine-grained semantic alignment.

[0009] (3) The problem of balancing model efficiency and generation effect: Although some methods improve the generation effect to a certain extent, they introduce a large number of additional parameters or complex inference processes, which significantly slows down the model inference speed and increases the memory usage, making it difficult to deploy efficiently in real-world application scenarios. For example, some methods based on large language models that guide the entire process enhance semantic understanding capabilities, but the computational overhead is huge, and they do not fundamentally solve the problems of negative sample noise interference and space-attribute joint modeling.

[0010] (4) Problem of imperfect evaluation system: Existing evaluation indicators focus on overall image quality (such as FID, CLIPScore) or single-dimensional semantic alignment (such as only evaluating the object presence rate or only evaluating the attribute accuracy). There is a lack of a comprehensive benchmark testing system that can simultaneously and jointly evaluate the accuracy of spatial relationships and the fidelity of attribute binding, making it difficult to comprehensively and objectively reflect the real performance of the model in complex combined scenarios.

[0011] Therefore, there is an urgent need for an image synthesis method that can simultaneously optimize spatial relationship modeling and attribute binding alignment, adopt a fine-grained strategy for constructing difficult negative samples, and maintain the characteristics of lightweight and efficient models. This method would systematically address the shortcomings of existing technologies in terms of semantic fidelity, negative sample noise control, model efficiency, and evaluation system, and meet the requirements for high-quality and high-reliability image generation of complex combined text prompts. Summary of the Invention

[0012] In view of the above, the purpose of this invention is to provide an image synthesis method and system based on structure awareness and fine-grained semantic alignment using a diffusion model. Building upon existing latent diffusion models, it introduces two complementary lightweight enhancement modules: a spatial relationship enhancement module and an attribute enhancement module. This invention aims to solve the problems of missing objects, attribute mismatches, and incorrect spatial relationships in generated images by explicitly modeling the spatial topology and fine-grained attribute constraints in the text, without significantly increasing model parameter overhead.

[0013] To achieve the above-mentioned objectives, an embodiment provides an image synthesis method based on a diffusion model with structure awareness and fine-grained semantic alignment, comprising the following steps: Based on the spatial relationship enhancement module, the spatial layout structure knowledge of the input text is modeled and integrated into the embedded representation of the input text, which together serve as the guiding condition for the diffusion model. The attribute enhancement module constructs positive sample pairs for the input text and enhances fine-grained negative sample pairs between objects and attributes. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's denoising process based on guided conditions, generating high-quality images.

[0014] Preferably, the spatial layout structured knowledge of the input text is modeled based on the spatial relationship enhancement module, including: After extracting relation triples from the input text, a heterogeneous semantic graph is constructed based on the extracted relation triples. The nodes in the graph are divided into relation nodes and object nodes. A dedicated adjustment node is introduced for each type of relation to learn the adjustment vector of a specific relation. The edges in the graph represent the connection relationship between objects and relations. Heterogeneous graph neural networks are used to perform message passing and feature learning on heterogeneous semantic graphs to obtain spatial layout structured knowledge of the input text.

[0015] Preferably, heterogeneous graph neural networks are used to perform message passing and feature learning on heterogeneous semantic graphs, including: The initial embeddings of object nodes and relation nodes in the heterogeneous semantic graph are initialized using pre-trained CLIP word embeddings to obtain node features; Heterogeneous graph neural networks employ the GraphSAGE architecture, which uses aggregation functions to aggregate features from neighboring nodes onto the adjustment node. After layer graph propagation, each relation type adjustment node learns an adjustment vector containing contextual information as structured knowledge of spatial layout.

[0016] Preferably, the spatial layout structure knowledge is integrated into the embedded representation of the input text, including: After weighted summation of the adjustment vectors of all relation types corresponding to the adjustment nodes, this summation is added to the encoded vector of the input text in the text encoder to obtain the final text embedding representation rich in spatial layout structured knowledge. The final text embedding representation serves as the guiding condition for the diffusion model.

[0017] Preferably, constructing positive sample pairs and fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module includes: Perform syntactic dependency parsing on the input text, identify object-attribute dependency pairs, construct positive sample pairs, and represent the correct attribute binding relationships; By arranging attributes, objects are paired with non-corresponding attributes to construct negative sample pairs, and / or by comparing features between different objects to construct negative sample pairs, thereby enhancing the model's ability to distinguish different objects.

[0018] Preferably, the denoising process of the diffusion model based on guiding conditions is constrained by constructing a contrastive cross-attention loss based on positive and negative sample pairs, including: In the cross-attention layer of the U-net network in the diffusion model, the latent spatial features Z of the image output at the previous time step are... tTo query matrix Q, using the text embedding representation as the guiding condition as the key matrix K and the value matrix V, we calculate the attention probability distribution map of the association weight between each token in the text embedding representation and the spatial location of the image, and obtain the attention probability distribution map sequence. For each sample pair, extract two attention probability distribution maps from the attention probability distribution map sequence for each element in the sample pair corresponding to the token. and ; Using symmetric KL divergence to measure the difference distance between the two attention probability distribution maps for each sample pair, the contrastive cross-attention loss consists of minimizing the difference distance between the two attention probability distribution maps corresponding to positive sample pairs and maximizing the difference distance between the two attention probability distribution maps corresponding to negative sample pairs. Using contrastive cross-attention loss gradient Image latent spatial features Perform an update to obtain the updated latent space features of the image. The updated formula is: in, To update the step size.

[0019] Preferably, the spatial relationship enhancement module and diffusion model undergo parameter optimization before application. The optimization process includes: In the first half of the denoising time step, the spatial relationship enhancement module and the diffusion model are simultaneously optimized using the standard denoising loss of the diffusion model and the contrastive cross-attention loss constructed based on positive and negative sample pairs. In the latter half of the denoising time step, only the standard denoising loss is used to simultaneously optimize the spatial relation enhancement module and the diffusion model.

[0020] To achieve the above-mentioned objectives, embodiments of the present invention also provide an image synthesis system based on a diffusion model with structure awareness and fine-grained semantic alignment, comprising: The spatial relation enhancement text processing unit is used to model the spatial layout structured knowledge of the input text based on the spatial relation enhancement module, and integrates the spatial layout structured knowledge into the embedded representation of the input text, which together serve as the guiding condition for the diffusion model. The attribute-enhanced image generation unit is used to construct positive sample pairs and enhance fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's guided denoising process, thereby generating high-quality images.

[0021] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment.

[0022] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-mentioned diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment.

[0023] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. Significantly Improves Semantic Faithfulness and Generation Accuracy: This invention significantly improves the consistency between generated images and text descriptions through dual optimization of structure awareness and fine-grained semantic alignment. Based on benchmark tests of the SRA-Bench dataset, which contains both object-to-object attributes and object-to-object relationships, experimental data shows: Overall score improvement: In the final score, this invention improved by 167.7% compared to the base model (StableDiffusion 1.4) and by 121.6% compared to Stable Diffusion 2.1; Key performance indicator optimizations: In complex composite scenarios, object recall increased from 30.62% to 57.38%, attribute Fidelity increased from 20.13% to 52.38%, and spatial relationship accuracy (VISOR) increased from 17.63% to 34.00%. This demonstrates that the present invention has significant advantages in handling complex descriptions of multiple objects and multiple attributes.

[0024] 2. Effectively solves three core problems: missing objects, attribute mismatch, and spatial errors: This invention provides targeted solutions to the three common types of errors in existing diffusion models. Reduced object missing rate: By explicitly injecting spatial layout structured knowledge into the heterogeneous graph neural network, the diffusion model's understanding of global semantics is enhanced, resulting in a significant reduction in the number of missing objects in the generated images. Reduce attribute mismatch: By using fine-grained contrastive cross-attention learning, the attention graph of the object token is forced to align with that of its correct attribute token, effectively avoiding errors in swapping attributes such as color and size; Correcting spatial relationships: By extracting relation triples and modeling heterogeneous semantic graph structures, the error in the relative positional relationship between objects is significantly improved. Under the complex prompt that simultaneously satisfies spatial relationships and attribute constraints, the generation success rate exceeds 40%, far exceeding the level of existing methods which is usually below 20%.

[0025] 3. Lightweight design and high parameter efficiency: While ensuring performance improvement, this invention strictly controls model complexity, and has good economy and deployment feasibility.

[0026] 4. Excellent Compatibility and Generalization Ability: This invention can be seamlessly integrated into existing latent diffusion model architectures (such as Stable Diffusion 1.4 / 2.1) without requiring large-scale reconstruction or retraining of the base model, demonstrating excellent compatibility and scalability. In prompt tests at different complexities (easy, medium, and hard), this invention consistently exhibits superior robustness compared to existing technologies. Particularly in high-difficulty scenarios, when the performance of other methods significantly degrades, this invention maintains stable generation quality, proving its strong generalization ability. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment provided in the embodiment; Figure 2 This is a flowchart of the diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment provided in the embodiment. Figure 3 This is a schematic diagram of the structure of the diffusion model image synthesis system based on structure awareness and fine-grained semantic alignment provided in the embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0030] The inventive concept of this invention is as follows: This invention aims to solve the problems of missing objects, mismatched attributes, and incorrect spatial relationships in the text-to-image generation process of existing diffusion models. Specific objectives include: (1) providing a method that can explicitly model spatial relationships, reducing object omissions and improving positional accuracy through structured knowledge injection; (2) providing a fine-grained attribute alignment mechanism, enhancing the binding relationship between objects and their correct attributes through contrastive learning, and avoiding attribute confusion; (3) improving the semantic fidelity of generated images while maintaining the lightweight characteristics of the model, minimizing additional parameter overhead, and ensuring inference efficiency.

[0031] like Figure 1 As shown in the embodiment, an image synthesis method based on a diffusion model with structure awareness and fine-grained semantic alignment is provided, including the following steps: S1, based on the spatial relationship enhancement module, models the spatial layout structured knowledge of the input text, integrates the spatial layout structured knowledge into the embedded representation of the input text, and uses them together as the guiding conditions for the diffusion model.

[0032] In this embodiment, the spatial relationship enhancement module is used to enhance the spatial layout structured knowledge in the text embedding through a heterogeneous graph neural network. For example... Figure 2 As shown, it specifically includes: (a) Extract relation triples from the input text.

[0033] It receives natural language descriptive text input from the user. For example, the input text D is: "A brown bear is sitting on a book, with a pink vase to its left and a green toothbrush underneath."

[0034] A pre-trained Large Language Model (LLM), preferably DeepSeek-Chat, GPT-3.5, or Qwen-2.5, is invoked to perform semantic analysis on the input text D and extract the set of relation triples R. d The extracted triples are represented in the form of (subject, relation, object). LLM can handle complex natural language expressions, ensuring the semantic accuracy of the extracted relations.

[0035] For the input text D above, the extracted set of triples R d For {(bear, on top of..., book), (toothbrush, below..., bear), (vase, to the left of..., bear)}.

[0036] (b) Construct a heterogeneous semantic graph based on the extracted relation triples.

[0037] Based on the extracted set of triples R dFor each unique spatial relationship type (e.g., "above" or "to the left"), an independent heterogeneous semantic graph G=(V,E) is constructed, where V represents the set of nodes and E represents the set of edges. The node types include object nodes V. o : Represents the subject and object in a triple, such as {bear, book, vase, toothbrush}; relation node V r : Represents spatial relationships in a triple, such as {above..., to the left of..., below...}; Adjust node V Δ : Introduce a dedicated adjustment node for each relation r to learn relation-specific adjustment vectors.

[0038] The edges in the graph represent the connections between objects and relationships, thus forming graph data that reflects the spatial topology of the text. Edge types include object-attribute edges, which represent attribute affiliation relationships, and object-relationship edges, which represent spatial topological relationships.

[0039] (c) Use heterogeneous graph neural networks to perform message passing and feature learning on heterogeneous semantic graphs to obtain spatial layout structured knowledge of the input text.

[0040] In this embodiment, the initial embedding of all nodes in the heterogeneous semantic graph is initialized using pre-trained CLIP word vectors, preferably with a dimension of 768, to ensure rich semantic information.

[0041] The heterogeneous graph neural network preferably adopts the GraphSAGE architecture, with a network layer count of 2. The aggregation function uses either mean-pooling or max-pooling, with mean-pooling being preferred. The number of neighbor nodes sampled in each layer is set to N.

[0042] During message passing and feature learning, the aggregation functions of the GraphSAGE architecture aggregate the features of neighboring nodes onto the adjustment node. After layer graph propagation, each adjustment node of a relation type learns an adjustment vector containing contextual information. As structured knowledge of spatial layout.

[0043] in, Representing relation type r The set of neighboring nodes of the regulating node, Representing neighboring nodes In the l Hidden vectors of the layer Represents aggregate functions, and Indicates activation function The weight parameters and bias parameters.

[0044] (d) Integrate the spatial layout structure knowledge into the embedded representation of the input text, and use it together as the guiding condition for the diffusion model.

[0045] In this embodiment, the input text is processed by a text encoder to obtain a text embedding. The text embedding vector output by the text encoder includes the embedding vectors corresponding to the words in the text and the end token. During knowledge fusion, the spatial layout structured knowledge output by the heterogeneous graph neural network is fused into the end token ([EOT]) embedding of the text encoder.

[0046] Specifically, the adjustment vectors of all relation types corresponding to the adjustment nodes are weighted and summed, and then added to the encoded vector of the input text in the text encoder to obtain the final text embedding representation rich in spatial layout structure knowledge. The final text embedding representation As a guiding condition for the diffusion model, it is input into the U-Net network of the diffusion model to guide the noise prediction process.

[0047] in, This is the encoded vector of the input text in the text encoder. This represents the adjustment vector for the r-th relation type. For a set of relation types, This is a hyperparameter for controlling the modulation intensity of the structure.

[0048] In the embodiments, the preferred range of the modulation intensity parameter λ is [0.1, 1.0], with a default value of 0.5. Experiments show that when the scene contains multiple relationships, a smaller λ value (such as 0.3) helps to avoid overcorrection.

[0049] S2 constructs positive sample pairs and fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's denoising process based on guiding conditions, generating high-quality images.

[0050] In this embodiment, the attribute enhancement module is used to enhance the fine-grained binding between objects and attributes through contrastive cross-attention learning. For example... Figure 2 As shown, it specifically includes: (a) Construct positive sample pairs.

[0051] Using a Transformer-based syntactic dependency parser, preferably spaCy, the input text is parsed to identify object and attribute dependency pairs, and a set N of positive sample pairs is constructed. posThis indicates a correct attribute binding relationship. For the input text D above, the constructed positive sample pairs include {(bear, brown), (vase, pink), (toothbrush, green)}.

[0052] (b) Construct fine-grained negative sample pairs.

[0053] To provide strong supervisory signals, a fine-grained set N of negative samples is constructed, containing semantically similar samples but with mismatched attributes. neg The following two strategies are used to generate negative sample pairs: Strategy 1, Attribute Swap Sampling: This involves swapping the attributes of different objects within the same text, forcing the diffusion model to distinguish the attribute features of different objects within the same scene. For example, swapping the attributes of "brown bear" and "pink vase" generates negative samples (bear, pink), (bear, green), and (vase, brown).

[0054] Strategy 2: Inter-Object Sampling: Compare the feature differences between different object instances to enhance the diffusion model's ability to distinguish between them. For example, construct negative sample pairs (bear, vase), (vase, toothbrush), (bear, toothbrush).

[0055] Final negative sample set N neg Generate the union of negative sample pairs for the two strategies described above.

[0056] (3) Construct a contrastive cross-attention loss based on positive and negative sample pairs.

[0057] During the denoising process at each time step of the diffusion model, an attention map is computed. Specifically, in the cross-attention layer of the U-net network in the diffusion model, the latent spatial features Z of the image output from the previous time step are used. t To query matrix Q, we use the text embedding representation as the guiding condition as the key matrix K and the value matrix V, and calculate the attention probability distribution map of the association weight between each token in the text embedding representation and the spatial location of the image, thus obtaining the attention probability distribution map sequence.

[0058] For each sample pair (including positive and negative sample pairs), extract two attention probability distribution maps from the attention probability distribution map sequence for each element (object and / or attribute) in each sample pair corresponding to the token. and ; Symmetric KL divergence is used to measure the difference distance between the two attention probability distribution maps for each sample pair. Then compare the contrastive cross-attention loss (FCL). It consists of positive sample loss and negative sample loss, where the positive sample loss is to minimize the difference distance between the two attention probability distribution maps corresponding to a positive sample pair, and the negative sample loss is to maximize the difference distance between the two attention probability distribution maps corresponding to a negative sample pair. The formula is as follows: in, This indicates that it comes from the set of positive sample pairs. Positive sample pairs in , These are two elements from a positive sample pair. This represents the difference distance between the two attention probability distribution maps corresponding to a positive sample pair. For the set of positive sample pairs Negative sample pairs in , These are two elements from a negative sample pair.

[0059] (4) Based on the contrastive cross-attention loss, the diffusion model is constrained by the guided denoising process to generate high-quality images.

[0060] During the denoising process, a latent variable update strategy is adopted: at each denoising time step, U-Net first predicts the attention probability distribution sequence, and then calculates the contrastive cross-attention loss. and take advantage of the loss gradient Image latent spatial features Perform an update to obtain the updated latent space features of the image. The updated formula is: in, To update the step size, the updated image latent space features are then used. The data is used as input to U-Net for the next step of denoising. This strategy achieves seamless integration without retraining the entire diffusion model.

[0061] In this embodiment, the spatial relationship enhancement module and the diffusion model are optimized before application. The optimization process includes: in the first half (i.e., the first 50%) of the denoising time step, the standard denoising loss of the diffusion model is used simultaneously. Simultaneously optimize the spatial relationship enhancement module and diffusion model using the contrastive cross-attention loss constructed based on positive and negative sample pairs, i.e., utilizing the total loss. To update, This is the balance coefficient, set to 0.1~0.3 by default; In the latter half (the last 50%) of the denoising time step, only the standard denoising loss is used. The spatial relationship enhancement module and diffusion model are optimized simultaneously. This training method avoids interference from high-noise stages in the later stages on contrastive learning, ensuring that attribute binding constraints effectively guide latent variable updates in the early stages of generation.

[0062] The following specific experimental examples will be given to illustrate the method of the present invention in detail.

[0063] First, for the basic model configuration, a latent space diffusion model is used as the basic architecture, preferably StableDiffusion 1.4 or Stable Diffusion 2.1. The text encoder uses CLIP ViT-L / 14, and the parameters can be frozen or fine-tuned during training. U-Net adopts the standard UNet2DConditionModel structure.

[0064] Secondly, during the training phase, image-text pairs containing spatial relationship descriptions were selected from the MS-COCO dataset; a training set containing 4,717 pairs of samples was preferably constructed. Then, a large language model was used to analyze entities and spatial relationships, and invalid samples were manually filtered out. The optimizer used was AdamW, with a learning rate set to 1×10⁻⁶. -4 Weight decay is set to ×10 -2 Batch size is 0 (single card), supports multi-card parallel training, and updates step size. Set to 0.01; Balance coefficient Set it to 0.2.

[0065] After training, during image synthesis, the inference efficiency is as follows: Generation time for a single 512×512 image: approximately 7.0~7.6 seconds based on an NVIDIA A800 GPU; an increase of approximately 4~5 seconds compared to the base model; and a speedup of approximately 22.2% compared to similar methods (such as SynGen's 9.0 seconds). Regarding parameter overhead: Additional parameters: Stable Diffusion version 1.4 adds 2.95M parameters, and version 2.1 adds 5.25M parameters. The percentage of parameters in the base model is less than 0.5%.

[0066] Example of effect verification: Taking the test sample "a pink clock and a brown chair" as an example: Basic model output: Color swap error (pink chair + brown clock). Output of this invention: Correct attribute binding (pink clock + brown chair). Quantitative metrics: In the SRA-Bench benchmark test, attribute Fidelity improved from 20.13% to 52.38%, and spatial relationship accuracy (VISOR) improved from 17.63% to 34.00%.

[0067] like Figure 3 As shown, the embodiment also provides an image synthesis system 30 based on structure awareness and fine-grained semantic alignment for a diffusion model, including a text processing unit 31 for spatial relationship enhancement and an image generation unit 32 for attribute enhancement. The text processing unit 31 for spatial relationship enhancement is used to model the spatial layout structured knowledge of the input text based on the spatial relationship enhancement module, and integrate the spatial layout structured knowledge into the embedding representation of the input text, which together serve as the guiding condition for the diffusion model. The image generation unit 32 for attribute enhancement is used to construct positive sample pairs and enhance fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module, and construct a contrastive cross-attention loss based on the positive and negative sample pairs to constrain the denoising process of the diffusion model based on the guiding condition, thereby generating a high-quality image.

[0068] It should be noted that the image synthesis system based on a diffusion model with structure awareness and fine-grained semantic alignment provided in the above embodiments should be illustrated using the above-described division of functional units as an example when performing image synthesis. The functions described above can be assigned to different functional units as needed, that is, the internal structure of the terminal or server can be divided into different functional units to complete all or part of the functions described above. Furthermore, the image synthesis system based on a diffusion model with structure awareness and fine-grained semantic alignment provided in the above embodiments and the image synthesis method based on a diffusion model with structure awareness and fine-grained semantic alignment belong to the same concept. For details of their specific implementation, please refer to the image synthesis method embodiment based on a diffusion model with structure awareness and fine-grained semantic alignment, which will not be repeated here.

[0069] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment, specifically including the following steps: S1, based on the spatial relationship enhancement module, models the spatial layout structured knowledge of the input text, integrates the spatial layout structured knowledge into the embedded representation of the input text, and uses them together as the guiding conditions for the diffusion model; S2 constructs positive sample pairs and fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's denoising process based on guiding conditions, generating high-quality images.

[0070] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment described in S1-S2 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0071] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment, specifically including the following steps: S1, based on the spatial relationship enhancement module, models the spatial layout structured knowledge of the input text, integrates the spatial layout structured knowledge into the embedded representation of the input text, and uses them together as the guiding conditions for the diffusion model; S2 constructs positive sample pairs and fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's denoising process based on guiding conditions, generating high-quality images.

[0072] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0073] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image synthesis method based on a diffusion model with structure awareness and fine-grained semantic alignment, characterized in that, Includes the following steps: Based on the spatial relationship enhancement module, the spatial layout structure knowledge of the input text is modeled and integrated into the embedded representation of the input text, which together serve as the guiding condition for the diffusion model. The attribute enhancement module constructs positive sample pairs for the input text and enhances fine-grained negative sample pairs between objects and attributes. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's guided denoising process, generating high-quality images.

2. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 1, characterized in that, Based on the spatial relationship enhancement module, the spatial layout structure knowledge of the input text is modeled, including: After extracting relation triples from the input text, a heterogeneous semantic graph is constructed based on the extracted relation triples. The nodes in the graph are divided into relation nodes and object nodes. A dedicated adjustment node is introduced for each type of relation to learn the adjustment vector of a specific relation. The edges in the graph represent the connection relationship between objects and relations. Heterogeneous graph neural networks are used to perform message passing and feature learning on heterogeneous semantic graphs to obtain spatial layout structured knowledge of the input text.

3. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 2, characterized in that, Using heterogeneous graph neural networks for message passing and feature learning on heterogeneous semantic graphs, including: The initial embeddings of object nodes and relation nodes in the heterogeneous semantic graph are initialized using pre-trained CLIP word embeddings to obtain node features; Heterogeneous graph neural networks employ the GraphSAGE architecture, which uses aggregation functions to aggregate features from neighboring nodes onto the adjustment node. After layer graph propagation, each relation type adjustment node learns an adjustment vector containing contextual information as structured knowledge of spatial layout.

4. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 1, characterized in that, Integrating structured knowledge of spatial layout into the embedded representation of input text includes: After weighted summation of the adjustment vectors of all relation types corresponding to the adjustment nodes, this summation is added to the encoded vector of the input text in the text encoder to obtain the final text embedding representation rich in spatial layout structured knowledge. The final text embedding representation serves as the guiding condition for the diffusion model.

5. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 1, characterized in that, The attribute enhancement module constructs positive sample pairs for the input text and enhances fine-grained negative sample pairs between objects and attributes, including: Perform syntactic dependency parsing on the input text, identify object-attribute dependency pairs, construct positive sample pairs, and represent the correct attribute binding relationships; By arranging attributes, objects are paired with non-corresponding attributes to construct negative sample pairs, and / or by comparing features between different objects to construct negative sample pairs, thereby enhancing the model's ability to distinguish different objects.

6. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 1, characterized in that, The contrastive cross-attention loss, constructed based on positive and negative sample pairs, constrains the denoising process of the diffusion model based on guiding conditions, including: In the cross-attention layer of the U-net network in the diffusion model, the latent spatial features Z of the image output at the previous time step are... t To query matrix Q, using the text embedding representation as the guiding condition as the key matrix K and the value matrix V, we calculate the attention probability distribution map of the association weight between each token in the text embedding representation and the spatial location of the image, and obtain the attention probability distribution map sequence. For each sample pair, extract two attention probability distribution maps from the attention probability distribution map sequence for each element in the sample pair corresponding to the token. and ; Using symmetric KL divergence to measure the difference distance between the two attention probability distribution maps for each sample pair, the contrastive cross-attention loss consists of minimizing the difference distance between the two attention probability distribution maps corresponding to positive sample pairs and maximizing the difference distance between the two attention probability distribution maps corresponding to negative sample pairs. Using contrastive cross-attention loss gradient Image latent spatial features Perform an update to obtain the updated latent space features of the image. The updated formula is: in, To update the step size.

7. The diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment according to claim 1, characterized in that, The spatial relationship enhancement module and diffusion model underwent parameter optimization before application. The optimization process included: In the first half of the denoising time step, the spatial relationship enhancement module and the diffusion model are simultaneously optimized using the standard denoising loss of the diffusion model and the contrastive cross-attention loss constructed based on positive and negative sample pairs. In the latter half of the denoising time step, only the standard denoising loss is used to simultaneously optimize the spatial relation enhancement module and the diffusion model.

8. An image synthesis system based on a diffusion model with structure awareness and fine-grained semantic alignment, characterized in that, include: The spatial relation enhancement text processing unit is used to model the spatial layout structured knowledge of the input text based on the spatial relation enhancement module, and integrates the spatial layout structured knowledge into the embedded representation of the input text, which together serve as the guiding condition for the diffusion model. The attribute-enhanced image generation unit is used to construct positive sample pairs and enhance fine-grained negative sample pairs between objects and attributes for the input text based on the attribute enhancement module. Based on the positive and negative sample pairs, a contrastive cross-attention loss is constructed to constrain the diffusion model's guided denoising process, thereby generating high-quality images.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the diffusion model image synthesis method based on structure awareness and fine-grained semantic alignment as described in any one of claims 1-7.

Images