Diffusion model image generation semantic alignment method and system based on double loss optimization

Abstract

This invention belongs to the field of image generation technology, specifically relating to a semantic alignment method and system for image generation based on a diffusion model with dual-loss optimization. The method includes: extracting cross-attention maps corresponding to each topic word in the text prompt during denoising; calculating aggregation loss and isolation loss based on the cross-attention maps, whereby the aggregation loss aggregates attention activations corresponding to the same topic word, and the isolation loss separates attention activations corresponding to different topic words; constructing a joint loss function based on the aggregation loss and isolation loss, and updating the latent representation along the gradient descent direction; inputting the updated latent representation into the denoising process at the next time step, until a preset number of denoising steps is reached, generating the output image. This invention optimizes the aggregation loss and isolation loss, simultaneously solving the problems of internal scattering and overlap in the attention maps, and can be applied to downstream tasks such as controllable layout generation and personalized generation.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology, specifically relating to a semantic alignment method and system for image generation based on a diffusion model with dual-loss optimization. Background Technology

[0002] Text-to-image synthesis aims to generate semantically correct visual images based on given text descriptions. In recent years, diffusion models have become the mainstream approach in this field due to their powerful generative capabilities. In latent diffusion models, the cross-attention mechanism is a key module connecting text prompts and the image generation process. It guides the denoising process by calculating the association score between text tags and the latent image representation (i.e., the cross-attention map).

[0003] However, existing text-to-image synthesis methods based on diffusion models still face significant problems of inaccurate text-image alignment, mainly manifested in two aspects: (1) Internal scattering: attention activations corresponding to the same topic word (e.g., "rabbit") are spatially dispersed at different locations in the attention map, resulting in fragmented or incorrectly placed object parts (e.g., the rabbit's ears are on the bird's body). (2) Mutual overlap: attention activations corresponding to different topic words (e.g., "cat" and "turtle") overlap excessively in space, resulting in mixed objects or mismatched attributes (e.g., the cat and turtle are merged into one). Existing technologies attempt to address these problems. For example, some methods focus on reducing overlap by maximizing the cosine similarity or KL divergence between attention maps of different topic words, but ignore the internal scattering problem. Other methods attempt to alleviate scattering by maintaining the consistency of attention maps between consecutive denoising steps, but this method heavily relies on the selection of initial noise, lacks stability, and fails to fundamentally model the spatial aggregation relationship of activation values. Therefore, there is an urgent need for an optimization method that can simultaneously and effectively solve the problems of internal scattering and mutual overlap in cross-attention maps to achieve more accurate and reliable text-to-image synthesis. Summary of the Invention

[0004] The purpose of this invention is to provide a semantic alignment method and system for image generation based on a diffusion model with dual loss optimization, so as to solve the problems of internal scattering and mutual overlap of cross attention maps in existing text-to-image generation methods, and improve the semantic alignment accuracy between generated images and text descriptions.

[0005] The present invention achieves the above objectives through the following technical solutions: Firstly, this invention proposes a semantic alignment method for image generation based on a diffusion model with dual-loss optimization, applied to text-to-image generation tasks using a diffusion model. The method includes: According to the preset number of denoising steps, during the denoising process of the diffusion model, the cross-attention map corresponding to each topic word in the text prompt is extracted; Based on the cross-attention map, the aggregation loss and the isolation loss are calculated. The aggregation loss is used to aggregate attention activations corresponding to the same topic word, and the isolation loss is used to separate attention activations corresponding to different topic words. A joint loss function is constructed based on the aggregation loss and the isolation loss. The loss gradient of the joint loss function with respect to the latent representation at the current time step is calculated, and the latent representation is updated along the gradient descent direction. The updated latent representation is input into the denoising process at the next time step until the preset number of denoising steps is reached, and then the output image is generated by the decoder.

[0006] Furthermore, before extracting the cross-attention map corresponding to each topic word in the text prompt, the method further includes: Initial latent noise is sampled from a standard Gaussian distribution and used as the latent representation for the first denoising time step; The text prompt is input into the encoder of the diffusion model to obtain the text embedding; The text embedding is used to compute the cross-attention map during the denoising process, and the latent representation includes the initial latent noise and the updated latent representation during the denoising process.

[0007] Furthermore, the calculation of the aggregation loss includes: For any topic word, the position with the largest activation value in the cross-attention map is determined as the first center point. The first circular region is divided with the first center point as the center and a preset radius as the first grouping region; Outside the first circular region, the location with the highest activation value in the remaining region is determined as the second center point. The second circular region is divided with the second center point as the center and the preset radius as the second group region. This process is repeated until the preset number of group regions is reached. Calculate the weighted centroid of all activation values ​​within each group region; The aggregation loss is obtained by calculating the sum of the spatial distances between each pair of weighted centroids in all grouped regions.

[0008] Furthermore, the weighted center point for calculating the sum of all activation values ​​within each grouped region is... We obtain the following formula: ; in, ; ; in, For the index of keywords, and For the first The two-dimensional spatial coordinates of the weighted centroids of each grouped region on the cross-attention map. For the first Grouped areas For the first The first circular grouping area Activator The value, and For the first The first circular grouping area Activator Two-dimensional spatial coordinates on the cross-attention map; The aggregation loss is obtained by the following formula: ; in, This represents the Euclidean distance metric function. and For indexes of circular grouped regions, This indicates the total number of grouped regions.

[0009] Furthermore, the isolation loss is calculated, including: Cross-attention graphs at time t for any two distinct keywords and First, calculate the center point of the overall weighting for each: ; in, Indicates a subject index. Indicates the center of gravity. and The two-dimensional spatial coordinates of the centroid are calculated using the following formula: ; ; in, The activation at index j in the cross-attention graph The value; Calculate the Euclidean distance between two centroids As shown in the following formula: ; in, This represents the Euclidean distance metric function. and An index representing the subject terms; Based on the size of the cross-attention map, After normalization, the negative correlation function of the spatial distance is used as the isolation loss component corresponding to the two different keywords to obtain the isolation loss function: ; in , where is the diagonal length of the attention map, and W and H are the width and height of the attention map, respectively.

[0010] Furthermore, constructing the joint loss function based on the aggregation loss and the isolation loss includes: Obtain an auxiliary loss, which is used to enhance the maximum activation value in the cross-attention map; The weighted sum of the aggregation loss, the isolation loss, and the auxiliary loss yields the joint loss function, as shown in the following equation: ; in, These are the weight coefficients of their respective loss functions. For aggregation loss, To isolate the losses, To mitigate losses; The auxiliary loss is obtained by the following formula: .

[0011] Further, the step of calculating the loss gradient of the joint loss function with respect to the latent representation at the current time step, and updating the latent representation along the gradient descent direction, includes: Calculate the joint loss function with respect to the latent representation z at the current time step. t loss gradient ; The potential representation z of the current time step is given by the following formula. t An update will be made: ; in, This represents the updated potential representation. Indicates the step size.

[0012] Furthermore, the text-to-image generation task applied to the diffusion model also includes a controllable layout generation task or a personalized generation task. When applied to a controllable layout generation task, the method further includes obtaining the bounding box corresponding to the text prompt, and the aggregation loss is also used to guide the attention activation region of the topic words to concentrate inside the bounding box; In the personalized generation task, the text prompt includes a special marker, which corresponds to a preset visual concept, and the embedding of the special marker is determined through prior concept learning; extracting the cross-attention map includes extracting the cross-attention map corresponding to the special marker; the aggregation loss is used to constrain the attention activations corresponding to the special marker to cluster spatially.

[0013] Secondly, this invention proposes a diffusion model-based image generation semantic alignment system based on dual-loss optimization, used to implement the above-described semantic alignment method. The system includes: The extraction module is used to extract the cross-attention map corresponding to each topic word in the text prompt during the denoising process of the diffusion model, according to a preset number of denoising steps. The calculation module is used to calculate the aggregation loss and the isolation loss based on the cross-attention map. The aggregation loss is used to aggregate attention activations corresponding to the same topic word, and the isolation loss is used to separate attention activations corresponding to different topic words. The update module is used to construct a joint loss function based on the aggregation loss and the isolation loss, calculate the loss gradient of the joint loss function with respect to the potential representation at the current time step, and update the potential representation along the gradient descent direction; The generation module is used to input the updated latent representation into the denoising process of the next time step until the preset number of denoising steps is reached, and then generate the output image through the decoder.

[0014] Furthermore, the extraction module is also used to sample initial latent noise from a standard Gaussian distribution and use the initial latent noise as the latent representation of the first denoising time step; input the text prompt into the encoder of the diffusion model to obtain a text embedding; wherein the text embedding is used to calculate the cross-attention map during the denoising process, and the latent representation includes the initial latent noise and the latent representation updated during the denoising process.

[0015] The beneficial effects of this invention are as follows: This invention simultaneously optimizes aggregation and isolation losses, updating the latent representation with gradients in the early stages of denoising the diffusion model. This spatially clusters attention activations for the same topic and spatially separates attention activations for different topics, thereby correcting the internal scattering and overlap issues in the cross-attention map. This method requires no retraining or fine-tuning of the base diffusion model, can be directly integrated into diffusion models with various architectures, and can be extended to downstream tasks such as controlled layout generation and personalized generation. It achieves simultaneous optimization of generation quality and semantic alignment while introducing acceptable computational overhead. Attached Figure Description

[0016] Figure 1A flowchart of an image generation semantic alignment method based on a diffusion model with dual-loss optimization provided by the present invention; Figure 2 This is another flowchart of the semantic alignment method for image generation based on a diffusion model with dual-loss optimization provided by the present invention; Figure 3 The system block diagram of the image generation semantic alignment system based on the diffusion model with dual loss optimization provided by the present invention is shown. Detailed Implementation

[0017] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content. Example 1

[0018] See Figure 1 and Figure 2 This invention proposes a semantic alignment method for image generation based on a diffusion model with dual-loss optimization, applied to text-to-image generation tasks using a diffusion model. In this embodiment, the diffusion model employs a latent diffusion model with a total denoising step count T=50. The alignment method of this invention is executed in the first 25 steps of the denoising process, while standard denoising is performed in the subsequent 25 steps. The reason for selecting the first 25 steps for optimization is that during the image generation process of the diffusion model, the overall layout and object structure of the image are mainly determined in the early and middle stages of denoising (i.e., the high-noise-level stage). Guiding the latent representation at this stage can more effectively correct the internal scattering and overlap problems of the attention map from the root, thereby laying a clear semantic layout foundation.

[0019] The method includes the following steps: Step S1: According to the preset number of denoising steps, extract the cross-attention map corresponding to each topic word in the text prompt during the denoising process of the diffusion model.

[0020] Specifically, the initialization operation is performed first: initial latent noise is sampled from a standard Gaussian distribution, and this initial latent noise is used as the latent representation of the first denoising time step (t=T=50). Simultaneously, the text prompts are input into the encoder of the diffusion model to obtain text embeddings. These text embeddings are used to compute cross-attention maps during denoising, and the latent representations include the initial latent noise and the updated latent representations during denoising.

[0021] At the t-th denoising iteration, the denoising network receives the latent representation. At time step t and with the text embedding, a forward propagation is performed. Simultaneously, cross-attention maps corresponding to each topic word in the text prompt are extracted from the model's cross-attention layer. For example, for the text prompt "an apple and a bird," cross-attention maps corresponding to the topic words "apple" and "bird" are extracted. Each cross-attention map is a two-dimensional matrix, where each element (activation value) represents the association strength between a spatial location in the image's latent representation and the text embedding of that topic word.

[0022] Step S2: Based on the cross-attention map, calculate the aggregation loss and the isolation loss. The aggregation loss is used to aggregate attention activations corresponding to the same topic word, and the isolation loss is used to separate attention activations corresponding to different topic words.

[0023] (1) Calculation of polymerization loss The aggregation loss aims to address the problem of internal scattering, which forces scattered attentional responses under the same topic to converge in space, resulting in a more complete and compact object structure.

[0024] For any given keyword in the cross-attention map, the position with the highest activation value is first determined as the first center point. A first circular region with a preset radius r, centered on this first center point, is then divided into the first grouping region R1. The preset radius r is a positive integer, calculated in units of the spatial units of the cross-attention map; for example, r = 3.

[0025] Outside the first grouped region, determine the location with the highest activation value in the remaining region as the second center point. Divide the area into a second circular region with the same preset radius r centered on the second center point, as the second grouped region R2. Repeat this process until the preset number of grouped regions N is reached. The number of grouped regions N can be set according to the proportion of the area occupied by the keyword in the image. For example, for "apple" and "bird", N=3 and N=2 can be set respectively.

[0026] Then, calculate the weighted centroid of all activation values ​​within each group region. We obtain the following formula: ; in, ; ; in, For the index of keywords, and For the first The two-dimensional spatial coordinates of the weighted centroids of each grouped region on the cross-attention map. For the first Grouped areas For the first The first circular grouping area Activator The value, and For the first The first circular grouping area Activator Two-dimensional spatial coordinates on the cross-attention map; Finally, the sum of the Euclidean distances between each pair of weighted centroids in all grouped regions is calculated to obtain the aggregate loss: ; in, This represents the Euclidean distance metric function. and For indexes of circular grouped regions, This indicates the total number of grouped regions.

[0027] Minimizing this loss will drive these centroids to move closer to each other, thereby spatially aggregating the attentional response of the keyword, resulting in a complete and compact structure of the generated object components.

[0028] (2) Calculation of isolation loss Isolation loss aims to address the problem of mutual overlap, that is, to force the attention response areas of different keywords to be spatially separated, avoiding object mixing or attribute mismatch.

[0029] For multiple keywords in the text prompt, calculate the overall weighted centroid of the cross-attention map for each keyword. Taking the s-th keyword as an example, the formula for calculating the coordinates of its overall weighted centroid is as follows: ; in, Indicates a subject index. Indicates the center of gravity. and The two-dimensional spatial coordinates of the centroid are calculated using the following formula: ; ; in and These represent the x and y coordinates, respectively. Indicates a subject index. The activation at index j in the cross-attention graph The value of .

[0030] Cross-attention graphs at time t for any two distinct keywords and Calculate the Euclidean distance between the centroids of the cross-attention maps corresponding to each keyword. As shown in the following formula: ; in, This represents the Euclidean distance metric function. and Index representing subject terms, Keywords The coordinates of the centroid of the corresponding cross attention map.

[0031] Finally, based on the size of the cross-attention map, After normalization, the negative correlation function of the spatial distance is used as the isolation loss component corresponding to the two different keywords, and the isolation loss function value between the two cross-attention maps is obtained as follows: ; in , where is the diagonal length of the attention map, and W and H are the width and height of the attention map, respectively. When the text prompt contains more than two keywords, the sum of the distances between all different combinations of keywords in the text prompt is used: ; in It is the sum of the distances between the weighted centers of all cross-attention maps. It is the number of keywords. and This is the keyword index. The final isolation loss is as follows: ; By minimizing the isolation loss, the spatial distance between the attention activation regions of different topics can be maximized, thereby effectively pushing the activation regions of different topics further apart and avoiding their overlap.

[0032] Step S3: Construct a joint loss function based on the aggregation loss and isolation loss, calculate the loss gradient of the joint loss function with respect to the latent representation at the current time step, and update the latent representation along the gradient descent direction.

[0033] First, an auxiliary loss is obtained, which is used to enhance the maximum activation value in the cross-attention graph, as follows: ; in, This represents the maximum activation value in the cross-attention map. Minimizing this loss helps to enhance the most significant attention response, making the main activation regions corresponding to the topic words more prominent.

[0034] Then, the polymerization loss The aforementioned isolation loss and the auxiliary loss The joint loss function is obtained by performing a weighted summation. As shown in the following formula: ; in, These are the weight coefficients for their respective loss functions, used to balance the impact of different loss terms on the optimization process. In this embodiment, The value is 1.25. The value is 2. The value is 0.25. It should be noted that the specific values ​​of the above weighting coefficients are not unique and fixed. Those skilled in the art can adaptively adjust them according to actual task requirements, dataset size, or model complexity. For example, [the value could be adjusted]. Any value set within the range of 0.75 to 1.5 falls within the protection scope of this invention.

[0035] Next, using the backpropagation algorithm, the joint loss function is calculated with respect to the latent representation z at the current time step. t loss gradient ; Finally, the potential representation z of the current time step is expressed according to the following formula. t An update will be made: ; in, This represents the updated potential representation. Indicates step size, step size It can be determined based on the preset step size corresponding to time step t in the diffusion model.

[0036] Step S4: Input the updated latent representation into the denoising process of the next time step until the preset number of denoising steps is reached, and generate the output image through the decoder.

[0037] The updated latent representation z{t-1} is input into the standard denoising process at the next time step (t-1), and steps S1-S3 are repeated until the preset number of optimization steps is reached. In this embodiment, the number of optimization steps is set to the first 25 steps, that is, the above optimization steps are performed when t decreases from 50 to 26, and standard denoising is only performed in subsequent time steps (t decreases from 25 to 1).

[0038] After denoising, the final latent representation z0 is obtained, and z0 is reconstructed into the final output image by the decoder of the diffusion model.

[0039] The above method guides the latent representation in the early stages of denoising, causing attention activations for the same topic to spatially cluster and attention activations for different topic terms to spatially separate, thereby fundamentally correcting the internal scattering and overlap problems in the cross-attention map. This method does not require retraining or fine-tuning of the basic diffusion model and can be directly integrated into diffusion models with various architectures, achieving simultaneous optimization of generation quality and semantic alignment with acceptable computational overhead.

[0040] As an optional solution, this method can also be extended to controllable layout generation tasks and personalized generation tasks, as follows: (a) Applied to controllable layout generation tasks When this method is applied to controlled layout generation tasks, it is used in conjunction with existing controlled layout generation models (such as GLIGEN). Specifically, in addition to the text prompt, the bounding boxes corresponding to each topic word are additionally obtained as layout control conditions. For example, for the text prompt "a pear and a banana", two specified bounding boxes are obtained simultaneously to constrain the position and extent of "pear" and "banana" in the generated image, respectively.

[0041] In the early stages of the denoising process (such as the first 25 steps), the cross-attention maps corresponding to "pear" and "banana" are extracted in the same manner as in Example 1. When calculating the aggregation loss, in addition to aggregating attention activations within the same topic term, the aggregation loss is also used to guide the attention activation regions of the topic terms to concentrate within the corresponding bounding boxes. Specifically, when calculating the aggregation loss, the weighted center points of the grouped regions are constrained within the bounding boxes, so that the high-response regions of attention activation are concentrated within the space defined by the bounding boxes.

[0042] Through the above optimizations, the aggregation loss implicitly guides the high-response regions of the "pear" and "banana" to concentrate within their respective bounding boxes, while the isolation loss prevents the attention regions of the two from overlapping. Compared with the baseline method, this method effectively reduces erroneous generated objects outside the bounding boxes, making the generated objects better conform to the specified layout and improving the accuracy and stability of controllable layout generation.

[0043] (ii) Application to personalized generation tasks When applied to personalized generation tasks, this method is used in conjunction with existing personalized generation methods such as text inversion or DreamBooth. Specifically, the text prompt contains special markers that represent a predefined visual concept. For example, the text prompt might be "A [V] dog is in the garden," where "[V]" is a special marker representing a specific visual concept (such as a particular dog breed). The embedding of this special marker is not obtained directly from a standard text encoder, but is predetermined through a pre-concept learning process. That is, using a small number of reference images containing the visual concept, feature embeddings that accurately represent the visual concept are learned.

[0044] In the early stages of the denoising process, the cross-attention map corresponding to the special marker "[V]" is extracted, along with cross-attention maps corresponding to other common keywords (such as "dog" and "garden"). When calculating the aggregation loss, it constrains the spatial clustering of attention activations corresponding to the special marker, ensuring that the visual concept it represents is cohesive and fully presented in the generated image, rather than fragmented or distorted. For example, for the specific dog breed represented by "[V]", the aggregation loss prompts the attention activations corresponding to its ears, body, limbs, and other parts to move closer together, forming a complete image of the dog.

[0045] Meanwhile, isolation loss is used to prevent the visual concept of the special marker from being inappropriately mixed with other elements in the textual cues, such as the "garden" background. Specifically, by maximizing the spatial distance between the overall weighted center of gravity of the special marker and other topic words, isolation loss spatially separates the attention area of ​​the dog represented by "[V]" from the attention area of ​​the "garden" background, avoiding the dog blending into the background or attribute mismatch.

[0046] Through the optimizations described above, this method enables the generated images to better preserve the integrity of the learned concepts while accurately placing the learned concepts within the target scene. For example, for a specific dog breed learned, it can generate dogs that retain the unique appearance characteristics of that breed and place them naturally in a garden scene without issues such as loss of dog features or blending into the background.

[0047] As an example, this embodiment describes in detail the execution flow of the method of the present invention within one denoising time step as follows: 1. Input and Initialization: The input text prompt is "an apple and a bird". The diffusion model encoder converts the prompt into a text embedding. Initial latent noise is sampled from a standard Gaussian distribution. .

[0048] 2. Iterative denoising and optimization (for t from T to 1): (1) Forward propagation and attention extraction: During the t-th denoising process, the denoising network Receive potential representation At time step t and text embedding, a forward propagation is performed. Simultaneously, cross-attention maps corresponding to the keywords "apple" and "bird" are extracted from the model's cross-attention layer. ; (2) Calculate the aggregation loss: For cross-attention maps First, find its largest activation unit. With it as the center, the radius Divide into a circular aggregation area .in, The value is a positive integer, expressed as the number of spatial units in the feature map. Similarly, the above operation is repeated outside this region until the preset maximum number of grouped regions is reached. The series of aggregated regions obtained are denoted as In this embodiment, the area ratios of the "apple" and the "bird" in the image are set respectively. , , , Then, calculate the weighted centroid coordinates for each grouped region using the following formula. For the i-th grouped region, the centroid coordinates are marked as follows: ,in and This represents the two-dimensional spatial coordinates constructed based on the pixel indices within the cross-attention map. The specific calculation process is shown in the following formula: ; ; Finally, the sum of the Euclidean distances between each pair of centroids in all grouped regions is calculated to obtain the value of the aggregation loss function. The specific calculation process is as follows: ; Cross-attention maps for "apple" and "bird" After performing the above operations on each, obtain their respective aggregate loss function values. .

[0049] Final total theme aggregation loss The sum of the two: Optimization can enhance the aggregation of activation element distributions within the cross-attention maps of "apple" and "bird"; (3) Calculate the isolation loss: First, calculate the isolation loss separately. Each of their overall weighting centers Then calculate the distance between the weighted centers of the overall activation values ​​of the two attention maps. The specific formula is as follows: ; Finally, based on the size of the cross-attention map, Normalization is performed to obtain the isolation loss function value. .in , where is the length of the diagonal of the attention map, and W and H are the width and height of the attention map, respectively. This is used to normalize the distance d to the interval [0, 1], thereby ensuring the loss function... The value range is between [0, 1], which facilitates balancing with other loss terms. Through optimization... This can increase the degree of isolation in the spatial distribution of activation elements in the cross-attention maps corresponding to "apple" and "bird".

[0050] (4) Calculate the auxiliary loss function: ; (5) Based on the above loss values, the final joint loss is formed: , in These are the weight coefficients of their respective loss functions.

[0051] (6) Gradient update: Calculate the gradient And use the preset step size corresponding to time step t in the diffusion model. Update potential representation: .

[0052] (7) Continue noise reduction: update the noise reduction process. Input the standard denoising process for the next time step (t-1) and repeat steps (1) to (6) until the predetermined number of optimization steps is reached (set to 25 steps in this embodiment). Subsequent time steps only perform standard denoising.

[0053] 3. Image Decoding: After denoising, the latent representation is obtained, and it is reconstructed into the final output image by the decoder of the diffusion model. Example 2

[0054] See Figure 3 A specific embodiment of the present invention proposes an image generation semantic alignment system based on a dual-loss optimization diffusion model, used to implement the semantic alignment method described in Embodiment 1. The system includes: The extraction module is used to extract the cross-attention map corresponding to each topic word in the text prompt during the denoising process of the diffusion model, according to a preset number of denoising steps. Specifically, the extraction module is also used to sample initial latent noise from a standard Gaussian distribution, using the initial latent noise as the latent representation for the first denoising time step; and to input the text prompt into the encoder of the diffusion model to obtain the text embedding. The text embedding is used to calculate the cross-attention map during the denoising process, and the latent representation includes the initial latent noise and the updated latent representation during the denoising process.

[0055] The calculation module is used to calculate the aggregation loss and isolation loss based on the cross-attention map. The aggregation loss is used to aggregate attention activations corresponding to the same topic, and the isolation loss is used to separate attention activations corresponding to different topics. The specific calculation methods of the aggregation loss and isolation loss are as described in step S2 of Example 1.

[0056] The update module is used to construct a joint loss function based on the aggregation loss and isolation loss, calculate the loss gradient of the joint loss function with respect to the latent representation at the current time step, and update the latent representation along the gradient descent direction. The construction of the joint loss function and the gradient update are performed as described in step S3 of Example 1.

[0057] The generation module is used to input the updated latent representation into the denoising process of the next time step until the preset number of denoising steps is reached, and then the decoder generates the output image.

[0058] In practical applications, the diffusion model-based image generation semantic alignment system, optimized by dual loss, can be integrated into the diffusion model architecture and deployed as a plug-and-play optimization module on image generation servers or terminal devices. During use, the user inputs text prompts, and the system extracts the text embedding and initializes latent noise through the extraction module. At each time step before denoising, the computation module extracts the cross-attention maps of each topic word in real time and calculates the aggregation loss and isolation loss. The update module performs single-step gradient updates on the latent representation based on the joint loss function, guiding the spatial distribution of attention activation. After iteration, the generation module outputs an image highly aligned with the text description. This system requires no retraining or fine-tuning of the basic diffusion model and can be directly applied to scenarios such as standard text-to-image generation, controllable layout generation, and personalized generation, optimizing semantic alignment while maintaining the original generation speed.

[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0060] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0061] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A diffusion model image generation semantic alignment method based on double loss optimization, characterized in that, The method, applied to a diffusion model for text-to-image generation tasks, includes: According to the preset number of denoising steps, during the denoising process of the diffusion model, the cross-attention map corresponding to each topic word in the text prompt is extracted; Based on the cross-attention map, the aggregation loss and the isolation loss are calculated. The aggregation loss is used to aggregate attention activations corresponding to the same topic word, and the isolation loss is used to separate attention activations corresponding to different topic words. A joint loss function is constructed based on the aggregation loss and the isolation loss. The loss gradient of the joint loss function with respect to the latent representation at the current time step is calculated, and the latent representation is updated along the gradient descent direction. The updated latent representation is input into the denoising process at the next time step until the preset number of denoising steps is reached, and then the output image is generated by the decoder.

2. The diffusion model image generation semantic alignment method based on double loss optimization according to claim 1, characterized in that, Before extracting the cross-attention map corresponding to each topic word in the text prompt, the method further includes: Initial latent noise is sampled from a standard Gaussian distribution and used as the latent representation for the first denoising time step; The text prompt is input into the encoder of the diffusion model to obtain the text embedding; The text embedding is used to compute the cross-attention map during the denoising process, and the latent representation includes the initial latent noise and the updated latent representation during the denoising process.

3. The diffusion model image generation semantic alignment method based on double loss optimization according to claim 1, characterized in that, The calculation of aggregation loss includes: For any topic word, the position with the largest activation value in the cross-attention map is determined as the first center point. The first circular region is divided with the first center point as the center and a preset radius as the first grouping region; Outside the first circular region, the location with the highest activation value in the remaining region is determined as the second center point. The second circular region is divided with the second center point as the center and the preset radius as the second group region. This process is repeated until the preset number of group regions is reached. Calculate the weighted centroid of all activation values ​​within each group region; The aggregation loss is obtained by calculating the sum of the spatial distances between each pair of weighted centroids in all grouped regions.

4. The semantic alignment method for image generation based on a diffusion model with dual-loss optimization according to claim 3, characterized in that, said calculation of a weighted barycenter point of all active values within each grouping area is obtained by the following equation: ； in, ； ； wherein, an index of the topic, with a first weighting centroid point in two-dimensional space on the cross-attention map for the th grouping area, a first circular grouping area, value of the th active element in the th circular grouping area, two-dimensional space coordinates on the cross-attention map for the th active element in the th circular grouping area; The aggregation loss is obtained by the following formula: ； in, This represents the Euclidean distance metric function. and For indexes of circular grouped regions, This indicates the total number of grouped regions.

5. The semantic alignment method for image generation based on a diffusion model with dual-loss optimization according to claim 4, characterized in that, Calculate the isolation loss, including: Cross-attention graphs at time t for any two distinct keywords and First, calculate the center point of the overall weighting for each: ； in, Indicates a subject index. Indicates the center of gravity. and The two-dimensional spatial coordinates of the centroid are calculated using the following formula: ； ； in, The activation at index j in the cross-attention graph The value; Calculate the Euclidean distance between two centroids As shown in the following formula: ； in, This represents the Euclidean distance metric function. and An index representing the subject terms; Based on the size of the cross-attention map, After normalization, the negative correlation function of the spatial distance is used as the isolation loss component corresponding to the two different keywords to obtain the isolation loss function: in , where is the diagonal length of the attention map, and W and H are the width and height of the attention map, respectively.

6. The semantic alignment method for image generation based on a diffusion model with dual-loss optimization according to claim 5, characterized in that, The step of constructing a joint loss function based on the aggregation loss and the isolation loss includes: Obtain an auxiliary loss, which is used to enhance the maximum activation value in the cross-attention map; The weighted sum of the aggregation loss, the isolation loss, and the auxiliary loss yields the joint loss function, as shown in the following equation: ； in, These are the weight coefficients of their respective loss functions. For aggregation loss, To isolate the losses, To mitigate losses; The auxiliary loss is obtained by the following formula: 。 7. The semantic alignment method for image generation based on a diffusion model with dual-loss optimization according to claim 1, characterized in that, The step of calculating the loss gradient of the joint loss function with respect to the latent representation at the current time step, and updating the latent representation along the gradient descent direction, includes: Calculate the joint loss function with respect to the latent representation z at the current time step. t loss gradient ; According to the following formula for the latent representation z of the current time step t One update is made: ； in, This represents the updated potential representation. Indicates the step size.

8. The semantic alignment method for image generation based on a diffusion model with dual-loss optimization according to claim 1, characterized in that, The text-to-image generation task applied to the diffusion model also includes a controllable layout generation task or a personalized generation task. When applied to a controllable layout generation task, the method further includes obtaining the bounding box corresponding to the text prompt, and the aggregation loss is also used to guide the attention activation region of the topic words to concentrate inside the bounding box; In the personalized generation task, the text prompt includes a special marker, which corresponds to a preset visual concept, and the embedding of the special marker is determined through prior concept learning; extracting the cross-attention map includes extracting the cross-attention map corresponding to the special marker; the aggregation loss is used to constrain the attention activations corresponding to the special marker to cluster spatially.

9. A semantic alignment system for image generation based on a diffusion model with dual-loss optimization, used to implement the semantic alignment method as described in any one of claims 1-8, characterized in that, The system includes: The extraction module is used to extract the cross-attention map corresponding to each topic word in the text prompt during the denoising process of the diffusion model, according to a preset number of denoising steps. The calculation module is used to calculate the aggregation loss and the isolation loss based on the cross-attention map. The aggregation loss is used to aggregate attention activations corresponding to the same topic word, and the isolation loss is used to separate attention activations corresponding to different topic words. The update module is used to construct a joint loss function based on the aggregation loss and the isolation loss, calculate the loss gradient of the joint loss function with respect to the potential representation at the current time step, and update the potential representation along the gradient descent direction; The generation module is used to input the updated latent representation into the denoising process of the next time step until the preset number of denoising steps is reached, and then generate the output image through the decoder.

10. The image generation semantic alignment system based on dual-loss optimization using a diffusion model according to claim 9, characterized in that, The extraction module is further configured to sample initial latent noise from a standard Gaussian distribution and use the initial latent noise as the latent representation of the first denoising time step; input the text prompt into the encoder of the diffusion model to obtain a text embedding; wherein the text embedding is used to calculate the cross-attention map during the denoising process, and the latent representation includes the initial latent noise and the latent representation updated during the denoising process.

Images