Multi-subject image generation method based on text description

Abstract

The embodiment of the invention provides a multi-agent image generation method based on text description, and belongs to the technical field of data processing, and the method specifically comprises the steps: 1, obtaining the text description inputted by a user, and managing agent interaction and task distribution through a multi-agent coordinator; 2, analyzing the text description by a multi-modal deep understanding module, and extracting semantic information; 3, the hierarchical layout planning agent generates an initial layout according to the semantic information; 4, a graph neural network relation modeling module establishes and updates a scene graph according to the initial layout; 5, the reinforcement learning layout optimization agent iteratively optimizes the layout through a strategy gradient method; and step 6, the adaptive diffusion sampling module generates a multi-subject image according to the optimized layout. Through the scheme disclosed by the invention, the image generation efficiency, accuracy and adaptability are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of data processing technology, and in particular to a method for generating multi-subject images based on text description. Background Art

[0002] In recent years, text-to-image generation technology has made significant progress. Here is a detailed analysis of the current state of the art:

[0003] 1. Diffusion models: such as Stab le Diffusion, DALL-E, and Midjourney, etc., perform well in image generation of a single subject or simple scene. However, these models still face challenges in dealing with complex multi-subject scenes, especially in maintaining consistency between subjects and accurately controlling layout.

[0004] 2. Control technology: Methods such as Control Net guide image generation by introducing additional conditional information (such as edge maps, postures, etc.). However, these methods often require additional inputs and cannot directly infer complex layout information from text.

[0005] 3. Layout-based methods: Some studies have attempted to guide image generation through pre-defined layouts. However, these methods usually require human intervention to create the initial layout and are difficult to automatically process complex text descriptions.

[0006] 4. Large language model assistance: Recent studies such as LLM-grounded Diffusion attempt to use large language models to enhance text understanding. However, these methods mainly focus on text interpretation rather than precise layout control.

[0007] In summary, the limitations of existing technologies are: difficult to handle complex multi-agent scenes; lack of in-depth understanding and modeling of the relationship between subjects; layout control is not precise enough to meet the specific requirements of users; the generation process lacks explainability and controllability; it is difficult to maintain consistency between multiple images, especially in continuous creation.

[0008] It can be seen that there is an urgent need for a multi-subject image generation method based on text description with high generation efficiency, accuracy and adaptability. Summary of the invention

[0009] In view of this, an embodiment of the present disclosure provides a multi-subject image generation method based on text description, which at least partially solves the problems of poor generation efficiency, accuracy and adaptability in the prior art.

[0010] The present disclosure provides a multi-agent image generation method based on text description, which is applied to a multi-agent automatic layout generation system. The multi-agent automatic layout generation system includes a multimodal deep understanding module, a hierarchical layout planning agent, a graph neural network relationship modeling module, a reinforcement learning layout optimization agent, an adaptive diffusion sampling module and a multi-agent coordinator. The method includes:

[0011] Step 1: Get the text description input by the user, and the multi-agent coordinator manages the agent interaction and task allocation;

[0012] Step 2: The multimodal deep understanding module parses the text description and extracts semantic information;

[0013] Step 3, the hierarchical layout planning agent generates an initial layout based on the semantic information;

[0014] Step 4: The graph neural network relationship modeling module establishes and updates the scene graph according to the initial layout;

[0015] Step 5: The reinforcement learning layout optimization agent iteratively optimizes the layout through the policy gradient method;

[0016] Step 6: The adaptive diffusion sampling module generates a multi-subject image according to the optimized layout.

[0017] According to a specific implementation method of an embodiment of the present disclosure, the multimodal deep understanding module includes a text encoder, an image encoder and a cross-modal fusion layer, the text encoder includes a 12-layer Transformer with a hidden dimension of 1024, the image encoder is an improved Vision Transformer, and the cross-modal fusion layer is a 4-layer bidirectional attention mechanism.

[0018] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0019] Step 2.1, extracting text features corresponding to the text description and performing linear projection, and extracting image features corresponding to the text description using an image generation model and performing linear projection;

[0020] Step 2.2, use the linear projection of text features as the query matrix Q, and the linear projection of image features as the key matrix K and the value matrix V;

[0021] Step 2.3: Encode the text features using the text encoder, encode the image features using the image encoder, and combine the query matrix Q, key matrix K, and value matrix V to obtain the cross-modal attention output H through the cross-modal fusion layer. cross

[0022]

[0023] Among them, H text and H image are the hidden representations after encoding text features and image features, is a learnable projection matrix, T represents matrix transpose, and H cross is the cross-modal attention output, representing the fused text and image features, and A is the function representation of the bidirectional attention mechanism;

[0024]

[0025] Among them, A(Q,K,V) is the output of the bidirectional attention mechanism, which represents the weighted sum calculated by the query matrix Q, the key matrix K and the value matrix V. Softmax(*) represents the soft maximization function, which is used to convert the input into a probability distribution. k represents the dimension of the key vector;

[0026] Step 2.4, apply a feedforward neural network and layer normalization to output H according to the cross-modal attention. cross Generate fusion features

[0027] F fused =LayerNorm(H cross +FFN(H cross ))

[0028] Among them, FFN is a feed-forward neural network, and LayerNorm is a layer normalization operation;

[0029] Step 2.5, according to the fusion feature F fused Calculate the probability of entity and relationship extraction, obtain the entity list and relationship list and use them as semantic information

[0030]

[0031] Among them, P(e|F i ) represents a given feature F i When the probability of entity e, F i is the i-th feature vector, representing the fusion feature F fused one of the, represents the weight vector associated with entity e, exp(*) represents the exponential function, represents the dot product of the weight vector and the feature vector, Represents the calculation of all possible entities e P(r|e i ,e j ,F) indicates that in a given entity e i ,e j The probability that the relationship r is established under the condition of and feature F, wr represents the weight vector associated with relation r, [F; e i ;e j ] represents the feature F and the representation of the two entities e i ,e j concatenate into a longer vector, represents the dot product of the weight vector and the connected vector, Indicates the calculation of all possible relations r' The sum.

[0032] According to a specific implementation method of an embodiment of the present disclosure, the hierarchical layout planning agent includes an input embedding layer, a multi-head self-attention layer, a feedforward network layer and a hierarchical decoder. The multi-head self-attention layer has 8 heads, and the dimension of each head is 128. The feedforward network layer includes two layers of MLP with a hidden dimension of 2048. The hierarchical decoder has 3 layers.

[0033] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:

[0034] Step 3.1, the input embedding layer encodes the entities in the entity list and the relationship information in the relationship list into vectors to obtain an embedded vector sequence, where the expression of the embedded vector sequence is

[0035]

[0036] Among them, pos is the position, i is the dimension index, and d model is the model dimension;

[0037] Step 3.2, input the embedding vector sequence into the multi-head self-attention layer to obtain the attention feature, where the expression of the multi-head self-attention layer is

[0038]

[0039] MultiHead(H)=Concat(head1,…,headh)w 0

[0040] in, is the parameter matrix of the ith head, w 0 is the output projection matrix;

[0041] Step 3.3, input the attention features into the feedforward network layer, and perform nonlinear transformation through two layers of MLP to obtain the transformed features;

[0042] Step 3.4, input the transformed features into a 3-layer top-down hierarchical decoder to generate an initial layout, where the expression of the hierarchical decoder is

[0043]

[0044] in, is the output of the i-th sublayer of the l-th layer. The first formula realizes the decoding of the encoder layer, and the second formula realizes the output of the feedforward network layer. Through nonlinear transformation, features are further extracted and integrated to finally complete hierarchical decoding.

[0045] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:

[0046] Step 4.1, mapping entity features in the entity list to graph nodes as initialization node features;

[0047] Step 4.2, map the relationship features in the relationship list to the graph edges as the initialization edge features;

[0048] Step 4.3, update the initial node features and initial edge features through the 3-layer graph attention network, where the expression of each layer of the graph attention network is

[0049]

[0050] Among them, K is the number of attention heads, a k and W k is the learnable parameter of the kth head, h j represents the feature vector of node i and node j, m represents an index variable used for summation, N i represents the neighbor set of node i, σ represents the activation function, and LeakyReLU represents the activation function;

[0051] The expression for updating the initialization edge feature is

[0052]

[0053] in, is the feature of the edge (i, j) at layer l, is the feature of node i, is the learnable weight matrix;

[0054] Step 4.4, globally pool the node features updated by the last layer of graph attention network to generate a scene graph, where the expression of the scene graph is

[0055]

[0056] in, is the node feature of the last layer, W g and b g is a learnable parameter.

[0057] According to a specific implementation of the embodiment of the present disclosure, step 5 specifically includes:

[0058] Step 5.1, encode the initial layout into a state vector to obtain a state representation;

[0059] Step 5.2, based on the state representation, generate actions through the policy network, where the expression of the policy network is

[0060]

[0061] μ θ (s) = W 2 ReLU(W 1 s+b 1 )+b 2

[0062] σ θ (s) = softplus(W 3 ReLU(W 1 s+b 1 )+b 3 )

[0063] Among them, π θ (a|s) represents the probability distribution of selecting action a in state s, with mean μ θ (s), the variance is The normal distribution of θ (s) is the mean output of the policy network, calculated by a two-layer neural network, σ θ (s) is the standard deviation output of the policy network, which is also calculated by a two-layer neural network and uses the softplus function to ensure that the output is positive. θ represents the parameter set of the policy network, s represents the input state, a is the output action, and W 1 ,W 2 ,W 3 represents the weight matrix, b 1 ,b 2 ,b 3 They are three bias vectors, ReLU is the rectified linear unit activation function, and softplus is the activation function to ensure that the output is always positive;

[0064] In step 5.3, the action is applied to the initial layout to obtain the optimized layout and reward.

[0065] According to a specific implementation of the embodiment of the present disclosure, after step 5.3, the method further includes:

[0066] Based on the state representation, the state value is estimated through the value network, where the expression of the value network is

[0067]

[0068] Among them, φ represents the parameter set of the value function network, W h represents the weight matrix of the hidden layer, b h represents the bias vector of the hidden layer, represents the weight vector of the output layer, b v Represents the bias scalar of the output layer, ReLU is the rectified linear unit activation function;

[0069] The advantage function and PPO-Cl ip objective function are calculated according to the state value and the strategy network is optimized accordingly, wherein the expression of the advantage function is:

[0070]

[0071] Among them, A t represents the advantage function value at time step t, γ represents the discount factor, which is used to balance the importance of immediate rewards and future rewards, and r t represents the instantaneous reward obtained at time step t, T represents the total number of time steps of the current trajectory, and V φ (s T ) represents the value function estimate under state s, s t represents the state at time step t, s T Indicates the state at the end of the trajectory;

[0072] The expression of the PPO-Clip objective function is:

[0073]

[0074] in, represents the expected value of sampling under the current strategy, r t (θ represents the probability ratio of the new and old strategies, Represents the estimated value of the advantage function, clip() represents the clipping function, which limits the input to a specified range, and ∈ represents the clipping parameter, which controls the allowable policy variation range.

[0075] According to a specific implementation of the embodiment of the present disclosure, step 6 specifically includes:

[0076] Step 6.1, initialize the latent variable x T is random noise;

[0077] Step 6.2, input the latent variable x at the current time step t t Perform reverse diffusion, predict noise and update latent variables, and output the latent variable x for the next time step t-1 ;

[0078] Step 6.3, based on the latent variable x at the current time step t t , fusing conditional information through a conditional attention mechanism, wherein the conditional information includes an optimized layout and text description;

[0079] Step 6.4, according to the current time step t and layout complexity C(L t ) Dynamically adjust the noise parameter β t , after reverse diffusion, the final latent variable x is obtained 0 ;

[0080] Step 6.5, the final latent variable x 0 Input into the decoder to generate multi-subject images.

[0081] The multi-agent image generation scheme based on text description in the embodiment of the present disclosure includes: step 1, obtaining the text description input by the user, and the multi-agent coordinator manages the agent interaction and task allocation; step 2, the multimodal deep understanding module parses the text description and extracts semantic information; step 3, the hierarchical layout planning agent generates an initial layout according to the semantic information; step 4, the graph neural network relationship modeling module establishes and updates the scene graph according to the initial layout; step 5, the reinforcement learning layout optimization agent iteratively optimizes the layout through the policy gradient method; step 6, the adaptive diffusion sampling module generates a multi-agent image according to the optimized layout.

[0082] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the multimodal understanding module parses the text and extracts semantic information. The hierarchical layout planning agent generates the initial layout and captures the long-distance dependencies between subjects. The graph neural network relationship modeling module builds the scene graph and models complex relationships. The reinforcement learning layout optimization agent optimizes the layout through the policy gradient method, balancing semantic consistency, aesthetic score and spatial rationality. The adaptive diffusion sampling module adjusts the sampling process and introduces the conditional attention mechanism. The multi-agent coordinator manages the interaction of agents and the allocation of tasks. Finally, high-quality, multi-agent image generation is achieved, which improves the efficiency, accuracy and adaptability of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 A flowchart of a method for generating a multi-subject image based on text description provided in an embodiment of the present disclosure;

[0085] Figure 2 A schematic diagram of a process framework of a multi-subject image generation method based on text description provided in an embodiment of the present disclosure;

[0086] Figure 3 A specific implementation flow chart of a method for generating multi-subject images based on text description provided in an embodiment of the present disclosure;

[0087] Figure 4 A schematic diagram of an improved CLIP-BERT model structure provided in an embodiment of the present disclosure;

[0088] Figure 5 A schematic diagram of an improved Transformer structure provided in an embodiment of the present disclosure;

[0089] Figure 6 A schematic diagram of a graph neural network structure provided in an embodiment of the present disclosure;

[0090] Figure 7 A schematic diagram of a Unet model structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0091] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0092] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0093] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0094] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0095] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0096] The embodiment of the present disclosure provides a multi-subject image generation method based on text description, which can be applied to the process of generating images from text in Internet scenarios.

[0097] See also Figure 1 , is a flow chart of a method for generating a multi-subject image based on text description provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:

[0098] Step 1: Get the text description input by the user, and the multi-agent coordinator manages the agent interaction and task allocation;

[0099] like Figure 2 As shown, the method disclosed in the present invention can be applied to a multi-agent automatic layout generation system, which includes a multimodal deep understanding module (MDU), a hierarchical layout planning agent (HLP), a graph neural network relationship modeling module (GRM), a reinforcement learning layout optimization agent (RLO), an adaptive diffusion sampling module (ADS) and a multi-agent coordinator (MAC).

[0100] These components are managed and coordinated uniformly by the Multi-Agent Coordinator (MAC). The workflow of the system is as follows:

[0101] Flowchart as Figure 3 As shown in the figure, the user first inputs a text description, and the Multimodal Deep Understanding Module (MDU) parses the text and extracts semantic information. If the extraction is successful, the Hierarchical Layout Planning Agent (HLP) generates an initial layout and judges its rationality; if it is unreasonable, the layout parameters are adjusted and regenerated. Then, the Graph Neural Network Relationship Modeling Module (GRM) builds a scene graph. If the relationship modeling is successful, it continues, otherwise it adjusts the relationship parameters and re-models.

[0102] Subsequently, the reinforcement learning layout optimization agent (RLO) optimizes the layout through the policy gradient method. If the optimization is successful, it will continue, otherwise it will adjust the optimization strategy and optimize again. The adaptive diffusion sampling module (ADS) generates images and dynamically adjusts the sampling process. If the image quality is qualified, it will output the final image, otherwise it will adjust the sampling strategy and regenerate it. Throughout the process, the multi-agent coordinator (MAC) manages the interaction and task allocation of each module to ensure the efficient operation of the system.

[0103] In specific implementation, the text description input by the user can be obtained first, and the multi-agent coordinator manages the agent interaction and task allocation.

[0104] Step 2: The multimodal deep understanding module parses the text description and extracts semantic information;

[0105] Furthermore, the multimodal deep understanding module includes a text encoder, an image encoder and a cross-modal fusion layer, the text encoder includes a 12-layer Transformer with a hidden dimension of 1024, the image encoder is an improved Vision Transformer, and the cross-modal fusion layer is a 4-layer bidirectional attention mechanism.

[0106] Furthermore, the step 2 specifically includes:

[0107] Step 2.1, extracting text features corresponding to the text description and performing linear projection, and extracting image features corresponding to the text description using an image generation model and performing linear projection;

[0108] Step 2.2, use the linear projection of text features as the query matrix Q, and the linear projection of image features as the key matrix K and the value matrix V;

[0109] Step 2.3: Encode the text features using the text encoder, encode the image features using the image encoder, and combine the query matrix Q, key matrix K, and value matrix V to obtain the cross-modal attention output H through the cross-modal fusion layer. cross

[0110]

[0111] Among them, H text and H image are the hidden representations after encoding text features and image features, is a learnable projection matrix, T represents matrix transpose, and H cross is the cross-modal attention output, representing the fused text and image features, and A is the function representation of the bidirectional attention mechanism;

[0112]

[0113] Among them, A(Q,K,V) is the output of the bidirectional attention mechanism, which represents the weighted sum calculated by the query matrix Q, the key matrix K and the value matrix V. Softmax(*) represents the soft maximization function, which is used to convert the input into a probability distribution. k represents the dimension of the key vector;

[0114] Step 2.4, apply a feedforward neural network and layer normalization to output H according to the cross-modal attention. cross Generate fusion features

[0115] F fused =LayerNorm(H cross +FFN(H cross ))

[0116] Among them, FFN is a feed-forward neural network, and LayerNorm is a layer normalization operation;

[0117] Step 2.5, based on the fusion feature F fused Calculate the probability of entity and relationship extraction, obtain the entity list and relationship list and use them as semantic information

[0118]

[0119] Among them, P(e|F i ) represents a given feature F i When the probability of entity e, F i is the i-th feature vector, representing the fusion feature F fused one of the, represents the weight vector associated with entity e, exp(*) represents the exponential function, represents the dot product of the weight vector and the feature vector, Represents the calculation of all possible entities e P(r|e i ,e j ,F) indicates that in a given entity e i ,e j The probability that the relationship r is established under the condition of and feature F, w r represents the weight vector associated with relation r, [F; e i ;e j ] represents the feature F and the representation of the two entities e i ,e j concatenate into a longer vector, represents the dot product of the weight vector and the connected vector, Indicates the calculation of all possible relations r' The sum.

[0120] In specific implementation, the MDU module is based on the pre-trained CLIP-BERT model for deep understanding of text prompts. The processing steps are as follows:

[0121] 1) Text encoding:

[0122] Input: text description provided by the user;

[0123] Processing: Encoding is done using a 12-layer Transformer text encoder.

[0124] Output: text feature vector H text ;

[0125] 2) Image coding (if there is a reference image, generate the image coding; if not, convert the text features into image coding):

[0126] Input: reference image;

[0127] Processing: Encoding using improved Vision Transformer;

[0128] Output: Image feature vector H image ;

[0129] 3) Cross-modal fusion:

[0130] Input: text feature vector H text and the image feature vector H image ;

[0131] Processing: Use a 4-layer bidirectional attention mechanism for fusion;

[0132] Output: Cross-modal fusion feature H cross ;

[0133] 4) Fusion feature generation:

[0134] Input: Cross-modal fusion feature H cross ;

[0135] Processing: Apply feed-forward neural network and layer normalization;

[0136] Output: Final fusion feature F fused ;

[0137] 5) Entity and Relation Extraction:

[0138] Input: Final fusion feature F fused ;

[0139] Processing: Calculate probability distribution of entities and relations

[0140] Output: entity list E and relationship list R;

[0141] 6) The improved CLIP-BERT model structure is as follows Figure 4 shown.

[0142] 7) Cross-modal attention mechanism:

[0143]

[0144] Where A(Q,K,V) is the attention output, which represents the weighted sum calculated by query (Q), key (K) and value (V). Q is the query matrix, which represents the projection of text features. K is the key matrix, which represents the projection of image features. V is the value matrix, which represents the projection of image features. softmax(): soft maximization function, used to convert the input into a probability distribution. d k Indicates the dimension of the key vector.

[0145]

[0146] Among them, Htext and Himage are the hidden representations of text and image respectively, and W q ,W k ,W v H is a learnable projection matrix, and T represents the matrix transpose operation. cross is the cross-modal attention output, representing the fused text and image features. A is the function representation of the attention mechanism.

[0147] 8) Fusion feature generation:

[0148] F fused =LayerNorm(H cross +FFN(H cross ))

[0149] Among them, FFN is a feedforward neural network and LayerNorm is a layer normalization operation.

[0150] 9) Entity and relationship extraction probability calculation:

[0151]

[0152] a)P(e|F i ): indicates that given the feature F i The probability of entity e appearing under the condition .

[0153] b)w e : is the weight vector associated with entity e.

[0154] c)F i : is the input feature vector, usually extracted from a neural network.

[0155] d) Represents the dot product of the weight vector and the feature vector, and the result is a scalar.

[0156] e) exp(): Exponential function, used to convert the dot product result into a positive number.

[0157] f) For all possible entities e, calculate

[0158] g) The entire formula is actually a softmax function, which ensures that the sum of the probabilities of all entities is 1.

[0159]

[0160] a)P(r|e i ,e j ,F): indicates that in a given entity e i ,e j The probability that relationship r holds under the conditions of and feature F.

[0161] b)w r : is the weight vector associated with relation r.

[0162] c)[F;e i ;e j ]: represents the representation of feature F and two entities e i ,e j Concatenate into a longer vector.

[0163] d) The dot product of the weight vector and the concatenated vector.

[0164] e)exp(): Also used to convert the result into a positive number.

[0165] f) For all possible relations r', calculate The sum.

[0166] g) This formula is also a softmax function, ensuring that the sum of the probabilities of all possible relationships is 1.

[0167] Step 3, the hierarchical layout planning agent generates an initial layout based on the semantic information;

[0168] Furthermore, the hierarchical layout planning agent includes an input embedding layer, a multi-head self-attention layer, a feedforward network layer and a hierarchical decoder. The multi-head self-attention layer has 8 heads, each with a dimension of 128. The feedforward network layer includes two layers of MLP with a hidden dimension of 2048. The hierarchical decoder has 3 layers.

[0169] Furthermore, the step 3 specifically includes:

[0170] Step 3.1, the input embedding layer encodes the entities in the entity list and the relationship information in the relationship list into vectors to obtain an embedded vector sequence, where the expression of the embedded vector sequence is

[0171]

[0172] Among them, pos is the position, i is the dimension index, and d model is the model dimension;

[0173] Step 3.2, input the embedding vector sequence into the multi-head self-attention layer to obtain the attention feature, where the expression of the multi-head self-attention layer is

[0174]

[0175] MultiHead(H)=Concat(head1,…,headh)w 0

[0176] in, is the parameter matrix of the ith head, w 0 is the output projection matrix;

[0177] Step 3.3, input the attention features into the feedforward network layer, and perform nonlinear transformation through two layers of MLP to obtain the transformed features;

[0178] Step 3.4, input the transformed features into a 3-layer top-down hierarchical decoder to generate an initial layout, where the expression of the hierarchical decoder is

[0179]

[0180] in, is the output of the i-th sublayer of the l-th layer. The first formula realizes the decoding of the encoder layer, and the second formula realizes the output of the feedforward network layer. Through nonlinear transformation, features are further extracted and integrated to finally complete hierarchical decoding.

[0181] In the specific implementation, the HLP agent uses an improved Transformer architecture to generate the initial hierarchical layout. The processing steps are as follows:

[0182] 1) Input embedding:

[0183] Input: entity list e and relationship list r output by the MDU module;

[0184] Processing: Encode entity and relationship information into vectors;

[0185] Output: embedding vector sequence X;

[0186] 2) Multi-head self-attention processing:

[0187] Input: embedding vector sequence X;

[0188] Processing: Use 8-head self-attention mechanism for feature extraction;

[0189] Output: attention feature A;

[0190] 3) Feedforward network processing:

[0191] Input: Attention feature A;

[0192] Processing: Nonlinear transformation through two layers of MLP;

[0193] Output: transformed feature F;

[0194] 4) Hierarchical decoding:

[0195] Input: transformed feature F;

[0196] Processing: Generate layout using 3-layer top-down decoder;

[0197] Output: Initial hierarchical layout

[0198] 5) The improved Transformer structure is as follows Figure 5 shown.

[0199] 6) Multi-head attention calculation:

[0200]

[0201] MultiHead(H)=Concat(head1,…,headh)w 0

[0202] in, is the parameter matrix of the ith head, w 0 is the output projection matrix.

[0203] 7) Positional encoding

[0204]

[0205] Among them, pos is the position, i is the dimension index, and d model is the model dimension.

[0206] 8) Hierarchical decoding:

[0207]

[0208] in, is the output of the i-th sublayer of the l-th layer. The first formula realizes the decoding of the encoder layer, and the second formula realizes the output of the feedforward network layer. Through nonlinear transformation, features are further extracted and integrated to finally complete hierarchical decoding.

[0209] Step 4: The graph neural network relationship modeling module establishes and updates the scene graph according to the initial layout;

[0210] Based on the above embodiment, step 4 specifically includes:

[0211] Step 4.1, mapping entity features in the entity list to graph nodes as initialization node features;

[0212] Step 4.2, map the relationship features in the relationship list to the graph edges as the initialization edge features;

[0213] Step 4.3, update the initial node features and initial edge features through the 3-layer graph attention network, where the expression of each layer of the graph attention network is

[0214]

[0215] Among them, K is the number of attention heads, a k and W k is the learnable parameter of the kth head, h j represents the feature vector of node i and node j, m represents an index variable used for summation, N i represents the neighbor set of node i, σ represents the activation function, and LeakyReLU represents the activation function;

[0216] The expression for updating the initialization edge feature is

[0217]

[0218] in, is the feature of the edge (i, j) at layer l, is the feature of node i, is the learnable weight matrix;

[0219] Step 4.4, globally pool the node features updated by the last layer of graph attention network to generate a scene graph, where the expression of the scene graph is

[0220]

[0221] in, is the node feature of the last layer, W g and b g is a learnable parameter.

[0222] In specific implementation, the GRM module uses an improved graph attention network (GAT) to model complex relationships. The processing steps are as follows:

[0223] 1) Node feature initialization:

[0224] Input: entity features extracted by the MDU module;

[0225] Processing: Map entity features to graph nodes;

[0226] Output: initial node features;

[0227] 2) Edge feature initialization:

[0228] Input: relation features extracted by MDU module;

[0229] Processing: Map relation features to graph edges;

[0230] Output: initial edge features;

[0231] 3) Graph attention layer processing:

[0232] Input: node features and edge features Processing: Update features through a 3-layer graph attention network;

[0233] Output: updated node feature H^(l+1) and edge feature E^(l+1);

[0234] 4) Global pooling:

[0235] Input: node features H^(L) of the last layer;

[0236] Processing: Generate a global representation of the entire scene;

[0237] Output: scene graph representation G;

[0238] 5) The structure of graph neural network is as follows Figure 6 shown.

[0239] 6) Edge feature update:

[0240]

[0241] in, is the feature of the edge (i,j) at layer l, is the feature of node i, is a learnable weight matrix.

[0242] 7) Multi-head graph attention:

[0243]

[0244] Among them, K is the number of attention heads, a k and W k is the learnable parameter of the kth head, h j represents the feature vector of node i and node j, m represents that m is an index variable used for summation. In this formula, m represents all neighbor nodes of node i. It is used to traverse all neighbors of node i to calculate the normalization factor (i.e. the denominator part), N i Represents the set of neighbors of node i. In the graph structure, the neighbors of a node refer to all other nodes directly connected to the node. σ represents the activation function, and LeakyReLU represents the activation function, which is used to introduce nonlinearity. Compared with ordinary ReLU, LeakyReLU has a very small slope for negative input, avoiding the "dead ReLU" problem.

[0245] 8) Global graph representation:

[0246]

[0247] Among them, h G is the global representation of the graph, is the node feature of the last layer, W g and b g is a learnable parameter.

[0248] Step 5: The reinforcement learning layout optimization agent iteratively optimizes the layout through the policy gradient method;

[0249] Based on the above embodiment, step 5 specifically includes:

[0250] Step 5.1, encode the initial layout into a state vector to obtain a state representation;

[0251] Step 5.2, based on the state representation, generate actions through the policy network, where the expression of the policy network is

[0252]

[0253] μ θ (s) = W 2 ReLU(W 1 s+b 1 )+b 2

[0254] σ θ (s) = softplus(W 3 ReLU(W 1 s+b 1 )+b 3 )

[0255] Among them, π θ (a|s) represents the probability distribution of selecting action a in state s, with mean μ θ (s), the variance is The normal distribution of θ (s) is the mean output of the policy network, calculated by a two-layer neural network, σ θ (s) is the standard deviation output of the policy network, which is also calculated by a two-layer neural network and uses the softplus function to ensure that the output is positive. θ represents the parameter set of the policy network, s represents the input state, a is the output action, and W 1 ,W 2 ,W 3 represents the weight matrix, b 1 ,b 2 ,b 3 They are three bias vectors, ReLU is the rectified linear unit activation function, and softplus is the activation function to ensure that the output is always positive;

[0256] In step 5.3, the action is applied to the initial layout to obtain the optimized layout and reward.

[0257] Furthermore, after step 5.3, the method further comprises:

[0258] Based on the state representation, the state value is estimated through the value network, where the expression of the value network is

[0259]

[0260] Among them, φ represents the parameter set of the value function network, W h represents the weight matrix of the hidden layer, b h represents the bias vector of the hidden layer, represents the weight vector of the output layer, b v Represents the bias scalar of the output layer, ReLU is the rectified linear unit activation function;

[0261] The advantage function and PPO-Cl ip objective function are calculated according to the state value and the strategy network is optimized accordingly, wherein the expression of the advantage function is:

[0262]

[0263] Among them, A t represents the advantage function value at time step t, γ represents the discount factor, which is used to balance the importance of immediate rewards and future rewards, and r t represents the instantaneous reward obtained at time step t, T represents the total number of time steps of the current trajectory, and V φ (s T) represents the value function estimate under state s, s t represents the state at time step t, s T Indicates the state at the end of the trajectory;

[0264] The expression of the PPO-Clip objective function is:

[0265]

[0266] in, represents the expected value of sampling under the current strategy, r t (θ) represents the probability ratio of the new and old strategies, Represents the estimated value of the advantage function, clip( ) represents the clipping function, which limits the input to a specified range, and ∈ represents the clipping parameter, which controls the allowable policy variation range.

[0267] In specific implementation, the RLO agent uses the policy gradient method to optimize the layout. The processing steps are as follows:

[0268] 1) Status code:

[0269] Input: Current layout;

[0270] Processing: Encode the layout into a state vector;

[0271] Output: status representation;

[0272] 2) Action generation:

[0273] Input: Status representation

[0274] Processing: Generate actions through the policy network;

[0275] Output: Actions (continuous values ​​for adjusting the position, size, and rotation of the subject);

[0276] 3) Environmental interaction:

[0277] Input: Action;

[0278] Processing: Apply actions to the current layout;

[0279] Output: New layout and rewards;

[0280] 4) Value function estimation:

[0281] Input: state representation;

[0282] Processing: Estimate state value through value network;

[0283] Output: state value estimate V(s_t);

[0284] 5) Strategy Update:

[0285] Input: trajectory;

[0286] Processing: Use PPO-Clip algorithm to update the strategy;

[0287] Output: updated policy parameters θ;

[0288] 6) Strategy Network:

[0289]

[0290] μ θ (s) = W 2 ReLU(W 1 s+b 1 )+b 2

[0291] σ θ (s) = softplus(W 3 ReLU(W 1 s+b 1 )+b 3 )

[0292] 7) Value Function Network:

[0293]

[0294] 8) Advantage function estimation:

[0295]

[0296] 9) PPO-Clip objective function:

[0297]

[0298] in, ∈ is the cropping parameter.

[0299] The RLO agent uses a policy gradient approach for layout optimization. The state space consists of an encoded representation of the current layout, and the action space consists of continuous values ​​for adjusting the position, size, and rotation of the subject. The policy network consists of a 3-layer fully connected network using the Tanh activation function. The value network consists of a 2-layer fully connected network for estimating state values. The policy network and value function network are calculated using specific formulas, and the advantage function estimation and PPO-Clip objective function are used to optimize the policy and improve the layout quality.

[0300] Step 6: The adaptive diffusion sampling module generates a multi-subject image according to the optimized layout.

[0301] Based on the above embodiment, step 6 specifically includes:

[0302] Step 6.1, initialize the latent variable x T is random noise;

[0303] Step 6.2, input the latent variable x at the current time step t t Perform reverse diffusion, predict noise and update latent variables, and output the latent variable x for the next time step t-1 ;

[0304] Step 6.3, based on the latent variable x at the current time step t t , fusing conditional information through a conditional attention mechanism, wherein the conditional information includes an optimized layout and text description;

[0305] Step 6.4, according to the current time step t and layout complexity C(L t ) Dynamically adjust the noise parameter β t , after reverse diffusion, the final latent variable x is obtained 0 ;

[0306] Step 6.5, the final latent variable x 0 Input into the decoder to generate multi-subject images.

[0307] In specific implementation, the ADS module introduces dynamic noise scheduling and conditional attention mechanism based on the potential diffusion model. The processing steps are as follows:

[0308] 1) Initialization:

[0309] Input: optimized layout and text description;

[0310] Processing: Initialize latent variable x t is random noise;

[0311] Output: Initial latent variable x t ;

[0312] 2) Reverse diffusion process:

[0313] Input: Latent variable x at current time step t t ;

[0314] Processing: predict noise and update latent variables;

[0315] Output: Latent variable x at the next time step t-1 ;

[0316] 3) Conditional injection:

[0317] Input: Current latent variable x t and condition information c (including layout and text description);

[0318] Processing: Fusion of conditional information through conditional attention mechanism;

[0319] 4) Dynamic noise scheduling:

[0320] Input: current time step t and layout complexity C(L t );

[0321] Processing: Dynamically adjust the noise level according to the layout complexity;

[0322] Output: Adjusted noise parameter β t ;

[0323] 5) Image generation:

[0324] Input: Final latent variable X θ ;

[0325] Processing: Generate the final image through the decoder;

[0326] Generated image I;

[0327] 6) Noise prediction network:

[0328] ∈ θ (x t ,t)=U θ (DownSample(x t ),t)

[0329] Among them, U θ It is a neural network with U-Net structure. Noise prediction network ∈ θ Used to estimate the value added to the image x at time step t t It uses the U-Net structure to make predictions. ∈ θ : Noise prediction network, θ represents the network parameters, x t is the noise image at time step t, t is the current time step, and DownSample represents the downsampling operation, which is used to reduce the resolution of the input image.

[0330] Reverse process sampling:

[0331]

[0332] Among them, α t =1-β t , z~N(0,I).

[0333] This formula describes the inverse process of the diffusion model, which is used to gradually restore the original image from the noisy image. The inverse process sampling formula is used to calculate the image x at time step t-1 t-1 , based on the image x at the current time step t tand the predicted noise ∈ θ (x t ,t),x t-1 is the image at time step t-1, x t represents the image at time step t, α t is the noise scheduling parameter, controlling the amount of noise added at each time step, β t is the noise variance scheduling parameter, represents the cumulative product of α, and z is the random noise sampled from the standard normal distribution.

[0334] 7) Dynamic noise scheduling:

[0335] β t =β min +t(β max -β min )+λC(L t )

[0336] Among them, C(L t ) is the layout complexity function, and λ is the adaptive coefficient. In the formula, β t is the noise variance parameter at time step t, β min is the minimum noise variance, β max is the maximum noise variance, t represents the current time step, and λ is the adaptive coefficient, which controls the influence of layout complexity on noise scheduling. C(L t ) is the layout complexity function, according to the current layout L t Computational complexity.

[0337] 8) Conditional injection:

[0338] h t =Concat(U θ (x t ,t),f cond (c))

[0339] x t-1 =x t-1 +γ·Attention(h t ,f cond (c))

[0340] where f cond is the conditional encoding function, and c is the conditional information (such as layout and text features).

[0341] 9) Unet model structure is as follows Figure 7 shown.

[0342] Furthermore, MAC is responsible for managing the interaction and task allocation of each agent. Its processing steps are as follows: 1) Task decomposition:

[0343] Input: user request;

[0344] Processing: Break down complex tasks into subtasks;

[0345] Output: subtask list;

[0346] 2) Resource allocation:

[0347] Input: list of subtasks and available resources;

[0348] Processing: Allocate appropriate computing resources to each subtask;

[0349] Output: resource allocation plan;

[0350] 3) Agent Scheduling:

[0351] Input: subtask list and resource allocation plan;

[0352] Processing: Scheduling appropriate agents based on task type and resource availability;

[0353] Output: Agent scheduling plan;

[0354] 4) Message passing:

[0355] Input: Communication requirements between agents;

[0356] Processing: managing information exchange between agents;

[0357] Output: updated agent state;

[0358] 5) Conflict Resolution:

[0359] Input: Agent decision conflict;

[0360] Processing: Coordinate and resolve decision conflicts between agents;

[0361] Output: Coordinated decision plan;

[0362] 6) Task allocation utility calculation:

[0363] U(a,t)=w 1 C(a)+w 2 E(a,t)+w 3 P(a,t)

[0364] Among them, U(a,t): is the utility score of agent (a_i) performing task (t_j). C(a) is the ability score of agent a, E(a,t) is the experience value of agent a performing task t, and P(a,t) is the performance prediction value.

[0365] 7)Conflict detection and resolution:

[0366]

[0367] Among them, Conflict (a i ,a j ) is agent a i and a j The resource conflict count between them, R is the resource set, and r(a) is the resource set required by agent a. i ) The set of resources required by the agent (a), It means that when the condition in the brackets is true, the value is 1, otherwise it is 0. This formula means that the agent with a higher utility score is selected to perform the task among the two conflicting agents.

[0368] 8) Dynamic task generation:

[0369]

[0370] P(t new |G,R): The probability of generating a new task (t_{new}) given the global state (G) and resource state (R).

[0371] softmax(): The softmax function is used to convert the input into a probability distribution.

[0372] Weight matrix, a linear transformation for task generation.

[0373] [G; R]: The concatenation vector of the global state (G) and the resource state (R).

[0374] b t : Bias vector, used for linear transformation of task generation.

[0375] This formula represents the probability distribution of generating new tasks based on the current global state and resource state, which is normalized by the softmax function. This agent mainly generates relevant task instructions and agent scheduling instructions.

[0376] The multi-agent image generation method based on text description provided in this embodiment parses the text through a multimodal understanding module to extract semantic information. The hierarchical layout planning agent generates the initial layout to capture the long-distance dependencies between subjects. The graph neural network relationship modeling module builds a scene graph to model complex relationships. The reinforcement learning layout optimization agent optimizes the layout through the policy gradient method to balance semantic consistency, aesthetic score and spatial rationality. The adaptive diffusion sampling module adjusts the sampling process and introduces a conditional attention mechanism. The multi-agent coordinator manages agent interactions and task allocation. Ultimately, high-quality, multi-agent image generation is achieved, which improves the efficiency, accuracy and adaptability of image generation.

[0377] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0378] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A multi-subject image generation method based on text description, characterized in that: The method is applied to a multi-agent automatic layout generation system, which includes a multimodal deep understanding module, a hierarchical layout planning agent, a graph neural network relationship modeling module, a reinforcement learning layout optimization agent, an adaptive diffusion sampling module and a multi-agent coordinator. The method includes: Step 1: Get the text description input by the user, and the multi-agent coordinator manages the agent interaction and task allocation; Step 2: The multimodal deep understanding module parses the text description and extracts semantic information; Step 3, the hierarchical layout planning agent generates an initial layout based on the semantic information; Step 4: The graph neural network relationship modeling module establishes and updates the scene graph according to the initial layout; Step 5: The reinforcement learning layout optimization agent iteratively optimizes the layout through the policy gradient method; Step 6: The adaptive diffusion sampling module generates a multi-subject image according to the optimized layout.

2. The method according to claim 1, characterized in that The multimodal deep understanding module includes a text encoder, an image encoder and a cross-modal fusion layer. The text encoder includes a 12-layer Transformer with a hidden dimension of 1024. The image encoder is an improved Vision Transformer. The cross-modal fusion layer is a 4-layer bidirectional attention mechanism.

3. The method according to claim 2, characterized in that The step 2 specifically includes: Step 2.1, extracting text features corresponding to the text description and performing linear projection, and extracting image features corresponding to the text description using an image generation model and performing linear projection; Step 2.2, use the linear projection of text features as the query matrix Q, and the linear projection of image features as the key matrix K and the value matrix V; Step 2.3: Encode the text features using the text encoder, encode the image features using the image encoder, and combine the query matrix Q, key matrix K, and value matrix V to obtain the cross-modal attention output H through the cross-modal fusion layer. cross Among them, H text and H image are the hidden representations after encoding text features and image features, is a learnable projection matrix, T represents matrix transpose, and H cross is the cross-modal attention output, representing the fused text and image features, and A is the function representation of the bidirectional attention mechanism; Among them, A(Q, K, V) is the output of the bidirectional attention mechanism, which represents the weighted sum calculated by the query matrix Q, the key matrix K and the value matrix V. Softmax(*) represents the soft maximization function used to convert the input into a probability distribution. k represents the dimension of the key vector; Step 2.4, apply a feedforward neural network and layer normalization to output H according to the cross-modal attention. cross Generate fusion features F fused =LayerNorm(H cross +FFN(H Cross )) Among them, FFN is a feed-forward neural network, and LayerNorm is a layer normalization operation; Step 2.5, according to the fusion feature F fused Calculate the probability of entity and relationship extraction, obtain the entity list and relationship list and use them as semantic information Among them, P(e|F i ) represents a given feature F i When the probability of entity e, F i is the i-th feature vector, representing the fusion feature F fused one of the, represents the weight vector associated with entity e, exp(*) represents the exponential function, represents the dot product of the weight vector and the feature vector, Represents the calculation of all possible entities e P(r|e i , e j , F) means that for a given entity e i , e j The probability that the relationship r is established under the condition of and feature F, w r represents the weight vector associated with relation r, [F; e i ;e j ] represents the feature F and the representation of the two entities e i , e j concatenate into a longer vector, represents the dot product of the weight vector and the connected vector, Represents the calculation of all possible relations r′ The sum.

4. The method according to claim 3, characterized in that The hierarchical layout planning agent includes an input embedding layer, a multi-head self-attention layer, a feedforward network layer and a hierarchical decoder. The multi-head self-attention layer has 8 heads, each with a dimension of 128. The feedforward network layer includes two layers of MLP with a hidden dimension of 2048. The hierarchical decoder has 3 layers.

5. The method according to claim 4, characterized in that The step 3 specifically includes: Step 3.1, the input embedding layer encodes the entities in the entity list and the relationship information in the relationship list into vectors to obtain an embedded vector sequence, where the expression of the embedded vector sequence is Where pos is the position, i is the dimension index, and d model is the model dimension; Step 3.2, input the embedding vector sequence into the multi-head self-attention layer to obtain the attention feature, where the expression of the multi-head self-attention layer is MultiHead(H)=Concat(head1,...,headh)w 0 in, is the parameter matrix of the ith head, w 0 is the output projection matrix; Step 3.3, input the attention features into the feedforward network layer, and perform nonlinear transformation through two layers of MLP to obtain the transformed features; Step 3.4, input the transformed features into a 3-layer top-down hierarchical decoder to generate an initial layout, where the expression of the hierarchical decoder is in, is the output of the i-th sublayer of the l-th layer. The first formula realizes the decoding of the encoder layer, and the second formula realizes the output of the feedforward network layer. Through nonlinear transformation, features are further extracted and integrated to finally complete hierarchical decoding.

6. The method according to claim 5, characterized in that The step 4 specifically includes: Step 4.1, mapping entity features in the entity list to graph nodes as initialization node features; Step 4.2, map the relationship features in the relationship list to the graph edges as the initialization edge features; Step 4.3, update the initial node features and initial edge features through the 3-layer graph attention network, where the expression of each layer of the graph attention network is Among them, K is the number of attention heads, a k and W k is the learnable parameter of the kth head, h j represents the feature vector of node i and node j, m represents an index variable used for summation, N i represents the neighbor set of node i, σ represents the activation function, and LeakyReLU represents the activation function; The expression for updating the initialization edge feature is in, is the feature of the edge (i, j) at layer l, is the feature of node i, is a learnable weight matrix; Step 4.4, globally pool the node features updated by the last layer of graph attention network to generate a scene graph, where the expression of the scene graph is in, is the node feature of the last layer, W g and b g is a learnable parameter.

7. The method according to claim 6, characterized in that The step 5 specifically includes: Step 5.1, encode the initial layout into a state vector to obtain a state representation; Step 5.2, based on the state representation, generate actions through the policy network, where the expression of the policy network is μ θ (s)=W2ReLU(W1s+b1)+b2 s θ (s)=softplus(W3ReLU(W1s+b1)+b3) Among them, π θ (a|s) represents the probability distribution of selecting action a in state s, with mean μ θ (s), the variance is The normal distribution of μθ(s) is the mean output of the policy network, which is calculated by a two-layer neural network. θ (s) is the standard deviation output of the policy network, which is also calculated through a two-layer neural network, and the softplus function is used to ensure that the output is positive. θ represents the parameter set of the policy network, s represents the input state, a is the output action, W1, W2, W3 represent the weight matrix, b1, b2, b3 are three bias vectors, ReLU is the rectified linear unit activation function, and softplus is the activation function to ensure that the output is always positive: In step 5.3, the action is applied to the initial layout to obtain the optimized layout and reward.

8. The method according to claim 7, characterized in that After step 5.3, the method further comprises: Based on the state representation, the state value is estimated through the value network, where the expression of the value network is Among them, φ represents the parameter set of the value function network, W h represents the weight matrix of the hidden layer, b h represents the bias vector of the hidden layer, represents the weight vector of the output layer, b v Represents the bias scalar of the output layer, ReLU is the rectified linear unit activation function; The advantage function and the PPO-Clip objective function are calculated according to the state value and the strategy network is optimized accordingly, wherein the expression of the advantage function is: Among them, A t represents the advantage function value at time step t, γ represents the discount factor, which is used to balance the importance of immediate rewards and future rewards, and r t represents the instantaneous reward obtained at time step t, T represents the total number of time steps of the current trajectory, and V φ (s T ) represents the value function estimate under state s, s t represents the state at time step t, s T Indicates the state at the end of the trajectory; The expression of the PPO-Clip objective function is: in, represents the expected value of sampling under the current strategy, r t (θ) represents the probability ratio of the new and old strategies, Represents the estimated value of the advantage function, clip() represents the clipping function, which limits the input to a specified range, and ∈ represents the clipping parameter, which controls the allowable policy variation range.

9. The method according to claim 8, characterized in that The step 6 specifically includes: Step 6.1, initialize the latent variable x T is random noise; Step 6.2, input the latent variable x at the current time step t t Perform reverse diffusion, predict noise and update latent variables, and output the latent variable x for the next time step t-1 ; Step 6.3, based on the latent variable x at the current time step t t , fusing conditional information through a conditional attention mechanism, wherein the conditional information includes an optimized layout and text description; Step 6.4, according to the current time step t and layout complexity C(L t ) Dynamically adjust the noise parameter β t , after reverse diffusion, the final latent variable x0 is obtained; In step 6.5, the final latent variable x0 is input into the decoder to generate a multi-subject image.

Images