A multimodal large-scale AI creative generation system and method

Abstract

This invention relates to the field of artificial intelligence technology and discloses a multimodal large-scale AI creative generation system and method. The system includes a semantic parsing module, a feature encoding initialization module, an attention intervention module, and an image decoding module. The method includes: extracting independent creative entities and their bounding box coordinates from the input prompt sequence, constructing an entity semantic dependency matrix; generating a key matrix and initializing latent variables in the latent space; iterating incrementally over time steps, calculating cross-attention and extracting specific entity attention subgraphs, constructing a dynamic two-dimensional Gaussian mask based on the dependency matrix, calculating the gradient of the partial derivative of energy loss to update the current latent variables and denoise; obtaining the final deterministic latent variables and decoding the output visual content. This invention solves the problem of multi-entity layout confusion in multimodal large-scale AI creative generation by using semantic constraints and gradient intervention of the dynamic mask during the denoising cycle.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an AI creative generation system and method for multimodal large-scale models. Background Technology

[0002] Multimodal large models achieve semantic alignment between text and visual data by constructing a joint feature space. These models utilize network architecture to extract low-level features across modalities, establish a mapping relationship between natural language and pixel matrices, enabling computers to receive text instructions and convert them into corresponding visual content.

[0003] AI-powered creative generation technology, based on this foundation, primarily synthesizes images directly from text prompts. Current common applications often employ a diffusion network architecture. This process involves converting text into conditional features, then injecting these features into the latent space using a global cross-attention mechanism across consecutive denoising time steps. The latent variables undergo multiple iterations to eliminate noise components, and finally, the decoder outputs the image.

[0004] Existing generation mechanisms suffer from deficiencies in spatial structure control when handling complex cues involving multiple objects. Current models primarily rely on global textual features for guidance during iterations, lacking specific constraints on the spatial geometric boundaries of individual entities. When instructions involve physical support or spatial occlusion between different objects, conventional attention mechanisms struggle to confine features to specific regions, easily leading to mutual interference in feature responses. This interference prevents the denoising process from maintaining objective physical dependencies, resulting in output images that often exhibit entity position confusion and spatial hierarchy misalignment, making it difficult to accurately recreate the layout intent described in the text.

[0005] Therefore, this invention proposes an AI creative generation system and method for multimodal large models to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an AI creative generation system and method for multimodal large models, which solves the problems of entity position confusion and structural disorder caused by the lack of physical space constraints when generating complex multi-entity images using existing models.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides an AI creative generation system for multimodal large models, comprising:

[0009] The semantic parsing module extracts the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence, and constructs the entity semantic dependency matrix using the syntactic dependency relation analysis algorithm;

[0010] The feature encoding initialization module maps the set of independent creative entities to generate a key matrix and initializes a random noise tensor in the latent space as the initial latent variable corresponding to the maximum denoising time step.

[0011] The attention intervention module performs iterative loops in descending order of denoising time steps. At each denoising time step, a query matrix is ​​generated using the current latent variable. A global cross-attention matrix is ​​calculated using the query matrix and the key matrix. An attention subgraph corresponding to a specific entity in the set of independent creative entities is extracted. A dynamic two-dimensional Gaussian mask is constructed by combining the denoising time step and the entity semantic dependency matrix. An energy loss function is constructed using the dynamic two-dimensional Gaussian mask and the attention subgraph. The partial derivative gradient is calculated to update the current latent variable to obtain the updated latent variable. The denoising network is used to eliminate noise components from the updated latent variable and output the next latent variable.

[0012] The image decoding module obtains the final deterministic latent variable when the denoising time step decreases to the latent variable corresponding to the output time step zero, and inputs the final deterministic latent variable into the variational autoencoder to output the visual creative content.

[0013] Preferably, the step of the semantic parsing module extracting the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence includes:

[0014] The natural language input prompt sequence is input into a large language model to identify visual object nouns, and the identified visual object nouns are aggregated into the set of independent creative entities;

[0015] By combining the natural language input prompt sequence with the large language model, the two-dimensional geometric region of each entity on the target generation canvas is predicted, and the boundary endpoint coordinates of the two-dimensional geometric region are extracted to form the spatial bounding box coordinate set.

[0016] Preferably, the step of constructing the entity semantic dependency matrix using the semantic parsing module with the syntactic dependency relation analysis algorithm includes:

[0017] The natural language input prompt sequence is input into the syntactic parser to generate a syntactic dependency tree containing lexical nodes and dependency relation edges;

[0018] Initialize a two-dimensional tensor as the semantic dependency matrix of the entity;

[0019] Specifically, the shortest dependency path between any two entities in the set of independent creative entities is extracted from the syntactic dependency tree. The physical dependency type is determined based on the verb and preposition tags on the shortest dependency path. The matrix elements of the entity semantic dependency matrix are numerically assigned according to the determined physical dependency type.

[0020] Preferably, the step of the feature encoding initialization module mapping the set of independent creative entities to generate a key matrix includes:

[0021] Extract the entity strings from the set of independent creative entities, and use a preset text delimiter to concatenate each entity string in sequence to form a one-dimensional text prompt sequence;

[0022] The text prompt sequence is input into the text encoder, which outputs a text feature vector containing global contextual semantic features. A linear mapping is then performed on the text feature vector to generate the key matrix.

[0023] Preferably, the step of the attention intervention module generating a query matrix using the current latent variables and calculating the global cross-attention matrix using the query matrix and the key matrix includes:

[0024] The current latent variable is input into the denoising network and mapped to a latent feature sequence representation. A linear transformation is performed on the latent feature sequence representation and projected onto the attention feature space to generate the query matrix.

[0025] Calculate the product of the query matrix and the transpose of the key matrix, divide the product by the feature vector scaling dimension for normalization, and perform a normalization exponential function operation along the feature sequence dimension on the normalized result to obtain the global cross-attention matrix.

[0026] Preferably, the step of the attention intervention module extracting the attention subgraph corresponding to a specific entity in the set of independent creative entities includes:

[0027] Based on the lexical index mapping relationship between the specific entity and the natural language input prompt sequence, slice retrieval is performed along the sequence dimension on the global cross-attention matrix to extract channel response data of the lexical corresponding to the specific entity;

[0028] The channel response data is fused to obtain the attention subgraph corresponding to the specific entity.

[0029] Preferably, the step of the attention intervention module constructing a dynamic two-dimensional Gaussian mask combining the denoising time step and the entity semantic dependency matrix includes:

[0030] Map the two-dimensional center coordinates of the specific entity in the spatial bounding box coordinate set to the two-dimensional coordinate system where the attention subgraph is located to obtain the mean vector corresponding to the specific entity, and construct the initial covariance matrix with the mean vector corresponding to the specific entity as the anchor point.

[0031] Query the entity semantic dependency matrix, and when it is determined that the specific entity has a physical dependency relationship with the target entity in the set of independent creative entities, such as a physical support relationship, a spatial occlusion relationship, or a light source and shadow mapping relationship, extract the mean vector corresponding to the target entity.

[0032] Based on the mean vector corresponding to the specific entity and the mean vector corresponding to the target entity, a unit direction vector is determined, and an orthogonal vector of the unit direction vector is constructed. Combined with a preset time step decay coefficient, the initial covariance matrix is ​​controlled to stop the variance decay calculation on the unit direction vector. The variance shrinkage is performed by multiplying the orthogonal vector by the time step decay coefficient, and a dynamic covariance matrix is ​​output.

[0033] When it is not determined whether the specific entity has a physical support relationship, spatial occlusion relationship or light source and shadow mapping relationship with any other entity in the set of independent creative entities, synchronous variance decay is performed on the two spatial directions based on the initial covariance matrix and the time step decay coefficient, and a dynamic covariance matrix is ​​output.

[0034] The dynamic two-dimensional Gaussian mask is obtained by using the multidimensional Gaussian probability density function in combination with the mean vector corresponding to the specific entity and the inverse matrix of the dynamic covariance matrix.

[0035] Preferably, the step of the attention intervention module constructing an energy loss function using the dynamic two-dimensional Gaussian mask and the attention subgraph, and calculating the partial derivative gradient to update the current latent variable includes:

[0036] Initialize the all-one matrix, and calculate the difference matrix between the all-one matrix and the dynamic two-dimensional Gaussian mask;

[0037] Calculate the Hadamard product of the attention subgraph and the difference matrix, and sum the squared norms of the Hadamard products of the specific entity as the energy loss function.

[0038] The partial derivative gradient of the energy loss function with respect to the current latent variable is calculated based on the chain rule;

[0039] The product of the partial derivative gradient and the step size hyperparameter is used as the gradient update amount, and the current latent variable is updated by subtracting the gradient update amount from the current latent variable.

[0040] Preferably, the step of the image decoding module inputting the final deterministic latent variable into the variational autoencoder and outputting the visual creative content includes:

[0041] The final deterministic latent variables are fed into the decoder network of the variational autoencoder for upsampling and decoding to output pixel-level image tensors;

[0042] The visual creative content is obtained by performing a linear inverse normalization operation on the pixel-level image tensor.

[0043] A second aspect of this invention provides a method for generating AI creative ideas from a multimodal large model, comprising the following steps:

[0044] Extract the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence, and construct the entity semantic dependency matrix using the syntactic dependency parsing algorithm;

[0045] A key matrix is ​​generated by mapping the set of independent creative entities, and a random noise tensor is initialized in the latent space as the initial latent variable corresponding to the maximum denoising time step;

[0046] The loop iterates in descending order of denoising time steps. At each denoising time step, a query matrix is ​​generated using the current latent variable. A global cross-attention matrix is ​​calculated using the query matrix and the key matrix. An attention subgraph corresponding to a specific entity in the set of independent creative entities is extracted. A dynamic two-dimensional Gaussian mask is constructed by combining the denoising time step and the entity semantic dependency matrix. An energy loss function is constructed using the dynamic two-dimensional Gaussian mask and the attention subgraph. The partial derivative gradient is calculated to update the current latent variable to obtain the updated latent variable. The denoising network is used to eliminate noise components from the updated latent variable and output the next latent variable.

[0047] When the denoising time step decreases to the latent variable corresponding to the output time step zero, the final deterministic latent variable is obtained, and the final deterministic latent variable is input into the variational autoencoder to output the visual creative content.

[0048] This invention provides an AI creative generation system and method for multimodal large-scale models. It has the following beneficial effects:

[0049] 1. This invention solves the problem of object position confusion when parsing instructions in a multimodal large model by extracting entity sets and spatial bounding box coordinates from natural language prompt sequences and constructing an entity semantic dependency matrix through syntactic analysis. Before executing AI creative generation at the visual level, this process transforms the physical support relationships between elements into quantified structural constraints, ensuring accurate spatial layout correspondence between the final visual and textual intent.

[0050] 2. This invention improves upon the unnatural rendering issues encountered by existing multimodal large models when handling multi-element interactions by extracting attention subgraphs of specific entities and constructing dynamic two-dimensional Gaussian masks using dependency matrices. During the AI ​​creative generation stage of the image, differential variance shrinkage of the mask based on the physical directions between entities guides the network to rationally allocate features to the target interaction area, ensuring that the occlusion and lighting mapping of the output image conform to objective reality.

[0051] 3. This invention utilizes a dynamic two-dimensional Gaussian mask and attention subgraph to construct an energy loss function, and calculates the partial derivative gradient to update the current latent variables, thereby controlling the image structure shift that easily occurs in the later stages of iterative denoising for large multimodal models. This numerical intervention method within the latent space can dynamically adjust the feature distribution within the AI ​​creative generation cycle, continuously correcting the semantic accuracy of the visual output without changing the underlying pre-trained weights. Attached Figure Description

[0052] Figure 1 This is a diagram of the AI ​​creative generation system architecture for multimodal large models according to the present invention;

[0053] Figure 2 This is a flowchart of the AI ​​creative generation method for multimodal large models according to the present invention;

[0054] Figure 3 This is a schematic diagram of the semantic parsing process of the present invention;

[0055] Figure 4 This is a schematic diagram of the feature encoding initialization process of the present invention;

[0056] Figure 5 This is a schematic diagram of the attention intervention process of the present invention;

[0057] Figure 6 This is a schematic diagram of the image decoding process of the present invention.

[0058] Among them, 100 is the semantic parsing module; 200 is the feature encoding initialization module; 300 is the attention intervention module; and 400 is the image decoding module. Detailed Implementation

[0059] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Please see Figure 1This invention provides a multimodal large-scale AI creative generation system, deployed in an environment equipped with a central processing unit (CPU), a graphics processing unit (GPU), and video memory. The CPU performs system task scheduling, receives natural language input prompts, controls data flow between modules, and manages inference parameters. The GPU is pre-installed with a deep learning computing framework supporting dynamic computation graph construction and automatic differentiation mechanisms, meeting the computational power requirements of the diffusion model for tensor gradient backpropagation of latent variables during the inference phase. The system includes: a semantic parsing module 100, a feature encoding initialization module 200, an attention intervention module 300, and an image decoding module 400.

[0061] The semantic parsing module 100 receives a natural language input prompt sequence. The semantic parsing module 100 calls a large language model to perform semantic processing on the natural language input prompt sequence to extract the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities. It then uses a syntactic dependency analysis algorithm to construct an entity semantic dependency matrix.

[0062] The feature encoding initialization module 200 is connected to the semantic parsing module 100. The feature encoding initialization module 200 uses a text encoder to perform feature mapping on the set of independent creative entities to generate a key matrix, and initializes a random noise tensor that conforms to a standard normal distribution in the latent space as the initial latent variable corresponding to the maximum denoising time step.

[0063] The attention intervention module 300 is connected to the feature encoding initialization module 200. It performs iterative iterations within the diffusion model, decreasing the denoising time step from the maximum to one during each iteration. At each denoising time step, the attention intervention module 300 generates a query matrix using the current latent variables and calculates the global cross-attention matrix using the query matrix and the key matrix. The attention intervention module 300 extracts attention subgraphs corresponding to specific entities in the independent creative entity set from slices of the global cross-attention matrix, constructing a dynamic two-dimensional Gaussian mask for the specific entity that combines the denoising time step and the entity semantic dependency matrix. The attention intervention module 300 constructs an energy loss function using the dynamic two-dimensional Gaussian mask and the attention subgraph, calculates the partial derivative gradient of the energy loss function with respect to the current latent variables, and updates the current latent variables using the partial derivative gradient to obtain the updated latent variables. The attention intervention module 300 uses the denoising network to remove noise components from the updated latent variables and outputs the next latent variable for the next denoising time step.

[0064] The image decoding module 400 is connected to the attention intervention module 300. When the denoising time step decreases to the latent variable corresponding to the output time step zero, the image decoding module 400 obtains the final deterministic latent variable. The final deterministic latent variable is input into the variational autoencoder to map the latent space features to the high-dimensional pixel space to complete image generation and output visual creative content.

[0065] See attached document Figure 2 This invention provides a method for generating AI creative ideas from multimodal large models, comprising the following steps:

[0066] S1 receives a sequence of natural language input prompts, calls a large language model to perform semantic processing on the natural language input prompt sequence to extract a set of independent creative entities and a set of spatial bounding box coordinates, and constructs an entity semantic dependency matrix.

[0067] S2, use a text encoder to perform feature mapping on the set of independent creative entities to generate a key matrix, and initialize a random noise tensor in the latent space as the initial latent variable corresponding to the maximum denoising time step;

[0068] S3, in the diffusion model, the loop iteration is executed in descending order of denoising time steps. At each denoising time step, a query matrix is ​​generated using the current latent variable. The global cross-attention matrix is ​​calculated using the query matrix and the key matrix. The attention subgraph corresponding to the specific entity is extracted. A dynamic two-dimensional Gaussian mask combining the denoising time step and the entity semantic dependency matrix is ​​constructed. An energy loss function is constructed using the mask and the subgraph. The partial derivative gradient is calculated to update the current latent variable. The next latent variable is output.

[0069] S4: When the denoising time step decreases to the latent variable corresponding to the zero output time step, the final deterministic latent variable is obtained, input to the variational autoencoder and mapped to the high-dimensional pixel space to complete image generation, and output visual creative content.

[0070] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the implementation of each functional module involved above and its internal processing flow.

[0071] See attached document Figure 3 In this embodiment, the specific implementation of the semantic parsing module 100 includes the following sub-steps:

[0072] S101, the semantic parsing module 100 extracts the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities.

[0073] In this embodiment, the large language model employs a deep neural network model based on the Transformer architecture, fine-tuned for a specific task. This large language model sequentially includes an input embedding layer, several stacked multi-head self-attention modules, and a fully connected decoding output layer. After the natural language input prompt sequence is input, the large language model first segments it using its built-in word segmenter and maps it to a dimension of [dimensionality missing]. The discrete word meta-index tensor, where The discrete term index tensor represents the number of terms in the sequence and takes positive integer values. The discrete term index tensor then enters the input embedding layer to be transformed into a high-dimensional feature vector. This vector undergoes feature interaction and fusion with the global context through a multi-head self-attention module, and finally, autoregressive prediction is performed through a fully connected decoding output layer.

[0074] To improve the stability of the large language model in performing extraction tasks, supervised fine-tuning training was performed during the model construction phase. Training samples were constructed as follows: images containing multiple independent objects were selected from an open-source image-text dataset, and corresponding text descriptions were manually labeled as network input data. Simultaneously, manually selected entity categories and their corresponding bounding box coordinates were serialized and used as label data. During training, the large language model used an autoregressive cross-entropy loss function to calculate the error between the predicted output and the label data, and backpropagation and the Adam optimizer were used to update the network weight parameters of the large language model. After supervised fine-tuning training, the large language model was able to output structured spatial constraint rules from natural language input prompt sequences.

[0075] The large language model processes the natural language input prompt sequence based on the aforementioned network layers, identifying and extracting visual object nouns that need to be independently controlled in the final image. The semantic parsing module 100 aggregates the identified visual object nouns into a set of independent creative entities. The set of independent creative entities is a collection containing multiple specific object elements that need to have clear physical boundaries within the image. Let the set of independent creative entities be:

[0076] ;

[0077] in, This represents the total number of entities extracted. The value of is a positive integer greater than or equal to 2; The first in the collection of independent creative entities Individual entities.

[0078] The large language model combines directional descriptions and proportion prepositions from the natural language input prompt sequence. Through a fully connected decoding output layer, it predicts the two-dimensional geometric region of each entity on the target generation canvas in JSON-compliant data format. The semantic parsing module 100 extracts the boundary endpoint coordinates of the two-dimensional geometric region, forming a spatial bounding box coordinate set. This spatial bounding box coordinate set defines the controlled positional range of each entity in the subsequent tensor feature generation stage. Let the spatial bounding box coordinate set be:

[0079] ;

[0080] in, For entities The corresponding bounding box data.

[0081] As a preferred parameter configuration method, bounding box data The specific data structure is represented as follows:

[0082] ;

[0083] in, and These are the horizontal and vertical pixel coordinates of the top-left corner of the two-dimensional geometric region. and These are the horizontal and vertical pixel coordinates of the lower right corner of the two-dimensional geometric region. The range of values ​​for the horizontal and vertical pixel coordinates is determined by the resolution of the target output image.

[0084] S102, the semantic parsing module 100 uses a syntactic dependency relation analysis algorithm to construct an entity semantic dependency matrix.

[0085] The semantic parsing module 100 inputs the natural language input prompt sequence into the syntactic analyzer to generate a syntactic dependency tree containing lexical nodes and dependency relation edges. In this embodiment, the syntactic analyzer employs a dual affine dependency parsing neural network model, which sequentially includes a word vector encoding layer, a bidirectional long short-term memory network layer, and a dual affine scoring network layer. After the natural language input prompt sequence is converted into a word embedding representation by the word vector encoding layer, the data stream is input to the bidirectional long short-term memory network layer to extract the forward and backward temporal features of each lexical node, generating a contextual hidden state tensor.

[0086] The contextual hidden state tensor is fed into the dual affine scoring network layer, where it is mapped to center word feature representation and dependency word feature representation through a multilayer perceptron, respectively. Matrix multiplication is then performed using dual affine functions to output the connection probability distribution matrix and dependency labels between word nodes. The semantic parsing module 100 then applies the maximum spanning tree algorithm to decode the connection probability distribution matrix, obtaining the globally optimal dependency path combination. This combines the discrete word nodes and dependency labels to generate a structured syntactic dependency tree.

[0087] When constructing and training the dual affine dependency parsing neural network model, a large-scale open-source syntactic dependency tree library is used as training samples and labels, and cross-entropy is used as the network loss function of the dual affine dependency parsing neural network model to perform supervised learning.

[0088] The semantic parsing module 100 defines the matrix elements of the entity semantic dependency matrix as corresponding to the physical dependency type between any two entities in the independent creative entity set. The physical dependency type is limited to one of the following: no dependency, physical support, spatial occlusion, and light source and shadow mapping. The semantic parsing module 100 extracts entities from the syntactic dependency tree. With entity The shortest dependency path between them is used to perform physical dependency type mapping based on the verb and preposition tags on the shortest dependency path.

[0089] In the specific entity semantic dependency matrix construction operation, the semantic parsing module 100 initializes the dimension in memory as follows: A two-dimensional tensor, used as an entity semantic dependency matrix Among them, the dimension value Consistent with the total number of entities in the set of independent creative entities, the entity semantic dependency matrix is ​​used to record the physical dependency types between entities.

[0090] The semantic parsing module iterates through the set of 100 independent creative entities and performs semantic dependency analysis on the entity semantic dependency matrix. matrix elements Perform numerical assignment. For example, if the entity With entity The verb and preposition tags on the shortest dependency path between them do not represent physical support relationships, spatial occlusion relationships, or light source and shadow mapping relationships, so they are determined to be undependent and set. If the syntactic relation indicates an entity Located in the entity The surface is supported by it, which is determined to be a physical support relationship and is set. If the syntactic relation indicates an entity Located in the entity If there is a visual obstruction in front, it is determined to be a spatial occlusion relationship and set... If the syntactic relation indicates an entity Lights up and illuminates the entity , or entity The shadow cast onto the entity The surface is determined to have a light source and shadow mapping relationship and set... Among them, matrix elements The possible values ​​of are strictly limited to a set. The entity semantic dependency matrix is ​​a structured data carrier that represents the underlying physical interaction patterns between entities, and is used to control the feature fusion boundary when generating each entity during the inference phase.

[0091] See attached document Figure 4 In this embodiment, the feature encoding initialization module 200 is implemented in the following sub-steps:

[0092] S201, Feature encoding initialization module 200 uses a text encoder to perform feature mapping on the set of independent creative entities to generate a key matrix.

[0093] During tensor construction, the feature encoding initialization module 200 converts the previously extracted set of independent creative entities into a text prompt sequence. As a specific implementation, the feature encoding initialization module 200 extracts entity strings from the set of independent creative entities in a specific order and concatenates these entity strings sequentially using a preset text delimiter to form a one-dimensional text prompt sequence.

[0094] To map text semantic features to subsequent image visual features in a shared data space, the feature encoding initialization module 200 uses a text encoder based on a contrastive language image pre-training architecture. During model building and pre-training, the text encoder utilizes a large-scale open-source paired image-text dataset as training samples. During training, paired image data and text descriptions are input into the corresponding visual image encoder and text encoder, respectively, to extract the corresponding image feature vectors and text feature vectors, and to calculate the cosine similarity matrix of the feature vector batches containing positive and negative sample pairs. The training process uses the correct pairing relationships of image-text matching as pseudo-label data, employs a symmetric cross-entropy loss function as the contrastive loss function, and updates the network weights through backpropagation and gradient descent optimizers. This increases the feature similarity of matching image-text pairs and decreases the feature similarity of mismatched image-text pairs, thereby forming a multimodal feature alignment mapping capability. In terms of network hierarchy and structure, the text encoder contains, in sequence, a lexicalization processing layer, a word embedding lookup table, an absolute position encoding layer, and a stacked multi-head self-attention feature extraction module. The layers of the stacked multi-head self-attention feature extraction module are connected and configured through residual connections and layer normalization operations.

[0095] After the text prompt sequence is input into the text encoder, it passes through the lexicalization layer, which performs truncation or zero-padding operations according to the preset maximum truncation length, such as 77 lexical units, to generate a fixed-length text. The discrete word index is entered into the word embedding lookup table and transformed into continuous word vectors through an embedding query operation. These continuous word vectors are then superimposed with the positional features output from the absolute position encoding layer to form a data dimension of [missing information]. The input embedding tensor, where, Characterizing the batch size dimension, A fixed numerical value representing the tensor in the dimension of sequence length. Characterizes the feature channel dimension. The input embedding tensor passes through the multi-head self-attention feature extraction module along the data flow.

[0096] The multi-head self-attention feature extraction module calculates the relevance weights between each word within a sequence using a masked self-attention mechanism, and outputs a text feature vector containing global contextual semantic features. Text feature vectors This is used to characterize the multidimensional semantic probability distribution of each independent creative entity object and its attribute concepts in the latent space of a natural language input prompt sequence. The feature encoding initialization module initializes 200 pairs of text feature vectors. Perform a linear mapping to generate the required key matrix.

[0097] The feature mapping process is accomplished through matrix multiplication, and the specific formula is as follows:

[0098] ;

[0099] in, This represents the output key matrix; This represents the text feature vector output from the text encoder, with a data dimension of . ; This is the preset maximum sequence length and is usually set to a positive integer. Input feature dimension; This is the pre-trained converged key weight matrix in the text encoder, with a data dimension of... , This represents the dimension of the mapped attention features.

[0100] The feature encoding initialization module 200 generates the key matrix therefrom. Key matrix It is used to represent the semantic context information of text in the underlying network cross-attention mechanism, and to provide conditional feature basis for query retrieval and matching in the subsequent image generation iteration process.

[0101] S202, the feature encoding initialization module 200 initializes a random noise tensor conforming to a standard normal distribution in the latent space as the initial latent variable corresponding to the maximum denoising time step.

[0102] The underlying diffusion model employs an implicit diffusion probability generation algorithm. Under this mechanism, the image denoising trajectory is not directly unfolded in the high-dimensional pixel space, but rather in the low-dimensional feature space after dimensionality reduction and compression via an autoencoder network. This low-dimensional feature space is the latent space. Based on the mathematical derivation of the Markov forward noise addition process, when the number of consecutive steps of applying Gaussian noise to the image features reaches a set maximum, the data distribution of the original image will degenerate into an isotropic pure noise state.

[0103] Based on the aforementioned physical state, the feature encoding initialization module 200 sets the total number of denoising iterations to the maximum denoising time step. The feature encoding initialization module 200 calls the random number generation operator of the graphics processor in the runtime environment to generate a random noise tensor conforming to a standard normal distribution in the latent space. Its mathematical formula is expressed as:

[0104] ;

[0105] in, Represents a Gaussian distribution; 0 indicates that the mean vector of the distribution is zero. The covariance matrix representing the distribution is the identity matrix; This represents the maximum denoising time step, and the total number of iterations in the reverse denoising process is set to a positive integer greater than or equal to 50. For the underlying hardware call interface logic of the random number generation operator, those skilled in the art can use existing deep learning tensor generation functions, which are well-known technologies in the field and will not be elaborated upon here.

[0106] The feature encoding initialization module 200 generates a random noise tensor that conforms to a standard normal distribution. Bind and specify as the maximum denoising time step The corresponding initial latent variables. The initial latent variables are a multidimensional floating-point data array, with data in the form of... .

[0107] The dimensions mentioned above correspond, in order, to the inference batch size, the number of latent feature channels, the latent height, and the latent width. In the specific system parameter configuration, The value of is a positive integer; The number of feature channels is set to a constant of 4. and The two-dimensional spatial resolution of the latent space is represented by values ​​set to one-eighth of the pixel height and pixel width corresponding to the target output visual creative content. The initial latent variables serve as the starting data for the diffusion model denoising network to perform inverse trajectory generation, providing the initial latent space state for subsequent iterations.

[0108] See attached document Figure 5 In this embodiment, the specific implementation of the attention intervention module 300 includes the following sub-steps:

[0109] S301, the attention intervention module 300 establishes the data flow mode of hidden variables when performing loop iteration.

[0110] After the reverse generation process is started, the attention intervention module 300 sets the denoising time step parameter to: The maximum denoising time step parameter is Noise reduction time step The value is determined according to The order gradually decreases.

[0111] When assigning data values, if the denoising time step is the maximum denoising time step, that is... At this time, the attention intervention module 300 extracts the random noise tensor that conforms to the standard normal distribution generated in the aforementioned feature encoding initialization stage, and assigns the current latent variable as the initial latent variable; when the denoising time step is less than the maximum denoising time step (i.e., When the attention intervention module 300 extracts the preceding time step (i.e., ... After denoising and iteration, the output tensor assigns the current latent variable the value of the next latent variable output from the previous iteration. The current latent variable is denoted as... This ensures continuous data transfer between each denoising time step.

[0112] S302, at each denoising time step, the attention intervention module 300 generates a query matrix using the current latent variables, and calculates the global cross-attention matrix using the query matrix and the key matrix.

[0113] The denoising network in the diffusion model employs a pre-trained U-shaped convolutional neural network (CNN) with its weights frozen during the current inference process. In the pre-training phase of model building, the U-shaped CNN utilizes a large-scale open-source real-world image-text pairing dataset as training samples. During training, a pre-trained variational autoencoder first compresses the images in the training samples into the latent space to obtain the true latent variables. Then, true Gaussian noise is sampled and added to the true latent variables according to a Markov forward process, generating noisy latent variables. Subsequently, the noisy latent variables, the temporal embedding vector, and the conditional features of the paired text are input into the U-shaped CNN, which outputs the predicted noise distribution.

[0114] In this process, real Gaussian noise is used as the label data, mean squared error (MSE) is used as the network loss function to calculate the error between the predicted noise distribution and the label data, and the weight parameters of the U-shaped convolutional neural network are updated using the backpropagation algorithm and optimizer until the model converges.

[0115] The trained U-shaped convolutional neural network is used to predict and remove noise components in the latent space. In terms of connectivity, the U-shaped convolutional neural network sequentially includes a downsampling feature extraction layer, an intermediate bottleneck layer that incorporates self-attention and cross-attention feature fusion calculations, and an upsampling feature reconstruction layer that combines skip connection structures. Current latent variables As having Dimensional data tensors, and denoising time steps The temporal embedding vector generated by sinusoidal position encoding is fed into the U-shaped convolutional neural network.

[0116] After passing through the convolutional processing of the downsampling feature extraction layer and the intermediate bottleneck layer, the data stream is mapped to a latent feature sequence representation containing global contextual information via a feature flattening operation. The attention intervention module 300 extracts the latent feature sequence representation within the network layers and performs a linear transformation on it to project it onto the attention feature space to generate a query matrix. The mathematical formula for this process is:

[0117] ;

[0118] in, For the generated query matrix; Indicates the current hidden variable The latent feature sequence representation derived from the above network convolution and flattening operations; This is a pre-trained, convergent, fixed query weight matrix used in the intermediate bottleneck layer based on large-scale image-text pairing data. Query matrix It represents the query response of the current latent variable to the semantic features of the text at each two-dimensional spatial location in the latent space.

[0119] Get query matrix Then, the attention intervention module 300 calculates the query matrix. The dot product is taken with the transpose of the key matrix generated in the aforementioned text feature mapping stage. To control the numerical magnitude of the dot product result, the attention intervention module 300 normalizes the matrix product by dividing it by the feature vector scaling dimension, and then performs a normalized exponential function operation along the feature sequence dimension on the normalized result to obtain the global cross-attention matrix. The complete calculation formula is as follows:

[0120] ;

[0121] in, This is the output global cross-attention matrix; Let be the transpose of the key matrix. The dimension of the feature vector is scaled to match the number of channels in the attention feature space; Represents the normalized exponential function. Global cross-attention matrix. It represents the response activation intensity between each spatial location within the current latent space image and each word in the input text sequence.

[0122] S303, the attention intervention module 300 extracts the attention subgraph corresponding to a specific entity in the set of independent creative entities from the slice of the global cross-attention matrix, and constructs a dynamic two-dimensional Gaussian mask for the specific entity that combines the denoising time step and the entity semantic dependency matrix.

[0123] The global cross-attention matrix contains response data for all text terms. For a specific entity requiring physical space constraints, let its object index in the set of independent creative entities be denoted as . The attention intervention module 300 performs slice retrieval along the sequence dimension based on the lexical index mapping relationship between specific entities and text prompt sequences, extracts channel response data of one or more lexical units corresponding to specific entities, and fuses the channel response data of multiple lexical units to obtain the attention subgraph corresponding to the specific entity. The attention subgraph is a two-dimensional floating-point numerical tensor, and its highly activated regions correspond to the feature-forming positions of a specific entity in the current generation stage.

[0124] To constrain the spatial diffusion range of the attention subgraph, the attention intervention module 300 extracts the two-dimensional center coordinates, bounding box width, and bounding box height corresponding to a specific entity from the aforementioned set of bounding box coordinates. Based on the resolution scaling ratio between the latent space and pixel space, or between the attention subgraph and pixel space, it linearly maps these two-dimensional center coordinates, bounding box width, and bounding box height from the pixel space to the two-dimensional coordinate system where the attention subgraph resides. The mapped two-dimensional center coordinates are then set as the expected dwell point in the two-dimensional space, and this is used as the mean vector corresponding to the specific entity. Using this mean vector As anchors, the attention intervention module 300 constructs an initial covariance matrix to normalize the feature distribution boundary. .

[0125] As a preferred parameter setting method, the initial covariance matrix Set as a diagonal matrix, the values ​​of its diagonal elements are directly calculated by proportionally mapping the squares of the width and height of the bounding box corresponding to a specific entity. To ensure that the generated range of the entity gradually focuses during the denoising process, the attention intervention module 300 introduces a time-step decay coefficient. Time step decay coefficient Set as noise reduction time step With maximum denoising time step The ratio of .

[0126] Based on the physical dependencies between entities, the attention intervention module 300 queries the aforementioned entity semantic dependency matrix. When the entity semantic dependency matrix The index is determined as Specific entities and indexes are When the target entity has a physical support relationship (i.e., the corresponding entity semantic dependency matrix element is 1), a spatial occlusion relationship (i.e., the corresponding entity semantic dependency matrix element is 2), or a light source and shadow mapping relationship (i.e., the corresponding entity semantic dependency matrix element is 3), the attention intervention module 300 triggers an anisotropic contraction strategy. The attention intervention module 300 extracts the two-dimensional center coordinates corresponding to the target entity from the aforementioned spatial bounding box coordinate set and sets them as the mean vector corresponding to the target entity. The attention intervention module 300 calculates the mean vector. Pointing to the mean vector corresponding to the target entity unit direction vector Its mathematical formula is defined as:

[0127] ;

[0128] Based on this calculation result, the attention intervention module 300 constructs a system containing unit direction vectors. and its orthogonal vectors The spatial basis matrix.

[0129] Based on the spatial basis matrix, the attention intervention module 300 controls the initial covariance matrix. unit direction vector The variance decay calculation is stopped at the top, maintaining a large feature fusion window, while only fusion is performed on orthogonal vectors. Multiply by the time step decay coefficient in the dimension Perform variance shrinkage to output a dynamic covariance matrix. If the entity semantic dependency matrix If no physical support relationship, spatial occlusion relationship, or light source and shadow mapping relationship is determined between a specific entity and the target entity, then the attention intervention module 300 will base its intervention on the initial covariance matrix. With time step decay coefficient Synchronous variance decay is performed in both spatial directions to generate a dynamic covariance matrix. .

[0130] After the above covariance modulation, the attention intervention module 300 outputs a dynamic two-dimensional Gaussian mask using a multidimensional Gaussian probability density function. Dynamic 2D Gaussian Mask At any point in the spatial coordinate system The formula for calculating the mask weight is:

[0131] ;

[0132] in, Represents a dynamic two-dimensional Gaussian mask At any point The mask weight value at the location; This forms a two-dimensional spatial coordinate column vector; This is the inverse matrix of the calculated dynamic covariance matrix. Dynamic two-dimensional Gaussian mask. Essentially, it is a continuous weight matrix whose values ​​are normalized to the interval [0, 1].

[0133] S304, the attention intervention module 300 constructs an energy loss function using a dynamic two-dimensional Gaussian mask and attention subgraph, calculates the partial derivative gradient of the energy loss function with respect to the current latent variable, and uses the partial derivative gradient to update the current latent variable to obtain the updated latent variable.

[0134] Using a dynamic two-dimensional Gaussian mask, the attention intervention module 300 applies mathematical penalties to attention overflow in irrational regions. When constructing the penalty mechanism, the attention intervention module 300 initializes a one-dimensional matrix in memory with the exact same dimensions as the dynamic two-dimensional Gaussian mask, and calculates the difference matrix between the one-dimensional matrix and the dynamic two-dimensional Gaussian mask. The high-value regions of the difference matrix essentially cover the background space where specific entities should not appear. Next, the Hadamard product of the attention subgraph corresponding to the specific entity and the difference matrix is ​​calculated to filter out abnormal attention activation values ​​that violate the physical spatial layout rules.

[0135] The energy loss function is constructed by summing the squared norms of the Hadamard products of all controlled specific entities. Its mathematical formula is defined as follows:

[0136] ;

[0137] in, The energy loss function is constructed; The total number of specific entities for which spatial constraints are enforced; This represents the Hadamard product operation, which multiplies corresponding elements. Represents a matrix of all ones; This indicates the solution for the Frobenius matrix norm.

[0138] After obtaining the energy loss function, the attention intervention module 300 calls the automatic differential computation graph engine of the underlying deep learning framework. While keeping all neural network layer weights frozen, the attention intervention module 300 calculates the partial derivative gradient of the energy loss function with respect to the current latent variables based on the chain rule. To prevent excessive damage to the manifold structure in a single update, a step size hyperparameter is set to control the update intensity. .

[0139] In this embodiment, to ensure effective correction of spatial bias without violating the Gaussian noise distribution assumption within the latent space, the step size hyperparameter... The value of is limited to a floating-point number within the interval [10.0, 50.0]. The gradient of the partial derivative is... With step size hyperparameter The product of these factors is used as the gradient update amount, and the current latent variable is updated accordingly. Subtracting the gradient update amount yields the updated latent variables. The updated formula is:

[0140] ;

[0141] Hidden variables after update It is an effective feature carrier that has been modified by explicit physical space punishment and relational intervention within the latent space.

[0142] S305, the attention intervention module 300 uses a denoising network to eliminate noise components from the updated latent variables and outputs the next latent variable for the next denoising time step.

[0143] After completing the gradient data intervention for the current step, the attention intervention module 300 uses a denoising network, namely the aforementioned pre-trained U-shaped convolutional neural network, to update the latent variables. Perform feature processing.

[0144] Specifically, the denoising network receives updated latent variables. And based on the current time step embedding vector and the aforementioned key matrix The semantic context information of the text is represented by the inner convolutional layer, which is used to update the latent variables. The forward propagation prediction is performed, and the corresponding noise component residual distribution is output. As a specific scheduling implementation, the attention intervention module 300 integrates the Denoising Diffusion Implicit Model (DDIM) algorithm as an ordinary differential equation scheduler. The attention intervention module 300 uses the DDIM algorithm to perform denoising integral calculation based on the predicted noise component residual distribution, thereby updating the latent variables. Eliminate noise components, and then deterministically output the next latent variable corresponding to the next denoising time step. .

[0145] See attached document Figure 6 In this embodiment, the specific implementation of the image decoding module 400 includes the following sub-steps:

[0146] S401, the image decoding module 400 obtains the final deterministic latent variable when the denoising time step decreases to the latent variable corresponding to the output time step zero.

[0147] The backward denoising process of the diffusion model is executed according to a loop control structure. In each iteration, the denoising network continuously extracts the predicted noise component from the current latent variable features, and the denoising time step decreases strictly by integer steps. When the denoising time step decreases to the point where the latent variable corresponding to time step zero is output, the numerical integration calculation of the denoising ordinary differential equation terminates. At this point, the image decoding module 400 extracts the latent variable corresponding to time step zero of the last forward propagation calculation of the denoising network and determines this latent variable as the final deterministic latent variable, denoted as... .

[0148] Final deterministic latent variables It is a multidimensional floating-point data array whose data dimensions are maintained at 1. .in, Corresponding to the batch size of the inference, Corresponding to the number of latent feature channels, Corresponding to the depth of the submerged layer, Corresponding to the width of the subsurface.

[0149] Final deterministic latent variables The low-dimensional feature distribution state after filtering out Gaussian noise components is characterized. Due to the dynamic intervention of cross-attention in the preceding steps, the final deterministic latent variables are obtained. The internal numerical distribution contains textual semantic information of a set of independent creative entities and maps to constraint information corresponding to the spatial bounding box, occlusion relationship and light and shadow mapping relationship.

[0150] S402, the image decoding module 400 inputs the final deterministic latent variable into the variational autoencoder to map the latent space features to a high-dimensional pixel space to complete image generation and output visual creative content.

[0151] To reconstruct low-dimensional latent feature data into a visible image, the image decoding module 400 calls a pre-trained variational autoencoder (VAE). During the model building and pre-training phases, the variational autoencoder uses a large-scale open-source high-resolution real image dataset as training samples, and the training samples themselves also serve as label data for the reconstruction target.

[0152] During training, image samples are input into the encoder network of the variational autoencoder for downsampling and convolutional compression, and two latent variables representing the mean and variance are output. Then, latent space features are obtained by sampling based on the mean and variance using the reparameterization technique. The latent space features are then input into the decoder network of the variational autoencoder for upsampling to restore the reconstructed image.

[0153] During this training process, a reconstruction loss function is constructed using the mean squared error between the reconstructed image and the initial input image samples, along with perceptual loss. A total loss function is constructed by combining this with the Kullback-Leibler Divergence (KL divergence), which measures the difference between the latent space feature distribution and the standard normal distribution. The network weights of the variational autoencoder are updated using backpropagation and a gradient descent optimizer until convergence. After training, the decoder network in the variational autoencoder is used to reconstruct the latent space features into the image pixel distribution.

[0154] The decoder network of the variational autoencoder sequentially comprises an input projection layer, stacked residual convolutional modules, multiple nearest-neighbor interpolation upsampling layers, and a final output convolutional layer for channel number restoration. During the current inference process, the image decoding module 400 acquires the final deterministic latent variables... The data is fed into the decoder network of the variational autoencoder. The data stream first undergoes dimension matching through the input projection layer, then sequentially passes through residual convolutional modules for feature refinement, and then undergoes progressive spatial dimension amplification through multiple nearest-neighbor interpolation upsampling layers. Finally, it is output through the final output convolutional layer. The mathematical representation of the data mapping process is:

[0155] ;

[0156] in, This represents the decoder network function of a pre-trained, converged variational autoencoder. The final deterministic latent variable for the input; This is the output pixel-level image tensor.

[0157] After upsampling and decoding by the decoder network, the output pixel-level image tensor The data dimensions are expanded to .in, This is the batch size parameter; The value is fixed at 3, representing the three basic color channels: red, green, and blue. and These represent the pixel height and pixel width of the output image, respectively.

[0158] As a specific implementation configuration, the total upsampling rate of the variational autoencoder is set to a constant of 8, thus satisfying the corresponding relation. and Thus, the image decoding module 400 completes the conversion of latent space features into a high-dimensional pixel space.

[0159] Since the tensor values ​​output by the neural network are normalized and distributed within the floating-point range of [-1, 1], to adapt to the display format of conventional images, the image decoding module has 400 pairs of pixel-level image tensors. Perform a linear inverse normalization operation. Specifically, map and truncate the numerical values ​​to the standard pixel value range [0, 255], and convert the data type to 8-bit unsigned integer (uint8) format. The digital image matrix formed by the above post-processing calculations is the output visual creative content.

[0160] In a business context, visual creative content is a color image that not only conforms to the semantics of the input text but also meets the input constraints in terms of physical object spatial layout, occlusion relationships, and lighting mapping. The encoding and persistent storage logic for the generated pixel tensors into standard image format files (such as JPEG or PNG) can be implemented using existing computer vision operation library functions by those skilled in the art; this is well-known technology in the field and will not be elaborated upon here.

[0161] To further illustrate the implementation process and technical effects of this invention, the following description is provided in conjunction with specific application scenarios and experimental data. The specific numerical values, scenario parameters, and comparative experiments described below are only used to explain the implementation principle of this invention and do not limit the scope of protection of this invention.

[0162] The system is set to receive the following natural language input prompt sequence: a glowing magic lantern hangs in front of a wooden sign. The target output visual creative content is set to have a pixel height of 512 and a pixel width of 512.

[0163] The semantic parsing module 100 receives a sequence of natural language input prompts and calls a large language model to perform semantic processing on the sequence of natural language input prompts to extract a set of independent creative entities. ,in For magic lanterns, It is a wooden signboard. The semantic parsing module 100 simultaneously extracts the spatial bounding box coordinates of the set of independent creative entities. Extracted bounding box data The top-left and bottom-right corner coordinates are [100, 200, 250, 400], bounding box data. The coordinates of the top-left and bottom-right corners are [150, 100, 450, 300]. The semantic parsing module 100 constructs an entity semantic dependency matrix using a syntactic dependency relation analysis algorithm. During the construction process, since the natural language input prompt sequence indicated that the magic lantern was located in front of the wooden signboard, it was determined to be a spatial occlusion relationship. The semantic parsing module 100 set the entity semantic dependency matrix. Matrix elements The value of is 2.

[0164] The feature encoding initialization module 200 is connected to the semantic parsing module 100. The feature encoding initialization module 200 uses a text encoder to perform feature mapping on the set of independent creative entities to generate a key matrix. The feature encoding initialization module 200 initializes a random noise tensor conforming to a standard normal distribution in the latent space as the initial latent variable corresponding to the maximum denoising time step. Based on the latent layer height and width being set to a scaling factor of one-eighth of the corresponding pixel height and width, the data dimensions of the initial latent variable are [1, 4, 64, 64]. The feature encoding initialization module 200 sets the maximum denoising time step. The value is 50.

[0165] Attention intervention module 300 is connected to feature encoding initialization module 200. Attention intervention module 300 operates in the diffusion model according to the denoising time step. The loop iterates from 50 to 1. The time step is used for denoising. Taking the iterative node as an example, the attention intervention module 300 generates a query matrix using the current latent variables, and calculates the global cross-attention matrix using the query matrix and the key matrix. The attention intervention module 300 then slices from the global cross-attention matrix to extract specific entities from the independent creative entity set. The corresponding attention subgraph. Attention intervention module 300 extracts specific entities. The corresponding two-dimensional center coordinates are linearly mapped from the pixel space to the two-dimensional coordinate system where the attention subgraph is located to obtain the specific entity. The corresponding mean vector Similarly, the attention intervention module 300 extracts the target entity. The corresponding mean vector .

[0166] Due to the entity semantic dependency matrix Matrix elements Determine a specific entity With the target entity Spatial occlusion exists. The attention intervention module 300 calculates based on specific entities. The corresponding mean vector Pointing to the target entity The corresponding mean vector unit direction vector The attention intervention module calculates the time-step decay coefficient 300. Combining physical constraints, the attention intervention module 300 constructs a unit direction vector. and its orthogonal vectors The spatial basis matrix controls the initial covariance matrix in the unit direction vector. The calculation of variance decay stops at the top, in the orthogonal vector. The variance is reduced by multiplying the dimension by a time step decay factor of 0.8, resulting in a dynamic covariance matrix. The attention intervention module 300 uses a multidimensional Gaussian probability density function to output a dynamic two-dimensional Gaussian mask that combines the denoised time step and the entity semantic dependency matrix.

[0167] Attention intervention module 300 utilizes dynamic two-dimensional Gaussian masks and specific entities The corresponding attention subgraph is used to construct the energy loss function, and the partial derivative gradient of the energy loss function with respect to the current latent variable is calculated. The step size hyperparameter is then set. The attention intervention module 300 uses the product of the partial derivative gradient and the step size hyperparameter 20.0 as the gradient update amount, and uses the partial derivative gradient to update the current latent variable to obtain the updated latent variable. The attention intervention module 300 uses a denoising network to remove noise components from the updated latent variable and outputs the next latent variable for the next denoising time step 39.

[0168] The image decoding module 400 is connected to the attention intervention module 300. The image decoding module 400 obtains the final deterministic latent variables when the denoising time step decreases to the point where the latent variables corresponding to the zero output time step are output. The image decoding module 400 inputs the final deterministic latent variables into the variational autoencoder to map the latent space features to a high-dimensional pixel space, outputting a pixel-level image tensor with data dimensions [1, 3, 512, 512]. The image decoding module 400 performs a linear inverse normalization operation on the pixel-level image tensor to complete image generation and output visual creative content.

[0169] Experimental verification and effect comparison:

[0170] To verify the technical effectiveness of a multimodal large-scale AI creative generation system, the following experiments and comparative tests were conducted.

[0171] Test dataset and environment configuration:

[0172] A test set containing 5000 natural language input prompt sequences was constructed. Each natural language input prompt sequence contains 2 to 5 specific entities, and these entities must have at least one of the following relationships: physical support, spatial occlusion, or light and shadow mapping. The system was deployed in an environment with a graphics processor configured with a single 24GB video memory. The pixel height and pixel width corresponding to the target output visual creative content were both set to 512, the maximum denoising time step was set to 50, and the step size hyperparameter was set to 25.0.

[0173] Comparison Model:

[0174] The following two representative technical solutions were selected as the benchmark for this comparative test:

[0175] Basic diffusion model: This corresponds to the conventional text-to-image diffusion generation model in existing technologies (the standard StableDiffusion model framework). This basic diffusion model only relies on global text prompts and temporal embedding vectors to perform generation iterations, does not accept spatial bounding box coordinate sets as input conditions, and does not have internal computational intervention logic for specific entity spatial dependencies.

[0176] Bounding-Bounds Constrained Diffusion Model: This text-to-image diffusion generation model introduces hard layout control (GLIGEN model framework or layout generation technique based on attention mask hard truncation). This bounding-box constraint diffusion model receives the same set of spatial bounding box coordinates as the one presented in this paper, and during the inference phase, it forces the attention activation values ​​of specific entities to be restricted to the corresponding bounding box region. However, this bounding-box constraint diffusion model does not employ an entity semantic dependency matrix, lacks a cross-entity relation calculation mechanism, and cannot execute anisotropic contraction strategies in the time step dimension.

[0177] Evaluation indicators:

[0178] Entity bounding box intersection-union ratio: The numerical value of the intersection-union ratio between the area of ​​the corresponding two-dimensional geometric region generated by a specific entity in the output visual creative content and the extracted set of spatial bounding box coordinates.

[0179] Physical Relationship Accuracy: Based on the natural language input prompt sequence, the percentage of entities in the output visual creative content whose physical interaction relationships match the determined physical dependency type.

[0180] Fraser initial distance: The numerical distance between the data feature distribution of the generated visual creative content and the feature distribution of the real image. The lower the Fraser initial distance, the better the generation quality.

[0181] Experimental data and results comparison:

[0182] Model classification Solid bounding box intersection Accuracy of physical relationship Fraser's initial distance Basic diffusion model 34.2% 28.5% 18.2 Boundary box constraint diffusion model 79.1% 46.3% 22.7 This invention system 87.6% 93.1% 17.5

[0183] Results Analysis and Discussion:

[0184] In terms of entity bounding box intersection-union ratio (IUU), the system solution provided by this invention achieves a test value of 87.6%, which is higher than the basic diffusion model and the bounding box constrained diffusion model. Data demonstrates that this system effectively limits the abnormal generation of features in non-bounding box regions by extracting attention subgraphs corresponding to specific entities through an attention intervention module, constructing a dynamic two-dimensional Gaussian mask for specific entities to build an energy loss function, and calculating the gradient of the partial derivative of the energy loss function with respect to the current latent variable to update the current latent variable.

[0185] Regarding the accuracy of physical relationships, the system solution provided by this invention achieves a test value of 93.1%, higher than the 46.3% of the bounding box constraint diffusion model. The bounding box constraint diffusion model independently generates features within its preset constraint region, resulting in a lack of visual fusion features at entity boundaries. This system utilizes a syntactic dependency parsing algorithm to construct an entity semantic dependency matrix, and executes an anisotropic contraction strategy based on this matrix, outputting a dynamic covariance matrix and a dynamic two-dimensional Gaussian mask. This calculation step controls the directional data extension of features of specific entities toward the target entity, thereby accurately reproducing spatial occlusion relationships and lighting mapping relationships.

[0186] Regarding the initial Fraser distance, the bounding box constrained diffusion model, due to its hard truncation of latent space features during generation, disrupts the continuous feature manifold structure at the bottom layer, resulting in an initial Fraser distance of 22.7. The system provided in this invention utilizes the energy loss function and partial derivative gradients to calculate the updated latent variables by subtracting the gradient update amount from the current latent variables, thus preserving the continuous distribution properties of the latent space variables. In comparison, the initial Fraser distance of this system is 17.5, ensuring the global feature integrity of the output visual creative content.

[0187] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal large-scale AI creative generation system, characterized in that, include: The semantic parsing module extracts the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence, and constructs the entity semantic dependency matrix using the syntactic dependency relation analysis algorithm; The feature encoding initialization module maps the set of independent creative entities to generate a key matrix and initializes a random noise tensor in the latent space as the initial latent variable corresponding to the maximum denoising time step. The attention intervention module executes a loop iteration with decreasing noise reduction time steps. At each denoising time step, a query matrix is ​​generated using the current latent variables. The global cross-attention matrix is ​​calculated using the query matrix and the key matrix. Attention subgraphs corresponding to specific entities in the set of independent creative entities are extracted. A dynamic two-dimensional Gaussian mask is constructed by combining the denoising time step and the entity semantic dependency matrix. An energy loss function is constructed using the dynamic two-dimensional Gaussian mask and the attention subgraph. The partial derivative gradient is calculated to update the current latent variable to obtain the updated latent variable. The noise component of the updated latent variable is eliminated by the denoising network and the next latent variable is output. The image decoding module obtains the final deterministic latent variable when the denoising time step decreases to the latent variable corresponding to the output time step zero, and inputs the final deterministic latent variable into the variational autoencoder to output the visual creative content.

2. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The steps of the semantic parsing module to extract the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence include: The natural language input prompt sequence is input into a large language model to identify visual object nouns, and the identified visual object nouns are aggregated into the set of independent creative entities; By combining the natural language input prompt sequence with the large language model, the two-dimensional geometric region of each entity on the target generation canvas is predicted, and the boundary endpoint coordinates of the two-dimensional geometric region are extracted to form the spatial bounding box coordinate set.

3. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The steps of constructing the entity semantic dependency matrix using the semantic parsing module with the syntactic dependency relation analysis algorithm include: The natural language input prompt sequence is input into the syntactic parser to generate a syntactic dependency tree containing lexical nodes and dependency relation edges; Initialize a two-dimensional tensor as the semantic dependency matrix of the entity; Specifically, the shortest dependency path between any two entities in the set of independent creative entities is extracted from the syntactic dependency tree. The physical dependency type is determined based on the verb and preposition tags on the shortest dependency path. The matrix elements of the entity semantic dependency matrix are numerically assigned according to the determined physical dependency type.

4. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The step of the feature encoding initialization module in mapping the set of independent creative entities to generate a key matrix includes: Extract the entity strings from the set of independent creative entities, and use a preset text delimiter to concatenate each entity string in sequence to form a one-dimensional text prompt sequence; The text prompt sequence is input into the text encoder, which outputs a text feature vector containing global contextual semantic features. A linear mapping is then performed on the text feature vector to generate the key matrix.

5. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The attention intervention module generates a query matrix using the current latent variables, and the step of calculating the global cross-attention matrix using the query matrix and the key matrix includes: The current latent variable is input into the denoising network and mapped to a latent feature sequence representation. A linear transformation is performed on the latent feature sequence representation and projected onto the attention feature space to generate the query matrix. Calculate the product of the query matrix and the transpose of the key matrix, divide the product by the feature vector scaling dimension for normalization, and perform a normalization exponential function operation along the feature sequence dimension on the normalized result to obtain the global cross-attention matrix.

6. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The steps of the attention intervention module to extract the attention subgraph corresponding to a specific entity in the set of independent creative entities include: Based on the lexical index mapping relationship between the specific entity and the natural language input prompt sequence, slice retrieval is performed along the sequence dimension on the global cross-attention matrix to extract channel response data of the lexical corresponding to the specific entity; The channel response data is fused to obtain the attention subgraph corresponding to the specific entity.

7. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The steps of constructing a dynamic two-dimensional Gaussian mask combining the denoising time step and the entity semantic dependency matrix by the attention intervention module include: Map the two-dimensional center coordinates of the specific entity in the spatial bounding box coordinate set to the two-dimensional coordinate system where the attention subgraph is located to obtain the mean vector corresponding to the specific entity, and construct the initial covariance matrix with the mean vector corresponding to the specific entity as the anchor point. Query the entity semantic dependency matrix, and when it is determined that the specific entity has a physical dependency relationship with the target entity in the set of independent creative entities, such as a physical support relationship, a spatial occlusion relationship, or a light source and shadow mapping relationship, extract the mean vector corresponding to the target entity. Based on the mean vector corresponding to the specific entity and the mean vector corresponding to the target entity, a unit direction vector is determined, and an orthogonal vector of the unit direction vector is constructed. Combined with a preset time step decay coefficient, the initial covariance matrix is ​​controlled to stop the variance decay calculation on the unit direction vector. The variance shrinkage is performed by multiplying the orthogonal vector by the time step decay coefficient, and a dynamic covariance matrix is ​​output. When it is not determined whether the specific entity has a physical support relationship, spatial occlusion relationship or light source and shadow mapping relationship with any other entity in the set of independent creative entities, synchronous variance decay is performed on the two spatial directions based on the initial covariance matrix and the time step decay coefficient, and a dynamic covariance matrix is ​​output. The dynamic two-dimensional Gaussian mask is obtained by using the multidimensional Gaussian probability density function in combination with the mean vector corresponding to the specific entity and the inverse matrix of the dynamic covariance matrix.

8. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The attention intervention module utilizes the dynamic two-dimensional Gaussian mask and the attention subgraph to construct an energy loss function, and the step of calculating the partial derivative gradient to update the current latent variable includes: Initialize the all-one matrix, and calculate the difference matrix between the all-one matrix and the dynamic two-dimensional Gaussian mask; Calculate the Hadamard product of the attention subgraph and the difference matrix, and sum the squared norms of the Hadamard products of the specific entity as the energy loss function. The partial derivative gradient of the energy loss function with respect to the current latent variable is calculated based on the chain rule; The product of the partial derivative gradient and the step size hyperparameter is used as the gradient update amount, and the current latent variable is updated by subtracting the gradient update amount from the current latent variable.

9. The AI ​​creative generation system for multimodal large models according to claim 1, characterized in that, The steps of the image decoding module inputting the final deterministic latent variable into the variational autoencoder to output visual creative content include: The final deterministic latent variables are fed into the decoder network of the variational autoencoder for upsampling and decoding to output pixel-level image tensors; The visual creative content is obtained by performing a linear inverse normalization operation on the pixel-level image tensor.

10. A method for generating AI ideas from multimodal large models, applied to the AI ​​idea generation system for multimodal large models as described in any one of claims 1 to 9, characterized in that, Includes the following steps: Extract the set of independent creative entities and the set of spatial bounding box coordinates corresponding to the set of independent creative entities from the natural language input prompt sequence, and construct the entity semantic dependency matrix using the syntactic dependency parsing algorithm; A key matrix is ​​generated by mapping the set of independent creative entities, and a random noise tensor is initialized in the latent space as the initial latent variable corresponding to the maximum denoising time step; The loop iterates by decreasing the denoising time step. At each denoising time step, a query matrix is ​​generated using the current latent variables. The global cross-attention matrix is ​​calculated using the query matrix and the key matrix. Attention subgraphs corresponding to specific entities in the set of independent creative entities are extracted. A dynamic two-dimensional Gaussian mask is constructed by combining the denoising time step and the entity semantic dependency matrix. An energy loss function is constructed using the dynamic two-dimensional Gaussian mask and the attention subgraph. The partial derivative gradient is calculated to update the current latent variable to obtain the updated latent variable. The noise component of the updated latent variable is eliminated by the denoising network and the next latent variable is output. When the denoising time step decreases to the latent variable corresponding to the output time step zero, the final deterministic latent variable is obtained, and the final deterministic latent variable is input into the variational autoencoder to output the visual creative content.

Images