A propaganda material intelligent generation method and device based on image-text joint embedding

Abstract

The application discloses a kind of based on graphic-text joint embedding's propaganda material intelligent generation method and device.The based on graphic-text joint embedding's propaganda material intelligent generation method includes: obtaining the propaganda material generation model guided by trained graphic-text joint embedding;Obtain the propaganda material input by user;According to the target multi-modal joint condition embedding vector for guiding generation obtained by user input propaganda material;The target multi-modal joint condition embedding vector for guiding generation is input to the trained graphic-text joint embedding guided propaganda material generation model, to obtain propaganda material image.The application uses four-level hierarchical multi-layer causal semantic ontology structure as semantic basis, and is matched with multi-layer causal perception graph attention network, and learning split matrix is used to realize the separation of selling point essential factor and scene mixed factor, which can solve the technical defects of graphic-text selling point misplacement, material attribute distortion, format irregularity and the like.

Description

Technical Field

[0001] This application relates to the field of multimodal large model technology, specifically to a method and apparatus for intelligent generation of promotional materials based on image-text joint embedding. Background Technology

[0002] With the rapid development of the digital marketing industry, the demand for product promotional materials from e-commerce platforms and social content platforms is growing exponentially. Traditional methods of manually designing and producing promotional posters, product detail pages, and promotional images suffer from problems such as long lead times, high costs, and severe style homogenization, making it difficult to meet the needs of omnichannel, high-frequency, and differentiated marketing campaigns. In recent years, AI-generated content technology, with diffusion models and cross-modal semantic alignment technology at its core, has been gradually applied to the field of promotional material production, significantly improving content production efficiency. However, several fundamental technical bottlenecks still exist in its practical application in high-end marketing scenarios.

[0003] Existing multimodal promotional material generation solutions mainly revolve around text-to-image models guided by prompts, extracting semantic features from product text descriptions to guide the image generation model to output corresponding materials. This type of solution has the following technical shortcomings in practical applications:

[0004] First, the representation of selling points suffers from systematic causal bias, failing to isolate confounding factors from the scene. Existing cross-modal alignment solutions are generally based on general semantic matching models, achieving image-text association through global feature similarity. However, they cannot distinguish the causal boundaries between the product's essential selling point attributes and confounding factors such as shooting scene, background environment, and accompanying objects. For example, when a product image is taken in a high-end shooting scene, the model may mistakenly associate the high-end attributes of the scene with the product's selling points, leading to problems such as exaggerated selling points, distorted attributes, and a disconnect from the product's actual physical characteristics in the generated promotional materials, thus failing to guarantee the factual accuracy of the promotional content.

[0005] Secondly, the generation process suffers from insufficient hierarchical control, and layout constraints and style adaptation costs are high. Existing diffusion models often employ global prefix injection for conditional injection, which fails to provide precise hierarchical control over semantic themes, layout, and pixel details. The fixed layout structure required for promotional materials is difficult to embed as a rigid constraint in the generation process, easily leading to issues such as subject misalignment and layouts that do not conform to placement guidelines. Furthermore, existing style adaptation solutions often rely on fine-tuning training for single channels. Each new channel style requires retraining the adaptation parameters, making zero-sample style transfer impossible. This results in long adaptation cycles, high computational costs, and repeated fine-tuning that can disrupt the model's original factual understanding, leading to fact drift risks.

[0006] Third, the attribution accuracy of selling point dissemination effectiveness is insufficient, lacking quantitative support for heterogeneity. Current promotional content relies heavily on manual experience or simple correlation statistics for selecting and weighting selling points, failing to scientifically quantify the true causal effect of individual selling points on user conversion. Traditional causal analysis methods can only calculate the average effect of selling points on all users, unable to differentiate the value of selling points across different groups, channels, and application scenarios, nor can they construct counterfactual samples to verify the authenticity of causal effects. This results in confounding biases in selling point attribution results, failing to provide accurate effectiveness data for targeted content generation. Consequently, the generated promotional content struggles to match the core needs of the target audience, leading to unstable conversion rates.

[0007] Fourth, the text and image generation process is fragmented, resulting in poor consistency of multimodal content. Most existing solutions can only generate promotional text or image materials independently. The generation processes of text and images are independent of each other, making it impossible to achieve a one-to-one correspondence and semantic linkage at the selling point level. Manual matching of text and image content is not only inefficient, but also prone to problems such as misalignment of selling points and semantic inconsistency, making it difficult to meet the high standards of collaborative text and image expression required in high-end marketing scenarios.

[0008] This application constructs a semantic transmission framework based on a four-level hierarchical multi-layered causal semantic ontology structure. It sets only bottom-up unidirectional causal dependency edges, combined with a multi-layered causal perception graph attention network, a learnable split matrix, and contrastive learning decoupling loss. This can completely remove scene-related confounding factors and retain only the essential selling point factors as generation guidance, effectively solving the shortcomings of existing solutions that easily misjudge shooting scenes as product selling points and distort the facts of promotional materials. At the same time, it can weight the essential selling point factors based on a three-dimensional selling point dissemination weight matrix, and quantify the conversion value of selling points by combining audience, channel, and scene, so that the generated content is more in line with the needs of the campaign. Summary of the Invention

[0009] The purpose of this invention is to provide a method for intelligently generating promotional materials based on image and text co-embedding, which at least solves one of the above-mentioned technical problems.

[0010] One aspect of the present invention provides a method for intelligently generating promotional materials based on image-text joint embedding, the method comprising:

[0011] Step 1: Obtain a trained promotional material generation model guided by image-text joint embedding;

[0012] Step 2: Obtain the promotional materials input by the user;

[0013] Step 3: Obtain the target multimodal joint conditional embedding vector for guiding generation based on the promotional materials input by the user;

[0014] Step 4: Input the target multimodal joint conditional embedding vector used for guidance into the trained image-text joint embedding guided promotional material generation model to obtain promotional material images.

[0015] Optionally, the promotional material generation model guided by the joint embedding of text and images includes:

[0016] A multi-granularity cross-modal conditional decoupling and hierarchical injection module is used to receive the target multi-modal joint conditional embedding vector, combine it with the current time step encoding generated during the diffusion iteration process, perform multi-granularity semantic decomposition and orthogonal decoupling processing on the joint embedding vector, and generate dynamic conditional enhancement features corresponding to different generation levels.

[0017] The layout prior encoding and injection module is used to extract the layout prior mask from the acquired user-input promotional materials, perform convolutional encoding and resolution alignment processing on the layout prior mask, and generate layout constraint features.

[0018] The hierarchical denoising UNet backbone network is used to receive initial noisy latent space features, diffusion time step encoding, dynamic conditional enhancement features and layout constraint features as generation guidance conditions. After multi-level condition-guided diffusion denoising operation, it outputs the denoised latent space features.

[0019] The latent space encoding and decoding module is used to receive latent space features, decode them to restore the image data in pixel space, and output the final promotional material image. During the training phase, the latent space encoding and decoding module is also used to perform encoding compression on the original promotional material image and output standardized latent space features for model training.

[0020] Optionally, the multi-granularity cross-modal conditional decoupling and hierarchical injection module includes a conditional decoupling subunit and a hierarchical gated injection subunit; wherein,

[0021] The conditional decoupling subunit is used to receive the target multimodal joint conditional embedding vector, and to map the joint embedding into three subspace vectors of semantic representation, style representation and detail representation through a three-way parallel projection network. Orthogonal constraint processing is performed on the three subspace vectors to output global semantic embedding, layout style embedding and element detail embedding.

[0022] The hierarchical gated injection subunit is used to receive the current time step encoding of global semantic embedding, layout style embedding and element detail embedding and diffusion iteration process, calculate the dynamic injection weights of the corresponding time steps of receiving global semantic embedding, layout style embedding and element detail embedding respectively, and inject global semantic embedding, layout style embedding and element detail embedding into the hierarchical denoising UNet backbone network respectively through cross attention mechanism, and generate dynamic conditional enhancement features of the corresponding level by combining the dynamic injection weights.

[0023] Optionally, the layout prior encoding and injection module includes a mask extraction subunit, a layout encoding subunit, and a feature alignment and fusion subunit; wherein,

[0024] The mask extraction subunit is used to perform region semantic segmentation on the acquired user input promotional materials, identify three types of functional areas in the materials: product display area, title copy area and white space area, and generate a binary layout prior mask.

[0025] The layout coding subunit is used to receive the layout prior mask, extract multi-scale layout structure features through multi-layer convolutional network and residual unit, and enhance the feature weight of core functional area and suppress the feature weight of white space area through spatial attention mechanism, and output the enhanced layout feature map.

[0026] The feature alignment and fusion subunit is used to perform resolution upsampling and channel dimension alignment processing on the enhanced layout feature map to generate layout constraint features that match the resolution of the hierarchical denoising UNet backbone network.

[0027] Optionally, the hierarchical denoising UNet backbone network includes a semantic control layer, a style layout layer, and a detail control layer, with the three layers connected sequentially from low to high feature resolution.

[0028] The semantic control layer includes a residual attention block at the end of the downsampling path, multiple bottleneck residual attention blocks, and a residual attention block at the front end of the upsampling path, corresponding to the lowest resolution feature dimension of the network. It is used to access the global semantic corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, and to perform denoising feature operation of the global semantic dimension in combination with diffusion time step encoding, and output the semantically constrained deep features to the style layout layer.

[0029] The style layout layer includes a residual attention block in the middle of the downsampling path and a residual attention block in the middle of the upsampling path, corresponding to the resolution feature dimension in the network. It is used to simultaneously access the layout style corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, as well as the layout constraint features output by the layout prior encoding and injection module. Combined with diffusion time step encoding, it performs denoising feature operation on the style layout dimension and outputs the mid-layer features after style and layout constraints to the detail control layer.

[0030] The detail control layer includes a residual attention block at the front end of the downsampling path and a residual attention block at the end of the upsampling path, corresponding to the highest resolution feature dimension of the network. It is used to access the element detail corresponding to the dynamic condition enhancement features output by the multi-granularity cross-modal conditional decoupling and hierarchical injection module, and to perform denoising feature operations on the pixel detail dimension in combination with diffusion time step encoding, and finally output the denoised latent space features.

[0031] Optionally, the latent space encoding / decoding module includes an encoder subunit and a decoder subunit;

[0032] The encoder subunit is used to receive the original promotional material image during the model training phase, and to perform dimensionality compression processing on the image through a multi-level convolutional downsampling unit and a residual operation unit, mapping the image data in the pixel space to a low-dimensional latent space, and outputting standardized latent space features as the training input of the hierarchical denoising UNet backbone network.

[0033] The decoder subunit is used to receive the denoised latent space features output by the hierarchical denoising UNet backbone network during the material generation stage. It performs dimension restoration processing on the latent features through multi-level transposed convolutional upsampling units and residual operation units, mapping the low-dimensional latent space data back to the pixel space and outputting the final promotional material image.

[0034] Optionally, the promotional material generation model guided by the joint embedding of text and images is obtained through iterative training using a multi-objective joint loss function. The total loss function during the training process is as follows:

[0035] ;

[0036] in, For the total loss, For layered noise prediction loss, For hierarchical cross-modal alignment loss, Due to layout constraints, For conditional orthogonal constraint loss, , , These are the preset balancing weight hyperparameters;

[0037] The hierarchical noise prediction loss is:

[0038] ;

[0039] in, , , The mean squared error of noise prediction for the semantic control layer, style layout layer, and detail control layer are respectively. , , Preset weights for the corresponding levels;

[0040] The hierarchical cross-modal alignment loss is:

[0041] ;

[0042] in, , , These are the global average pooling vectors representing the output features of the semantic control layer, style layout layer, and detail control layer, respectively. , , These are global semantic embedding, layout style embedding, and element detail embedding, respectively. The cosine similarity function;

[0043] The format constraint loss is:

[0044] ;

[0045] in, To generate salient subject masks corresponding to latent features, For layout prior mask, The function for calculating the intersection-union ratio;

[0046] The loss due to the orthogonal constraint is:

[0047] ;

[0048] in, The inner product of the global semantic embedding vector and the layout style embedding vector;

[0049] The inner product of the global semantic embedding vector and the element detail embedding vector;

[0050] The inner product of the vectors embedded for layout style and the vectors embedded for element details.

[0051] Optionally, step 3 includes:

[0052] Step 31: Construct a four-level hierarchical multi-layer causal semantic ontology structure. The multi-layer causal semantic ontology structure includes, from bottom to top, a pixel parameter layer, an entity layer, a selling point layer, and a value layer. Only one-way causal dependency edges are set between nodes of adjacent levels from bottom to top. No association edges are set between nodes of the reverse level and non-adjacent levels. This blocks the transmission path of non-causal mixed information from the ontology structure level.

[0053] Step 32: Extract the multimodal low-level features corresponding to the user-input promotional materials, attach all multimodal low-level features as leaf nodes to the pixel parameter layer, and construct a cross-modal causal heterogeneous graph according to the connection rules of the unidirectional causal dependency edges;

[0054] Step 33: Use a multi-layer causal perceptual graph attention network to perform message passing and feature aggregation along the causal forward direction on the cross-modal causal heterogeneous graph, and block the reverse and cross-layer information passing through the causal indicator factor mask to obtain the preliminary aggregated embedding vector corresponding to each selling point semantic node;

[0055] Step 34: Using a learnable split matrix, each initial aggregated embedding vector is split into essential selling point factors and scenario confounding factors. Combined with contrastive learning decoupling loss, optimization constraints are applied to make the essential factor features of the same selling point in different scenarios approach consistency and maximize the differences in scenario confounding factor features. After convergence, the essential selling point factor set corresponding to all selling points is extracted.

[0056] Step 35: Concatenate and linearly map the essential selling point factor set with the style semantic features input by the user to generate a target multimodal joint conditional embedding vector for guiding generation.

[0057] Optionally, after obtaining the essential factor set of selling points in step 34 and before generating the target multimodal joint conditional embedding vector in step 35, the method further includes:

[0058] Based on the historical feedback preprocessing dataset and the essential factor set of selling points, selling point perception processing variables, user feedback result variables, and three types of confounding covariates (audience, channel, and scenario) are constructed for each user feedback sample. The standardized observation dataset is obtained by splitting it according to the single selling point dimension.

[0059] Based on a pre-trained multimodal conditional result prediction network, under the premise of fixing all confounding covariates and non-target selling point features, counterfactual feedback results are generated where only the target selling point perception state is reversed, and are paired one-to-one with real samples to form a real-counterfactual paired sample set.

[0060] A causal forest model is used to fit the heterogeneous processing effect of each selling point. Multiple causal trees are constructed with the maximization of the difference in processing effect between nodes as the node splitting criterion. Discretization, aggregation and normalization are performed according to three dimensions: audience, channel and scenario, and a three-dimensional selling point dissemination weight matrix is ​​output.

[0061] Based on the target audience, target channel and target scenario corresponding to the current generated task, the propagation weight value corresponding to each selling point is indexed from the three-dimensional selling point propagation weight matrix, and the propagation weight value is used to perform factor-wise weighted adjustment on the essential factors of the corresponding selling point.

[0062] In step 35, the weighted and adjusted essential selling point factor set is concatenated and linearly mapped and fused with the style semantic features input by the user to generate a target multimodal joint conditional embedding vector for guiding generation.

[0063] This application also provides an intelligent promotional material generation device based on image-text joint embedding, the promotional material intelligent generation device based on image-text joint embedding includes:

[0064] The promotional material generation model acquisition module is used to acquire a trained promotional material generation model guided by image and text joint embedding.

[0065] A promotional material acquisition module, which is used to acquire promotional materials input by the user;

[0066] An embedding vector acquisition module is used to acquire a target multimodal joint conditional embedding vector for guiding generation based on promotional materials input by the user.

[0067] The promotional material image acquisition module is used to input the target multimodal joint conditional embedding vector used for guided generation into the trained image-text joint embedding guided promotional material generation model, thereby acquiring promotional material images.

[0068] This application constructs a multi-layered causal semantic ontology structure consisting of a value layer, a selling point layer, an entity layer, and a pixel layer. Traditional semantic association edges are replaced with causal dependency edges between nodes, clearly defining a unidirectional causal transmission relationship where pixel features support entity existence, entity features support the validity of selling points, and selling point features support value transmission, thus eliminating reverse causality and mixed associations. Image patch features, video frame features, text token features, and structured parameter features are used as four types of modal leaf nodes, connected to the ontology entity layer nodes, structurally blocking the aggregation path of mixed information. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating an embodiment of the intelligent generation method for promotional materials based on image-text joint embedding in this application. Detailed Implementation

[0070] To make the objectives, technical applications, and advantages of this application clearer, the technical applications in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0071] like Figure 1 As shown, the intelligent generation method for promotional materials based on image and text joint embedding described in this embodiment includes the following core steps:

[0072] Step 1: Obtain a trained promotional material generation model guided by image-text joint embedding;

[0073] Step 2: Obtain the promotional materials input by the user;

[0074] Step 3: Obtain the target multimodal joint conditional embedding vector for guiding generation based on the promotional materials input by the user;

[0075] Step 4: Input the target multimodal joint conditional embedding vector into the trained model and output the promotional material image.

[0076] In this embodiment, the promotional material generation model with combined text and image embedding includes:

[0077] A multi-granularity cross-modal conditional decoupling and hierarchical injection module is used to receive the target multi-modal joint conditional embedding vector, combine it with the current time step encoding generated during the diffusion iteration process, perform multi-granularity semantic decomposition and orthogonal decoupling processing on the joint embedding vector, and generate dynamic conditional enhancement features corresponding to different generation levels.

[0078] In this embodiment, the multi-granularity cross-modal conditional decoupling and hierarchical injection module includes a conditional decoupling subunit and a hierarchical gated injection subunit; wherein,

[0079] The conditional decoupling subunit is used to receive the target multimodal joint conditional embedding vector, and to map the joint embedding into three subspace vectors of semantic representation, style representation and detail representation through a three-way parallel projection network. Orthogonal constraint processing is performed on the three subspace vectors to output global semantic embedding, layout style embedding and element detail embedding.

[0080] The hierarchical gated injection subunit is used to receive the current time step encoding of global semantic embedding, layout style embedding and element detail embedding and diffusion iteration process, calculate the dynamic injection weights of the corresponding time steps of receiving global semantic embedding, layout style embedding and element detail embedding respectively, and inject global semantic embedding, layout style embedding and element detail embedding into the hierarchical denoising UNet backbone network respectively through cross attention mechanism, and generate dynamic conditional enhancement features of the corresponding level by combining the dynamic injection weights.

[0081] In this embodiment, the multi-granularity cross-modal condition decoupling and hierarchical injection module is the core unit for achieving precise hierarchical control of conditions. It is used to decompose a single global joint condition embedding into three functionally independent and orthogonal sub-granular representations of semantics, style, and details. Then, it dynamically adjusts the injection intensity of each level by combining the time step of diffusion iteration, and finally accurately matches the condition signals of different granularities to the corresponding control level of UNet. This fundamentally solves the technical problems of granularity mixing, insufficient control accuracy, and condition signal crosstalk in existing global condition injection schemes.

[0082] The inputs to this module include: the target multimodal joint conditional embedding vector and the current time step encoding generated in real time during the diffusion iteration process; the outputs are three sets of dynamic conditional enhancement features, corresponding to the semantic control layer, style layout layer, and detail control layer, respectively. The feature dimensions are fully matched with the channel dimensions of the corresponding UNet layers and can directly participate in the feature operations within the layers.

[0083] The module is divided into two parts: a conditional decoupling subunit and a hierarchical gated injection subunit. The structure, operation process and parameter design of each are described in detail below.

[0084] Conditional decoupling subunit:

[0085] The core function of the conditional decoupling subunit is to decompose the global joint embedding into three functionally independent sub-granular embeddings at the feature space level. Orthogonal constraints are used to enforce linear independence among the three subspaces, preventing crosstalk between conditions of different granularities and ensuring that adjusting a single-dimensional condition does not affect the generation effect of other dimensions. This subunit employs a three-way parallel projection network architecture with orthogonal regularization constraints. The three network structures are symmetrical but have independent parameters, corresponding to the semantic, style, and detail representation dimensions, respectively.

[0086] (1) Three-way parallel projection network:

[0087] Each projection network adopts a two-layer fully connected architecture: linear projection layer - normalization layer - GELU activation layer - linear projection layer. The specific structure and parameters of a single network are as follows:

[0088] The first linear projection layer has an input dimension of 768 (consistent with the joint embedding dimension) and an output dimension of 1024. Its function is to map the input embedding to a high-dimensional feature space, complete the initial semantic decomposition and feature dimensionality enhancement, and provide an expression space for subsequent nonlinear decoupling.

[0089] Layer normalization layer: Performs layer normalization (LayerNorm) on high-dimensional features. The normalization dimension is the feature channel dimension. It is used to stabilize the feature distribution during training and avoid gradient vanishing or exploding.

[0090] GELU Nonlinear Activation Layer: Introduces Gaussian error linear units as nonlinear activation functions, which have smoother gradient characteristics compared to ReLU, improving the network's ability to fit complex semantic relationships;

[0091] The second linear projection layer has an input dimension of 1024 and an output dimension of 512. It maps high-dimensional features to the corresponding subspace and outputs a single-granularity initial embedding representation.

[0092] The three projection networks operate in parallel, outputting initial global semantic embeddings, initial layout style embeddings, and initial element detail embeddings, each with an initial dimension of 512. The parameters of the three networks are independent, and each learns its corresponding granularity of feature extraction weights during training, without sharing them.

[0093] (2) Subspace orthogonal constraint processing

[0094] Based on the three-way projection output, a subspace orthogonality constraint mechanism is introduced to force the feature spaces of the three types of embeddings to be pairwise linearly independent. This constraint is achieved through an orthogonal regularization term, which participates in the end-to-end training of the model as a component of the total loss. The specific calculation formula is as follows:

[0095] ;

[0096] in, , , These are column vectors representing global semantic embedding, layout style embedding, and element detail embedding, respectively. This is a vector transpose operation.

[0097] During the training iterations, the regularization term continuously penalizes the non-orthogonality of the three types of embedding vectors, constantly reducing the absolute value of the vector inner product. After training convergence, the absolute value of the pairwise inner product of the three types of embedding vectors is less than 0.05, satisfying the constraint requirement of approximate linear independence. After orthogonal constraint calibration, the final global semantic embedding, layout style embedding, and element detail embedding are output and input into the hierarchical gated injection sub-unit.

[0098] 2. Layered gated injection subunit

[0099] The core function of the hierarchical gated injection subunit is to dynamically adjust the injection intensity of the three types of conditional embeddings according to the current stage of the diffusion iteration, and to accurately inject the conditional features into the corresponding layers of the hierarchical denoising UNet backbone network through a cross-attention mechanism, thereby achieving adaptive matching between the conditional signals and the generation process. This subunit consists of two parts: a time-step gated generation unit and a three-level cross-attention injection unit.

[0100] (1) Time step gating generation unit

[0101] The denoising iteration of the diffusion model exhibits a clear phased pattern: early iterations (large time steps) primarily determine the global semantics and overall layout, with a higher dependence on semantic and style conditions; late iterations (small time steps) mainly refine pixel details, with a higher dependence on detail conditions. This unit dynamically generates injection weights for three types of embeddings based on the encoding of the current time step, achieving adaptive matching with the diffusion stage. Specific implementation process:

[0102] Enhanced time-step position encoding: The time-step index t (an integer ranging from 0 to 1000) of the current diffusion iteration is mapped to a 256-dimensional sinusoidal time embedding, replacing the original scalar time step and providing richer temporal features for the gating network. The calculation formula is as follows:

[0103] ;

[0104] Where i is the dimension index, and its value range is... ; The total dimension for time embedding is 256 in this embodiment.

[0105] Dynamic weight generation network: The enhanced time-step encoding is input into the gating network, which outputs three dynamically injected weights in the 0-1 interval. The gating network adopts a two-layer fully connected structure.

[0106] First layer: Input 256-dimensional temporal embedding, output 128-dimensional features, followed by ReLU activation function;

[0107] The second layer: Inputs 128-dimensional features and outputs a 3-dimensional weight vector, which is then normalized to the [0,1] interval using a Sigmoid activation function to obtain the semantic injection weights. Style Injection Weight Detailed weighting .

[0108] The weight design follows the objective laws of the diffusion phase: when time step t > 500 (early iteration phase), it satisfies... > > Strong semantic constraints, weak detail constraints;

[0109] When time step t < 200 (late iteration stage), the following condition is met. > > Strong detail constraints and weak semantic constraints; the weights of the three tend to be balanced in the intermediate stage.

[0110] 2) The three-level cross-attention injection unit sets up an independent set of cross-attention operation units for each of the three layers of UNet: semantic control layer, style layout layer, and detail control layer. This injects conditional embeddings of the corresponding granularity into the feature operations of that layer. The operation logic of each set of cross-attention units is consistent. Taking the conditional injection of the semantic control layer as an example, the specific process is as follows:

[0111] Feature sequence reshaping: Reshape the current feature map (size: ) of the UNet semantic control layer Reshape the data into a sequence based on spatial location, and use it as the query vector Q for cross-attention, with dimension [missing information]. ,in = C represents the total number of pixels in the feature map, and C represents the number of feature channels at this level (the feature map size of the semantic control layer is 8×8×512, and the reshaped sequence length N=8×8=64; the feature map size of the style layout layer is 16×16×256, and the reshaped sequence length N=16×16=256; the feature map size of the detail control layer is 32×32×128, and the reshaped sequence length N=32×32=1024).

[0112] Conditional sequence construction: The global semantic embedding vector is expanded into a conditional sequence of length 1, serving as the key vector K and value vector V, with a dimension of 1. 512, aligned with the channel dimension of the query vector.

[0113] Scaling dot product cross attention operation: Perform standard scaling dot product cross attention calculation:

[0114] ;

[0115] In the formula, The dimension of the key vector is 512 in this embodiment; This is a scaling factor used to prevent the softmax gradient from becoming too large due to excessively large dot product results.

[0116] Weighted and residual fusion: reshaping the feature sequence output by cross-attention into the spatial size of the original feature map. Then, it is multiplied by the dynamically injected weights at the corresponding time step to obtain the weighted dynamic conditional enhancement feature; finally, the feature is added to the original input feature of the same level using residual addition, and then participates in the operation of the subsequent residual attention block.

[0117] The injection process for the style layout layer and the detail control layer is exactly the same as above, corresponding to the embedding of layout style and the embedding of element details, respectively. The style layout layer has 256 channels and the detail control layer has 128 channels. Both are aligned with the embedding dimension and the hierarchical channel dimension through an independent linear projection layer.

[0118] The multi-granularity cross-modal conditional decoupling and hierarchical injection module achieves precise decomposition and adaptation of conditional signals from global coarse-grained to hierarchical fine-grained through a three-level architecture of orthogonal decoupling, time-dynamic gating, and hierarchical cross-attention injection. Compared with the existing technology of global single conditional injection, this module can further improve the decoupling degree of control over different generation dimensions, while effectively avoiding the attenuation and crosstalk of conditional signals during the diffusion process. It can adjust the generation effect of any one dimension of semantics, layout, or detail without affecting other dimensions, significantly improving the controllability and accuracy of promotional material generation.

[0119] The layout prior encoding and injection module is used to extract the layout prior mask from the acquired user-input promotional materials, perform convolutional encoding and resolution alignment processing on the layout prior mask, and generate layout constraint features.

[0120] In this embodiment, the layout prior encoding and injection module includes a mask extraction subunit, a layout encoding subunit, and a feature alignment and fusion subunit; wherein,

[0121] The mask extraction subunit is used to perform region semantic segmentation on the acquired user input promotional materials, identify three types of functional areas in the materials: product display area, title copy area and white space area, and generate a binary layout prior mask.

[0122] The layout coding subunit is used to receive the layout prior mask, extract multi-scale layout structure features through multi-layer convolutional network and residual unit, and enhance the feature weight of core functional area and suppress the feature weight of white space area through spatial attention mechanism, and output the enhanced layout feature map.

[0123] The feature alignment and fusion subunit is used to perform resolution upsampling and channel dimension alignment processing on the enhanced layout feature map to generate layout constraint features that match the resolution of the hierarchical denoising UNet backbone network.

[0124] In this embodiment, the layout prior encoding and injection module is the core functional unit for achieving end-to-end controllability of the layout of promotional materials. It is used to parse structured layout area rules from user-provided reference promotional materials, convert discrete spatial area masks into continuous layout constraint features that can participate in neural network operations, and accurately inject them into the style layout layer of the hierarchical denoising UNet backbone network. It rigidly constrains the spatial position and proportion of the product body and copywriting area during the iterative process of diffusion generation, avoiding secondary operations of cropping and layout after generation. It fundamentally solves the technical problems of existing diffusion models being unable to accurately control the layout structure, the main body being prone to misalignment, and the area division not conforming to the delivery specifications.

[0125] The input to this module is the original promotional material image input by the user (in this embodiment, it is an RGB three-channel image with a resolution of 224×224); the output is a layout constraint feature, whose spatial resolution and channel dimension are completely matched with the style layout layer of the hierarchical denoising UNet backbone network, and can directly participate in the feature fusion operation of this layer.

[0126] The module is divided into three parts: mask extraction subunit, layout encoding subunit, and feature alignment and fusion subunit. The structure, operation process, formula definition and parameter design of each part are described in detail below.

[0127] Mask extraction subunit:

[0128] The core function of the mask extraction subunit is to perform semantic segmentation of the input promotional material image, automatically identify the functional area division in the material, generate a structured layout prior mask, and provide an annotated regional basis for subsequent layout feature extraction.

[0129] (1) Region Semantic Segmentation Network

[0130] This unit uses a lightweight semantic segmentation network, finely tuned for e-commerce promotional materials, as its segmentation backbone. The network encoder employs a MobileNetV2 architecture, and the decoder uses a three-level bilinear upsampling + convolutional structure, with the overall parameter count kept below 5M, balancing segmentation accuracy and inference speed. The network input is a 224×224×3 RGB promotional material image, and the output is a three-channel region probability map with the same resolution as the input. The three channels correspond to the pixel probabilities of three functional areas: product display area, title text area, and white space area.

[0131] (2) Binarization mask generation

[0132] The output three-channel region probability map is subjected to channel-by-channel threshold binarization to generate binarized prior masks for the corresponding three types of regions. The binarization calculation formula for a single channel is:

[0133] ;

[0134] Where c is the region category index, and its value range is... These correspond to the product display area, title and text area, and blank area, respectively.

[0135] For the binary mask of the c-th region at pixel coordinates The value at this location is 1, which indicates that the pixel belongs to the corresponding functional area, and 0, which indicates that it does not belong to it.

[0136] The probability map of the c-th class region output by the segmentation network at pixel coordinates The probability value at point [0, 1] takes a value in the range [0, 1].

[0137] The binarization threshold is preset to 0.5 in this embodiment, meaning that a pixel is identified as belonging to a certain region when the probability of the pixel belonging to that region is not less than 0.5.

[0138] (3) Post-processing of the mask

[0139] Morphological opening operations are performed on the generated three types of binary masks to eliminate fine noise and isolated pixels in the segmentation results, and to smooth the region edges. The morphological opening operation first performs erosion and then dilation; the calculation formula is as follows:

[0140] ;

[0141] in, This is the prior mask for the layout of the c-th region after post-processing. The erosion operator is used to shrink the region boundary and remove small, bright noise points. The expansion operator restores the main size of the region and smooths the edge contours; K is the morphological operation kernel, and in this embodiment, a 3×3 rectangular kernel is used. The final output is a layout prior mask composed of three types of region masks, with a total size of 224×224×3, which is input to the layout coding subunit.

[0142] Layout coding subunit:

[0143] The core function of the layout coding sub-unit is to perform multi-scale feature extraction on the layout prior mask, and to strengthen the feature weights of the core functional areas and suppress the weights of invalid blank areas through the spatial attention mechanism, so as to output an enhanced layout feature map with regional discrimination, providing a layout signal with strong representation ability for subsequent generation constraints.

[0144] This unit adopts a three-level architecture of multi-scale convolutional feature extraction, residual feature fusion, and spatial attention weighting. The specific computation process is as follows:

[0145] (1) Multi-scale convolution feature extraction

[0146] The input 224×224×3 prior mask first enters three parallel convolution branches, extracting layout structure features at different scales through convolution kernels of different lengths, capturing multi-granular layout information from local edges to global region division.

[0147] First branch: 3×3 convolutional layer with stride of 1, outputting fine-scale layout features of 224×224×32, capturing detailed layout information such as region edges and local shapes;

[0148] The second branch: a 3×3 convolutional layer with a stride of 2, outputs a medium-scale layout feature of 112×112×64, capturing the overall shape and relative position information of a single region;

[0149] The third branch consists of a 3×3 convolutional layer with a stride of 4, which outputs a coarse-scale layout feature of 56×56×128, capturing macro-layout information such as global region division and overall spatial proportion.

[0150] The features output from the three branches are processed by batch normalization (BN) and ReLU activation functions, then uniformly adjusted to a resolution of 56×56 through bilinear upsampling, and then stitched together according to the channel dimension to obtain a multi-scale fusion initial feature of 56×56×224.

[0151] (2) Deepening the characteristics of residual units

[0152] The initial features, fused across multiple scales, are input into two concatenated residual units to further enhance the semantic representation of the layout features. A single residual unit contains two 3×3 convolutional layers, two batch normalization layers, and one ReLU activation layer. The residual connections span two convolutional layers, and the calculation formula is as follows:

[0153] ;

[0154] in, This is the input feature map for the residual unit; The output feature map of the residual unit; , These are the first and second 3×3 convolutional layers, with kernel size of 3×3 and padding of 1 to keep the feature map size unchanged. , The first and second batch normalization layers are used to stabilize the feature distribution and accelerate training convergence; ReLU is a linear rectified activation function used to introduce non-linear expressive power. After processing by the two-stage residual units, the output is a depth layout feature map with a size of 56×56×224.

[0155] (3) Spatial attention weighting

[0156] Spatial attention weighting is applied to the deep layout feature map to strengthen the feature weights of core functional areas such as the product display area and title text area, while suppressing the ineffective feature weights of blank areas, ensuring that subsequent constraint generation focuses on the effective layout areas. The calculation formula for the spatial attention module is:

[0157] ;

[0158] in, The input is a depth layout feature map with dimensions of 56×56×224; To perform global average pooling along the channel dimension on the input feature map, we obtain an average pooled feature of size 56×56×1, which represents the average feature intensity at each spatial location; To perform global max pooling along the channel dimension on the input feature map, we obtain max pooled features of size 56×56×1, which represent the most salient feature intensity at each spatial location; It is a 7×7 convolutional layer used to map dual-channel pooling features to single-channel attention weight maps. The large-size convolutional kernel can capture a wider range of spatial dependencies. The Sigmoid activation function normalizes the weight values ​​to the [0,1] interval; To perform element-wise multiplication, the attention weight map is multiplied pixel by pixel with the original input features to complete the weighting process; The output is an enhanced layout feature map, with the same size as the input feature map, which is 56×56×224.

[0159] Feature Alignment Fusion Subunit

[0160] The core function of the feature alignment and fusion subunit is to adjust the enhanced layout feature map to a resolution and channel dimension that perfectly matches the style layout layer of the hierarchical denoising UNet backbone network, and inject it into the corresponding network layer in a residual fusion manner, so that the layout constraint participates in each step of the diffusion denoising iteration.

[0161] (1) Resolution and channel alignment

[0162] In this embodiment, the style layout layer feature size of the hierarchical denoising UNet backbone network is 16×16×256, and the enhanced layout feature map output by the layout encoding subunit is 56×56×224. First, the spatial resolution is compressed to 16×16 through two levels of convolutional downsampling with a stride of 2, and then the number of channels is aligned to 256 through a 1×1 convolution. Finally, the layout constraint features that perfectly match the style layout layer are output. The calculation formula is as follows:

[0163] ;

[0164] in, It is a 1×1 convolutional layer used to adjust the feature channel dimension without changing the spatial resolution; The layout constraint features for channel alignment are completely consistent with the feature dimensions of the style layout layer.

[0165] The layout prior mask starts from 224×224 in the pixel space, and after multi-scale convolution downsampling, it is finally compressed to the corresponding level of the latent space with a resolution of 16×16, ensuring that the layout constraints and the size of the latent space denoising operation are completely aligned.

[0166] If the target UNet layer resolution does not match the layout features, the spatial resolution is adjusted by transposed convolution upsampling or stride convolution downsampling, and then channel alignment is performed to ensure that the spatial size and number of channels are completely matched.

[0167] (2) Hierarchical feature injection

[0168] The obtained layout constraint features are added to the original input features of the style layout layer using element-wise residual summation to complete the injection of layout constraints. The fusion formula is as follows:

[0169] ;

[0170] in, The original input features for the style layout layer include intermediate features for diffusion denoising and style conditional features; The input features are used to inject layout constraints into the style layout layer, and the residual attention blocks at this level will then participate in the diffusion denoising operation.

[0171] The layout prior encoding and injection module employs a full-link design—semantic segmentation to extract layout rules, multi-scale encoding to enhance features, and hierarchical precise injection of constraints—to transform the layout specifications of promotional materials from reference images into rigid constraints during the generation process. Compared to existing technologies that rely on post-processing cropping and adjusting the layout, this module improves the compliance rate of the regional layout of generated materials. Furthermore, layout constraints participate in the diffusion and iteration process throughout, avoiding issues such as abrupt edges and subject truncation, and simultaneously meeting the dual requirements of layout standardization and natural generation.

[0172] The hierarchical denoising UNet backbone network is used to receive initial noisy latent space features, diffusion time step encoding, dynamic conditional enhancement features and layout constraint features as generation guidance conditions. After multi-level condition-guided diffusion denoising operation, it outputs the denoised latent space features.

[0173] In this embodiment, the hierarchical denoising UNet backbone network includes a semantic control layer, a style layout layer, and a detail control layer, with the three layers connected sequentially from low to high feature resolution.

[0174] The semantic control layer includes a residual attention block at the end of the downsampling path, multiple bottleneck residual attention blocks, and a residual attention block at the front end of the upsampling path, corresponding to the lowest resolution feature dimension of the network. It is used to access the global semantic corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, and to perform denoising feature operation of the global semantic dimension in combination with diffusion time step encoding, and output the semantically constrained deep features to the style layout layer.

[0175] The style layout layer includes a residual attention block in the middle of the downsampling path and a residual attention block in the middle of the upsampling path, corresponding to the resolution feature dimension in the network. It is used to simultaneously access the layout style corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, as well as the layout constraint features output by the layout prior encoding and injection module. Combined with diffusion time step encoding, it performs denoising feature operation on the style layout dimension and outputs the mid-layer features after style and layout constraints to the detail control layer.

[0176] The detail control layer includes a residual attention block at the front end of the downsampling path and a residual attention block at the end of the upsampling path, corresponding to the highest resolution feature dimension of the network. It is used to access the element detail corresponding to the dynamic condition enhancement features output by the multi-granularity cross-modal conditional decoupling and hierarchical injection module, and to perform denoising feature operations on the pixel detail dimension in combination with diffusion time step encoding, and finally output the denoised latent space features.

[0177] In this embodiment, the hierarchical denoising UNet backbone network is the core execution carrier of diffusion denoising operations. It breaks the paradigm of traditional UNet that only divides the encoding-decoding layers according to spatial resolution. Instead, it reconstructs the U-shaped encoding-decoding network into three functional layers according to the generation control function: semantic control layer, style layout layer, and detail control layer. These layers precisely connect to conditional signals and layout constraints of different granularities, enabling hierarchical and independent control of global semantics, mid-level layout, and pixel details. This solves the technical pain points of traditional diffusion models from the network architecture level, such as coarse conditional control granularity, weak layout constraints, and easy conflicts between detail optimization and semantic direction.

[0178] The input to this network includes four types of signals: initial noisy latent space features (random Gaussian noise latent features during inference and noisy real latent features during training), diffusion iteration current time step encoding, three sets of dynamic condition enhancement features output by the multi-granularity cross-modal conditional decoupling and hierarchical injection module, and layout constraint features output by the layout prior encoding and injection module; the output is the denoised latent space features, which are input to the decoder subunit of the latent space encoding and decoding module to restore the pixel image.

[0179] The network adopts a classic U-shaped symmetrical architecture, consisting of three parts: downsampling path, bottleneck layer, and upsampling path. All layers use residual attention blocks as the basic computational units. According to the control function, it is divided into semantic control layer, style layout layer, and detail control layer. The three layers are connected sequentially from low to high feature resolution, and together they cover the entire network structure.

[0180] 1. Basic computational unit: Residual attention block

[0181] The residual attention block is the smallest functional unit constituting the three control layers. Each unit integrates three types of operations: temporal embedding fusion, residual convolution feature extraction, and spatial self-attention enhancement, while also supporting the injection of external conditional features. The structure and operation flow of a single residual attention block are as follows:

[0182] (1) Diffusion time step encoding generation

[0183] Each denoising step in the diffusion model depends on the information of the current iteration time step. First, the discrete time step indices are converted into continuous high-dimensional embedding vectors, and the calculation formula is in the form of sinusoidal positional encoding:

[0184] ;

[0185] in, This is the time step index of the current diffusion iteration, with a value range of... In this embodiment, the total number of steps T = 1000; This is the time-step embedding vector for the output; i is the vector dimension index, with values ​​ranging from 1 to 2. ; This represents the total dimension of the time step embedding, which is 256 in this embodiment. These represent the even-numbered and odd-numbered dimensions of the vector, respectively.

[0186] The generated time-step embeddings will be injected into each residual attention block after linear projection, enabling the network to be aware of the current denoising stage.

[0187] 2) Residual convolution unit

[0188] The residual convolutional unit is responsible for the nonlinear transformation of features and the fusion of time-step information. It adopts a structure of two layers of convolution, time-step gating, and residual connections. The calculation formula is as follows:

[0189] ;

[0190] in, The input feature map for the residual attention block has dimensions H×W×C, where H is the feature map height, W is the feature map width, and C is the number of feature channels.

[0191] , It consists of two 3×3 convolutional layers with a padding value of 1, so the spatial size of the feature map remains unchanged before and after the operation; ReLU is a linear rectified activation function, which introduces non-linear expressive power. The time-step projection matrix has dimensions C× This is used to embed and map 256-dimensional time steps into vectors with the same number of channels as the input features, thereby achieving the fusion of time information and features; Embed the vector at the current time step; The output feature map of the residual convolutional unit, with dimensions similar to the input features. Figure 1 To.

[0192] If the current residual attention block corresponds to a level with external condition injection (dynamic condition enhancement feature or layout constraint feature), then the condition feature will be combined with... The output features are summed element by element before proceeding to the subsequent activation and second-layer convolution operations.

[0193] (3) Spatial self-attention unit

[0194] The features output by the residual convolutional unit are further fed into the spatial self-attention unit to model the dependencies between different spatial locations of the feature map, strengthening long-distance semantic associations. A standard scaled dot product self-attention mechanism is used, with the following calculation formula:

[0195] ;

[0196] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, derived from the input features. It is obtained by linear projection through three independent 1×1 convolutional layers, all three of which have the same dimension. ,in =H W represents the total number of spatial pixels in the feature map. For the feature dimension of the attention head; This is the transpose of the key matrix; This is a scaling factor used to prevent the gradient of the Softmax function from saturating due to an excessively large dot product result. The normalized exponential function outputs the attention weights for each spatial location. The input features for the residual connection are used to ensure the stability of gradient propagation. This is the final output feature map of the residual attention block, with the same size as the input.

[0197] Detailed structure of the three-level control hierarchy

[0198] Based on control functions and feature resolution, the network is divided into three layers from deep to shallow: semantic control layer, style layout layer, and detail control layer. These three layers are connected sequentially and respectively connect to conditional signals of corresponding granularities.

[0199] (1) Semantic control layer The semantic control layer is the core layer that controls the overall generation direction. It corresponds to the feature dimension with the lowest resolution of the network and is responsible for constraining the core subject and promotional theme of the generated content to avoid the generated content deviating from the product's selling points.

[0200] The semantic control layer includes two residual attention blocks at the end of the downsampling path, all three residual attention blocks in the bottleneck layer, and two residual attention blocks at the beginning of the upsampling path, for a total of seven residual attention blocks.

[0201] Feature resolution: The feature size at the downsampling end and bottleneck layer is 8×8×512, and the feature size at the upsampling front end is 16×16×256;

[0202] Access conditions: The global semantic correspondence of the dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module;

[0203] Calculation process:

[0204] The low-resolution features passed through the downsampling path enter the residual attention block of the semantic control layer, and are simultaneously injected with global semantic dynamic conditional enhancement features and the current time step encoding;

[0205] After multiple rounds of residual attention operations at the bottleneck layer, deep denoising feature operations are completed in the global semantic dimension to determine the main category of the generated content, the layout of core selling points, and other global semantic information.

[0206] The processed deep features are passed to the residual attention block at the front end of the upsampling path. Combined with the corresponding resolution downsampling features passed in by the skip connection, the semantically constrained deep features are output to the style layout layer.

[0207] The unique value of this layer lies in: concentrating global semantic constraints on low-resolution deep networks, controlling the direction of content from the source of generation, avoiding interference from mid-to-high-level detail optimizations with global semantic consistency, and significantly reducing the probability of generated content deviating from the selling points.

[0208] (2) Style layout layer

[0209] The style layout layer is the intermediate layer connecting semantics and details. It corresponds to the resolution feature dimension in the network and undertakes the dual functions of visual style control and layout structure constraint. It is the core carrier for achieving end-to-end controllability of layout.

[0210] The style layout layer includes 3 residual attention blocks in the middle of the downsampling path and 3 residual attention blocks in the middle of the upsampling path, for a total of 6 residual attention blocks;

[0211] Feature resolution: uniformly 16×16×256, which perfectly matches the layout constraint feature size output by the layout prior encoding and injection module;

[0212] Access conditions: The layout style output by the multi-granularity cross-modal condition decoupling and hierarchical injection module corresponds to the dynamic condition enhancement features, the layout prior coding and the layout constraint features output by the injection module;

[0213] Calculation process:

[0214] The medium-resolution features passed through the downsampling path are fed into the residual attention block of the style layout layer, and simultaneously injected with layout style dynamic condition enhancement features, layout constraint features and the current time step encoding.

[0215] After the two types of constraint features are fused with the original denoising features element by element, the visual style tone calibration and regional layout structure constraints are simultaneously completed through residual attention operation to ensure that the spatial position of the product body and the copywriting area conforms to the prior layout specifications.

[0216] The processed mid-level features are passed to the subsequent units of the upsampling path, and the mid-level features with style and layout constraints are output to the detail control layer.

[0217] The unique value of this layer lies in embedding layout constraints directly into the denoising process of the medium-resolution feature layer, rather than as a post-processing overlay. This allows the layout structure to blend naturally with the image content, avoiding problems such as harsh edges and subject truncation caused by post-processing cropping, while simultaneously achieving coordinated control of style and layout.

[0218] (3) Detail control layer

[0219] The detail control layer is a shallow network corresponding to the highest resolution, responsible for optimizing pixel-level texture, edges, and quality, thereby improving the clarity and realism of the generated image.

[0220] The detail control layer includes: two residual attention blocks at the beginning of the downsampling path and two residual attention blocks at the end of the upsampling path, for a total of four residual attention blocks;

[0221] Feature resolution: uniformly set to 32×32×128;

[0222] Access conditions: The element details output by the multi-granularity cross-modal condition decoupling and hierarchical injection module correspond to dynamic condition enhancement features;

[0223] Calculation process:

[0224] The initial noisy latent space features are first fed into the downsampled residual attention block of the detail control layer, where element-detail dynamic conditional enhancement features and time-step encoding are injected to complete the initial denoising and detail feature extraction at high resolution.

[0225] During the upsampling stage, the features transmitted from the style layout layer enter the upsampling residual attention block of the detail control layer, and then combine the detail conditions to optimize the pixel-level texture, edge and material texture.

[0226] Finally, the latent space features with 4 output channels are obtained through the terminal convolutional layer, which are the latent space features after denoising.

[0227] The unique value of this layer lies in: independently controlling the optimization of details in a high-resolution shallow network, avoiding interference from detail adjustments on the global semantics and layout, and achieving fine-grained control where the semantics remain unchanged while the details are adjustable.

[0228] 3. Skip connections and overall forward propagation process

[0229] The network employs a symmetric skip connection mechanism. The output features of each layer in the downsampling path are directly concatenated to the input of the corresponding resolution layer in the upsampling path via skip connections, supplementing the spatial detail information lost during downsampling. The concatenation calculation formula is as follows:

[0230] ;

[0231] in, This is the feature map output from the previous level of the upsampling path; This is the skip connection feature map at the resolution level corresponding to the downsampling path; This is a splicing operation along the channel dimension, and the number of feature channels after splicing is the sum of the two. This is the input feature map for the current upsampling level.

[0232] The complete forward propagation process is as follows:

[0233] The initial noisy latent space features (32×32×4) first enter the downsampled residual attention block of the detail control layer. After being constrained by detail conditions, the first downsampling is completed, and the output 16×16×128 features are entered into the style layout layer. The style layout layer completes the second downsampling, and the output 8×8×256 features are entered into the semantic control layer. Finally, after reaching the bottleneck layer, the feature size is 8×8×512, and the deep semantic denoising operation is completed.

[0234] Symmetrical upsampling path: The bottleneck layer output is upsampled to 16×16×256 by the semantic control layer, and then enters the style layout layer after fusing the corresponding skip connection features; the style layout layer is upsampled to 32×32×128, and then enters the detail control layer after fusing the skip connection features; finally, the latent space features are completed by denoising the 32×32×4 output by the convolution at the end of the detail control layer.

[0235] After the features reach the bottleneck layer, deep semantic denoising is performed in the semantic control layer.

[0236] The denoised deep features are upsampled step by step, passing through the upsampling units of the semantic control layer, style layout layer, and detail control layer in sequence. Each level integrates the features of the corresponding jump connection and injects the conditional signals and layout constraints of the corresponding level.

[0237] Finally, the output convolutional layer at the end of the detail control layer yields the denoised latent space features, which are then passed to the latent space encoding / decoding module.

[0238] The hierarchical denoising UNet backbone network achieves multi-dimensional precision and controllability in promotional material generation through a functional layering, hierarchical conditional injection, and layout embedding architecture. Compared to the traditional single-condition injection UNet structure, and compared to existing technologies, this network has a higher semantic consistency and layout compliance rate. It also supports individual adjustment of any one dimension—semantics, style, details, or layout—without interfering with other dimensions, significantly improving the controllability and quality of promotional material generation.

[0239] The latent space encoding and decoding module is used to receive latent space features, decode them to restore the image data in pixel space, and output the final promotional material image. During the training phase, the latent space encoding and decoding module is also used to perform encoding compression on the original promotional material image and output standardized latent space features for model training.

[0240] In this embodiment, the latent space encoding / decoding module includes an encoder subunit and a decoder subunit;

[0241] The encoder subunit is used to receive the original promotional material image during the model training phase, and to perform dimensionality compression processing on the image through a multi-level convolutional downsampling unit and a residual operation unit, mapping the image data in the pixel space to a low-dimensional latent space, and outputting standardized latent space features as the training input of the hierarchical denoising UNet backbone network.

[0242] The decoder subunit is used to receive the denoised latent space features output by the hierarchical denoising UNet backbone network during the material generation stage. It performs dimension restoration processing on the latent features through multi-level transposed convolutional upsampling units and residual operation units, mapping the low-dimensional latent space data back to the pixel space and outputting the final promotional material image.

[0243] In this embodiment, the latent space encoding / decoding module is built on a variational autoencoder architecture and undertakes the bidirectional mapping function between the pixel space and the low-dimensional latent space. It is the fundamental support module for realizing latent space diffusion generation. The latent space encoding / decoding module compresses high-resolution pixel images into a low-dimensional latent space through symmetrical multi-level convolutional encoding / decoding and residual structures. While significantly reducing the computational load of the diffusion network and improving the generation efficiency, it maximizes the preservation of product details, texture, and color information in the promotional materials, solving the technical pain points of high computational cost, low generation efficiency of high resolution, and insufficient detail reproduction in pixel space diffusion models.

[0244] This module comprises two parts: an encoder subunit and a decoder subunit. During the training phase, the encoder subunit encodes and compresses the original promotional material image, outputting standardized latent space features for training the hierarchical denoising UNet backbone network. During the inference and generation phase, the decoder subunit receives the denoised latent space features output by the UNet and decodes them back to pixel-level promotional material images. In this embodiment, the encoder input is a 256×256×3 RGB three-channel pixel image, and the output latent space feature size is 32×32×4 (spatial resolution 1 / 8 of the original image). The decoder input and output are completely symmetrical to the encoder.

[0245] Encoder subunit

[0246] The encoder subunit consists of an input preprocessing layer, a multi-level convolutional downsampling unit, a residual operation unit, and a mean / variance projection layer. It gradually compresses the spatial resolution and increases the number of feature channels through hierarchical convolutional downsampling, ultimately mapping the pixel image to the probability distribution parameters of the latent space.

[0247] (1) Input preprocessing layer

[0248] First, the input pixel image is numerically normalized, linearly mapping the original pixel values ​​in the range of 0~255 to the interval [-1,1], matching the input range of the activation function of the subsequent network. The calculation formula is as follows:

[0249] ;

[0250] in, The input is the original pixel image, with pixel values ​​ranging from [0, 255]. is the normalized input feature, with a value range of [-1, 1]; 255 is the maximum pixel value of an 8-bit color image.

[0251] (2) Multi-level convolutional downsampling unit and residual operation unit

[0252] The encoder consists of four cascaded downsampling units. Each unit integrates a residual operation structure, including two 3×3 convolutional layers, a group normalization layer, a GELU activation layer, and a downsampling convolutional layer with a stride of 2. Residual shorting connections are also implemented across units to avoid the gradient vanishing problem common in deep networks. The calculation formula for a single downsampling unit is:

[0253] ;

[0254] ;

[0255] in, This is the input feature map for the current downsampling unit;

[0256] , It consists of two 3×3 convolutional layers with a padding value of 1. The spatial size of the feature map remains unchanged before and after the operation, and it is responsible for feature extraction and nonlinear transformation.

[0257] GN stands for Group Normalization layer. It divides the channels into a fixed number of groups to perform normalization. Compared with batch normalization, it is more suitable for small batch training and the feature distribution is more stable. This represents the intermediate characteristics of the residual branch; The residual short-circuit feature is obtained by adjusting the number of channels through a 1×1 convolution of the input feature to ensure that the channel dimension is consistent with that of the intermediate feature, thus forming a short-circuit path for the residual operation unit. It is a 3×3 convolutional layer with a stride of 2, which performs downsampling operation. The spatial resolution of the output feature map is half that of the input. It is the core operation of the convolutional downsampling unit. This is the output feature map of the current downsampling unit, with the spatial resolution halved and the number of channels doubled.

[0258] The dimensional changes of the four-level downsampling units are as follows:

[0259] Level 1: Input 256×256×3, Output 128×128×64;

[0260] Second level: Input 128×128×64, output 64×64×128;

[0261] Level 3: Input 64×64×128, Output 32×32×256;

[0262] Level 4: Input 32×32×256, Output 32×32×128 (This level only performs residual feature enhancement, does not perform downsampling, and further integrates high-dimensional semantic information);

[0263] After four levels of unit processing, a 32×32×128 deep coding feature is obtained.

[0264] (3) Mean / variance projection layer and reparameterized sampling: The depth-encoded features are input into two parallel 1×1 convolutional layers respectively, and the two core parameters of the latent space distribution are mapped to obtain the following:

[0265] Mean matrix With the log-variance matrix Both are 32×32×4 in size, consistent with the final latent space feature dimensions. During training, a reparameterization technique is used to sample latent space features from the predicted distribution, ensuring gradient backpropagation. The sampling formula is:

[0266] ;

[0267] in, The standardized latent space features obtained from sampling, with a size of 32×32×4, are used as the training input for the hierarchical denoising UNet backbone network; The mean matrix of the latent space distribution is obtained by the encoder prediction; The standard deviation matrix of the latent space distribution is obtained from the log-variance: ; This is an element-wise multiplication operation; The standard Gaussian noise matrix is ​​a randomly sampled matrix that follows a standard normal distribution with a mean of 0 and a variance of 1. In the inference generation stage, the mean is used directly. As the final latent space feature, no random sampling is required, ensuring the stability and reproducibility of the generated result.

[0268] Decoder subunit

[0269] The decoder subunit is symmetrical to the encoder structure and consists of multi-level transposed convolution upsampling units, residual operation units, output convolutional layers, and activation layers. It gradually restores the spatial resolution through multi-level transposed convolution and maps low-dimensional latent space features back to pixel space.

[0270] (1) Multi-level transposed convolution upsampling unit and residual operation unit

[0271] The decoder has four cascaded upsampling units. Each unit also integrates a residual operation structure, including two 3×3 convolutional layers, a group normalization layer, a GELU activation layer, and a transposed convolutional layer with a stride of 2, and is configured with residual shorting connections.

[0272] The calculation formula for a single upsampling unit is:

[0273] ;

[0274] ;

[0275] in, This is the input feature map for the current upsampling unit; , Two 3×3 convolutional layers with padding of 1 ensure that the spatial dimensions remain unchanged before and after the operation, and are responsible for feature fusion and nonlinear transformation. GN is a group normalization layer that keeps the same as the encoder and stabilizes the feature distribution. The residual short-circuit feature is obtained by adjusting the number of channels through a 1×1 convolution of the input feature, which constitutes the short-circuit path of the residual operation unit; It is a transposed convolution with a stride of 2. It achieves feature map upsampling through deconvolution operation. The spatial resolution of the output feature map is twice that of the input. It is the core operation of the transposed convolution upsampling unit. This is the output feature map of the current upsampling unit, with the spatial resolution doubled and the number of channels halved.

[0276] The dimensional changes of the fourth-level upsampling unit are as follows:

[0277] Level 1: Input 32×32×4, output 32×32×128 (This level only performs residual feature dimensionality upsampling, and does not perform upsampling).

[0278] Second level: Input 32×32×128, output 64×64×128;

[0279] Level 3: Input 64×64×128, output 128×128×64;

[0280] Level 4: Input 128×128×64, Output 256×256×32;

[0281] After four levels of unit processing, a high-resolution decoding feature of 256×256×32 is obtained.

[0282] (2) Output layer and pixel restoration

[0283] The high-resolution decoded features are input into a 1×1 output convolutional layer, compressing the number of channels to 3 (corresponding to RGB three channels). Then, the values ​​are normalized to the [-1, 1] interval using the Tanh activation function, yielding the decoded normalized pixel features. The calculation formula is as follows:

[0284] ;

[0285] in, This represents the high-resolution features output by the last-stage upsampling unit; It is a 1×1 convolutional layer used for channel dimension compression, outputting 3-channel features; The hyperbolic tangent activation function constrains the output value to the interval [-1, 1]. The decoded output is a normalized pixel image with dimensions of 256×256×3. Finally, the normalized pixel values ​​are linearly mapped back to the standard image pixel range of 0~255 to obtain a promotional material image that can be directly saved and used. The inverse operation formula is:

[0286] ;

[0287] in, The final output is a pixel-level promotional image, with pixel values ​​ranging from [0, 255]. This is the normalized feature output by the decoder, with a value range of [-1, 1].

[0288] The latent space encoding and decoding module is pre-trained using a large-scale general image dataset. After training converges, all weights of the encoder and decoder are frozen and used only as fixed feature mapping tools in the training and inference of the diffusion model, thus avoiding the destruction of the reconstruction quality of the encoding and decoding by diffusion training.

[0289] During the pre-training phase, a joint loss function consisting of pixel reconstruction loss and KL divergence regularization loss is used for optimization. The total loss formula is as follows:

[0290] ;

[0291] in, This represents the total training loss of the encoding / decoding module; For pixel reconstruction loss, The KL divergence (Kullback-Leibler divergence) regularization loss is used to constrain the latent space distribution of the encoder output to approach the standard normal distribution, avoid latent space collapse, and ensure the sampling effectiveness of the subsequent diffusion model. The balancing weight hyperparameter for KL divergence is set to 0.001 in this embodiment, which is used to balance reconstruction accuracy and latent space regularization strength.

[0292] The mean square error between the decoded output image and the original input image is calculated to constrain the image restoration accuracy. The formula is as follows:

[0293] ;

[0294] Where H is the height of the input / output image in pixels, and in this embodiment, it is 256; W is the width of the input / output image in pixels, and in this embodiment, it is 256; C is the number of channels in the input / output image, and in this embodiment, it is RGB three channels, with a value of 3; i is the pixel index in the height direction of the image, and its value ranges from... ;j is the pixel index along the width direction of the image, and its value ranges from 1 to 1. ; k is the index of the image channel direction, and its value range is ; The spatial location of the original input image The pixel value of the k-th channel; The spatial location of the decoder output image The pixel value of the k-th channel; The total number of pixels in the image is used to normalize the loss and eliminate the influence of image size on the loss value.

[0295] (2) KL divergence regularization loss

[0296] The KL divergence regularization loss is calculated per spatial location to determine the difference between the latent space distribution predicted by the encoder and the standard normal distribution, and then the global average is taken. The formula is as follows:

[0297] ;

[0298] in, The height of the latent space feature map is 32 in this embodiment;

[0299] The width of the latent space feature map is 32 in this embodiment;

[0300] The number of channels in the latent space feature map is 4 in this embodiment;

[0301] i is the spatial index of the latent space in the height direction, and its value ranges from 1 to 2. ;

[0302] j is the spatial index along the width direction of the latent space, and its value ranges from 1 to 2. ;

[0303] Spatial location in latent space predicted by the encoder The distribution mean vector at point , with dimension . dimension;

[0304] Spatial location in latent space predicted by the encoder The variance vector at point has dimension . dimension;

[0305] is the natural logarithm of the variance, corresponding to the log-variance matrix of the encoder output;

[0306] This is the element-wise squaring operation for the mean vector;

[0307] The KL divergence term between the normal distribution and the standard normal distribution at a single spatial location is given.

[0308] This is a normalization coefficient used to perform a global average of the loss, eliminating the influence of the latent space size on the loss value.

[0309] The latent space encoding and decoding module achieves compression and high-precision restoration with 8 times spatial resolution through a four-level symmetric residual encoding and decoding structure and variational distribution constraints. It reduces the computational load of the diffusion network to 1 / 64 of that in pixel space, significantly improving the generation efficiency of high-resolution promotional materials. At the same time, it retains multi-scale detail information through the residual structure, and the restoration of the product's main body edges and material textures is improved compared to ordinary downsampling schemes, enabling accurate restoration of the detailed texture and color performance of promotional materials.

[0310] In this embodiment, the promotional material generation model guided by the joint embedding of text and images is obtained through iterative training using a multi-objective joint loss function. The total loss function during the training process is as follows:

[0311] ;

[0312] in, For the total loss, For layered noise prediction loss, For hierarchical cross-modal alignment loss, Due to layout constraints, For conditional orthogonal constraint loss, , , These are preset balancing weight hyperparameters;

[0313] The hierarchical noise prediction loss is:

[0314] ;

[0315] in, , , The mean squared error of noise prediction for the semantic control layer, style layout layer, and detail control layer are respectively. , , Preset weights for the corresponding levels;

[0316] The hierarchical cross-modal alignment loss is:

[0317] ;

[0318] in, , , These are the global average pooling vectors representing the output features of the semantic control layer, style layout layer, and detail control layer, respectively. , , These are global semantic embedding, layout style embedding, and element detail embedding, respectively. The cosine similarity function;

[0319] The layout constraint loss is:

[0320] ;

[0321] in, To generate salient subject masks corresponding to latent features, For layout prior mask, The function for calculating the intersection-union ratio;

[0322] The loss due to the orthogonal constraint is:

[0323] ;

[0324] in, The inner product of the global semantic embedding vector and the layout style embedding vector;

[0325] The inner product of the global semantic embedding vector and the element detail embedding vector;

[0326] The inner product of the layout style embedding vector and the element details embedding vector.

[0327] This application employs a multi-objective joint loss function, a dedicated constraint system for promotional material generation scenarios. This loss system simultaneously constrains the training process across four dimensions: noise prediction accuracy, condition matching degree, layout compliance, and subspace decoupling degree. Compared to the single global mean squared error loss of traditional diffusion models, it can simultaneously improve the overall performance of the generation model from multiple dimensions. Specific technical advantages are as follows:

[0328] First, the hierarchical noise prediction loss implements differentiated constraints on the generation function. This loss sets differentiated weights according to the control division of three functional levels, allocating loss weights based on the generation logic of semantic priority, layout second, and details secondary, breaking the constraint of the traditional globally uniform weight. This design strengthens the functional division of the hierarchical network at the loss level, prioritizing the accuracy of global semantics and core selling points generation, avoiding interference from high-resolution detail optimization processes with deep semantic direction, and effectively reducing the probability of generated content deviating from the promotional theme and selling points being misaligned. Simultaneously, the hierarchical constraint allows network units at different resolution levels to learn purer corresponding features, resulting in a significantly faster training convergence speed compared to the globally uniform loss, and eliminating redundant coupling of feature representations at each level.

[0329] This loss function establishes a one-to-one binding alignment relationship between network layers and conditional granularities, replacing the global conditional alignment scheme of existing technologies. Each layer only performs semantic matching with the conditional embeddings of its corresponding granularity. This design cuts off the mutual interference paths of conditional signals of different granularities at the loss constraint level, ensuring that semantic conditions only affect the global theme, style conditions only affect the visual tone, and detail conditions only affect pixel texture, truly achieving independent controllability of different generation dimensions. In practical applications, when users adjust the generation effect of a certain dimension alone, it will not cause a cascading shift in other dimensions, significantly improving the accuracy and flexibility of generation control.

[0330] This loss mechanism transforms the unique layout rules of promotional materials into trainable numerical constraints, ensuring that layout specifications are involved in the entire training process of the diffusion model. This differs from existing technologies that rely solely on post-processing cropping and layout adjustments. The trained network can actively learn and adhere to layout specifications during denoising iterations. The generated product subject and text areas naturally fit the preset layout, avoiding issues such as subject truncation, harsh edges, and content distortion caused by post-processing cropping. While ensuring the layout conforms to the distribution channel's specifications, it fully preserves the natural texture and detail integrity of the generated image.

[0331] This loss function enforces pairwise linear independence between the feature spaces of the three types of conditional embeddings through orthogonal regularization, fundamentally addressing the industry-wide problem of mutual coupling between conditions of different granularities and the tendency for adjusting one condition to lead to multiple shifts. After training convergence, the inner product of the vectors in the three conditional subspaces approaches zero, and the semantic, style, and detail representations are completely independent. This further improves the control accuracy of hierarchical injection and provides a clean feature foundation for subsequent extended applications such as single-dimensional conditional fine-tuning and zero-shot style transfer. At the same time, orthogonal constraints can also compress redundant feature dimensions, improve the representation efficiency of conditional embeddings, and allow the model to carry richer conditional information with lower feature dimensions.

[0332] The image-text joint embedding-guided promotional material generation model employs a two-stage progressive training scheme. The first stage involves independently pre-training the latent space encoding / decoding module, freezing all parameters after training convergence as a fixed feature mapping tool. The second stage, based on the frozen encoding / decoding module, jointly trains a multi-granularity cross-modal conditional decoupling and hierarchical injection module, a layout prior encoding and injection module, and a hierarchical denoising UNet backbone network. The two training stages have independent objectives and completely isolated parameters, ensuring both the image restoration accuracy of the encoding / decoding module and end-to-end adaptation between the diffusion-generated backbone and the dedicated constraint system.

[0333] I. First Stage: Pre-training of Latent Space Encoding / Decoding Module

[0334] The training dataset uses a publicly available large-scale general image dataset superimposed with a specific subset of e-commerce promotional materials as training data, with a total sample size of no less than 1 million images, covering various image scenarios such as daily items, industrial products, and marketing posters. All samples are uniformly preprocessed into 256×256 resolution RGB three-channel images, and data augmentation strategies such as random flipping, random cropping, and slight color perturbation are introduced during training to improve the model's generalization ability.

[0335] The training objective and optimization method use the joint loss of pixel reconstruction loss and KL divergence regularization loss as the optimization objective. The encoder is trained to learn the compressed mapping from pixel space to low-dimensional latent space, and the decoder is trained to learn the restored mapping from latent space to pixel space. This maximizes the preservation of image texture, color, and edge details while reducing computational cost. The Adam optimizer is used for end-to-end backpropagation optimization, with an initial learning rate set to 4.5 × 10⁻⁻⁶. 5 The weight decay factor is set to 1×10⁻ 6 The training batch size was set to 32; the learning rate was dynamically adjusted using a cosine annealing strategy; and the total number of training rounds was set to 100.

[0336] Convergence criterion and parameter freezing: When the pixel reconstruction loss of the validation set decreases by less than 1×10⁻ for 10 consecutive rounds. 6 When the model converges, it is determined that the model has converged. After training is completed, all network parameters of the encoder and decoder are permanently frozen. During the subsequent second-stage training and inference generation, the weights of this module remain fixed and no longer participate in gradient updates, thus avoiding diffusion training that could damage the image reconstruction quality.

[0337] II. Second Phase: Diffusion Generation Backbone Joint Training

[0338] The training dataset is constructed by creating a dedicated training set for promotional materials in the vertical domain. The effective training sample size for each product category is no less than 50,000. Each sample contains three types of annotation information: first, the original high-resolution image of the promotional material, which serves as the ground truth label for generation; second, the multimodal reference input of the corresponding promotional material (including product reference image, selling point description text, and style requirement label), which is used to generate the target multimodal joint conditional embedding; and third, the layout area annotation (annotating the boundaries of the product display area, title copy area, and white space area), which is used to generate the layout prior mask.

[0339] Data preprocessing workflow: Standardized preprocessing is performed on each sample before training:

[0340] Image side: The original promotional material image is input into the frozen encoder subunit and mapped to obtain the corresponding real latent space features; according to the standard training paradigm of the diffusion model, time steps in the range of 0 to 1000 are randomly sampled, and Gaussian noise of corresponding intensity is added to the real latent space features to generate noisy latent space features and corresponding real noise labels, which serve as training input and supervision labels for the diffusion network.

[0341] Conditional side: Features are extracted and fused from the multimodal reference input of the sample to generate the corresponding target multimodal joint conditional embedding vector, which serves as the input to the multi-granularity cross-modal conditional decoupling and hierarchical injection module.

[0342] Layout side: Generate a binary layout prior mask based on the layout area annotation of the sample, which serves as the input for the layout prior encoding and injection module.

[0343] The forward propagation training process simultaneously inputs the noisy latent space features, the current time step encoding, the target multimodal joint conditional embedding vector, and the format prior mask into the diffusion-generated backbone to be trained, and then performs forward propagation operations:

[0344] ① The multi-granularity cross-modal conditional decoupling and hierarchical injection module performs three-way orthogonal decoupling on the joint conditional embedding, and generates three levels of dynamic conditional enhancement features by combining the current time step encoding, which are injected into the three control levels of the hierarchical denoising UNet backbone network respectively;

[0345] ② The layout prior encoding and injection module performs multi-scale feature extraction, spatial attention enhancement and resolution alignment on the layout prior mask, generates layout constraint features, and injects them into the style layout layer of the hierarchical denoising UNet backbone network.

[0346] ③ The hierarchical denoising UNet backbone network is based on the noisy latent space features. Under the joint guidance of time step encoding, three-level dynamic conditional enhancement features, and layout constraint features, it performs multi-level diffusion denoising operations and outputs predicted noise.

[0347] ④ Based on the predicted noise and the actual noise labels, combined with the hierarchical cross-modal alignment loss, layout constraint loss, and conditional orthogonality constraint loss, the total loss value of this round of training is calculated by weighting according to the preset weights.

[0348] Backpropagation and parameter updates propagate the total loss back along the network, updating only the learnable parameters of the multi-granularity cross-modal conditional decoupling and hierarchical injection modules, the format prior encoding and injection module, and the hierarchical denoising UNet backbone network. The parameters of the latent space encoding / decoding module remain frozen throughout the process and do not participate in gradient updates. The AdamW optimizer is used for parameter optimization, with an initial learning rate set to 1×10⁻. 4 The weight decay factor is set to 1×10⁻ 5The training batch size was set to 16; the learning rate adopted a linear warm-up + cosine decay strategy, with the first 5% of the training steps completing the linear warm-up of the learning rate, and then gradually decaying to 1% of the initial value.

[0349] Convergence is determined by calculating the total loss and individual component losses on the independent validation set after each training epoch. A convergence criterion is reached when the decrease in the total loss on the validation set is less than 1 × 10⁻⁻⁴ over 10 consecutive training epochs. 5 When both semantic alignment loss and layout matching loss reach the preset accuracy threshold, the model training is considered to have converged, and the trained image-text joint embedding guided promotional material generation model is finally obtained.

[0350] In this embodiment, the promotional materials input by the user contain three types of core data, each serving a different reference function:

[0351] Product reference image: User-uploaded white background image of the target product, real shot image, or existing promotional image, used to extract the product's visual features, physical attributes, and selling points, serving as the factual basis for content generation; supports three common image formats: JPG, PNG, and WEBP.

[0352] Layout reference material: A reference image used to extract layout rules. It can be the same image as the main product reference image, or it can be uploaded separately by the user. The regional division of this material will serve as the basis for the layout constraints of the generated material, ensuring that the output material conforms to the layout specifications of the target distribution channel.

[0353] Supplementary text description: User-inputted text information such as promotional theme, style, and supplementary explanations of core selling points is used to supplement semantic conditions not covered by visual materials, further accurately targeting the generation direction; plain text input is supported, with a maximum length of 500 characters per input.

[0354] Standardized preprocessing of image materials

[0355] A unified, standardized preprocessing workflow is performed on both the main product reference image and the layout reference image. The specific steps are as follows:

[0356] Format and channel unification: Input images of different formats are uniformly converted into RGB three-channel format; for PNG images with alpha transparency channels, transparent areas are filled with pure white background to eliminate the interference of channel differences on subsequent feature extraction and layout segmentation.

[0357] Resolution normalization: Bicubic interpolation algorithm is used to perform size scaling, and all images are uniformly adjusted to the standard resolution of 224×224 pixels; for images whose aspect ratio is inconsistent with the standard size, proportional scaling and edge grayscale filling are used to maintain the original proportion of the main body of the product and avoid stretching and deformation of the main body; the filling area is filled with the grayscale value of the average pixels of the whole image to reduce the impact of invalid areas on feature extraction.

[0358] Pixel value normalization: The original pixel values ​​in the range of 0~255 are linearly mapped to the interval of [-1,1], which matches the input value range of the subsequent feature extraction network and layout segmentation network, ensuring the accuracy of feature extraction.

[0359] Image quality optimization: A 3×3 Gaussian kernel is used to perform Gaussian filtering on the image to remove random noise introduced during image acquisition and compression; at the same time, adaptive contrast equalization is performed to correct overexposed and underexposed input images, enhance the edge and texture features of the image, and improve the accuracy of subsequent feature extraction and layout area segmentation.

[0360] Standardized preprocessing of text-based materials

[0361] For the auxiliary text description input by the user, normalization processing is performed. The specific steps are as follows:

[0362] Perform a full clean of the original text, removing special characters, redundant spaces and meaningless stop words, and unifying the full-width / half-width punctuation format;

[0363] Semantic completion is performed on abbreviated keyword input, expanding fragmented keywords into complete semantic sentences to ensure the semantic integrity of subsequent text encoding;

[0364] The unified text encoding format is UTF-8 to avoid semantic recognition errors caused by incompatible encoding formats.

[0365] In this embodiment, step 3 includes:

[0366] Step 31: Construct a four-level hierarchical multi-layer causal semantic ontology structure. The multi-layer causal semantic ontology structure includes, from bottom to top, a pixel parameter layer, an entity layer, a selling point layer, and a value layer. Only one-way causal dependency edges are set between nodes of adjacent levels from bottom to top. No association edges are set between nodes of the reverse level and non-adjacent levels. This blocks the transmission path of non-causal mixed information from the ontology structure level.

[0367] Step 31 is the structured semantic skeleton of the entire causal decoupling system. Its core function is to define the legitimate transmission path of information at the ontology architecture level, cutting off the transmission path of mixed scene information to the selling point layer from the root. This ontology structure is pre-built based on the hierarchical cognitive laws in the field of product marketing, and strictly follows the unidirectional causal logic that the underlying pixel features support the existence of entities, the entity features support the establishment of selling points, and the selling point features support the value transmission.

[0368] Definition and node composition of the four-level hierarchy:

[0369] The ontology structure, from bottom to top, consists of the pixel parameter layer, entity layer, selling point layer, and value layer. The positioning, node composition, and attributes of each layer are strictly distinguished, as detailed below:

[0370] The pixel parameter layer is the bottom layer of the entire ontology structure. It belongs to the dynamic feature access layer and has no predefined fixed semantic nodes. It only serves as a container for all input multimodal bottom-level features and corresponds to all leaf nodes.

[0371] Node sources: dynamically generated based on user input materials, including three categories: patch-level visual feature nodes extracted from product reference images, semantic token feature nodes extracted from auxiliary text, and structured attribute feature nodes extracted from product parameters;

[0372] Node attributes: Each node carries the original feature vector and corresponding label information (such as the spatial location of the visual patch, the semantic label of the text token, and the attribute name of the parameter).

[0373] Number of nodes: The number of nodes changes dynamically with the amount of information in the input material. The number of nodes corresponding to a single input material is usually 200 to 300. All node feature dimensions are uniformly calibrated to 768 dimensions.

[0374] The entity layer is the intermediate layer that supports the lower layer. It is the objective and tangible carrier layer of the product and a key transitional layer that connects the underlying perceptual features with the upper semantic selling points. Its function is to filter out mixed features that lack physical support.

[0375] Node types: There are three types of fixed semantic nodes: component entity nodes (such as physical components of the product, such as the casing, battery, screen, processor, etc.), material entity nodes (such as material entities such as aluminum alloy, glass, silicone, etc.), and functional unit nodes (such as functional collections such as charging module, camera module, heat dissipation module, etc.).

[0376] Node size: A fixed set of nodes is predefined for the target product category. Each product category has 20 to 30 entity nodes, and each node corresponds to a unique standard term tag.

[0377] Initial feature generation: Using the standard term label of each node as input, semantic encoding is performed using the BERT-base pre-trained model which is the same as the text encoding. The output vector corresponding to the mask token is taken as the initial semantic feature vector of the node. The dimension is uniformly 768, and it is jointly fine-tuned with the overall network during the training phase.

[0378] The selling point layer is the core semantic layer, which is the core control unit for generating promotional content and corresponds to the core value points that the product can promote.

[0379] Node types: There are three types of fixed semantic nodes: physical attribute selling points (such as lightweight, high-definition display, narrow bezel and other objective attribute selling points), functional feature selling points (such as fast charging, long battery life, high frame rate and other functional selling points), and perceptual experience selling points (such as comfortable grip, smooth operation, premium look and feel and other experiential selling points).

[0380] Node scale: 15-20 selling point nodes are preset for each product category, and each node corresponds to a unified standardized selling point terminology to eliminate differences in colloquial expressions;

[0381] Initial feature generation: Consistent with the entity layer node generation method, a 768-dimensional initial feature vector is obtained through semantic encoding of standard terms, and joint fine-tuning is performed during the training phase.

[0382] The value layer is the highest layer of the entire ontology structure. It corresponds to the ultimate user value that the product conveys to the user and is the emotional and benefit focus of the promotion.

[0383] Node types: Includes four core value nodes: practical value, emotional value, social value, and identity value;

[0384] Node size: Each product category has a fixed number of 3 to 5 nodes, covering the core user needs.

[0385] Initial feature generation: Similarly, a 768-dimensional initial feature vector is obtained through semantic encoding, and joint fine-tuning is performed during the training phase.

[0386] The rules for constructing unidirectional causal dependency edges are the core of the ontology structure for implementing causal constraints. They strictly limit the existence of bottom-up directed causal dependency edges only between adjacent levels. No related edges are set between nodes in reverse levels, non-adjacent levels, or within the same level. The specific rules are as follows:

[0387] Valid edge connection rules:

[0388] Directed edges are established only between adjacent nodes at different levels, with the direction of the edges strictly from bottom to top. There are three types of valid edges:

[0389] Pixel parameter layer → entity layer: The bottom feature node points to its corresponding entity node, indicating that the pixel / parameter feature is the supporting basis for the existence of the corresponding entity;

[0390] Entity layer → Selling point layer: Entity nodes point to the selling point nodes they support, representing that the entity attribute is the physical reason for the corresponding selling point to be valid;

[0391] Selling point layer → Value layer: The selling point node points to the value node it conveys, indicating that the selling point can bring corresponding value to the user.

[0392] The initial weight of the connection is calculated by the cosine similarity of the initial feature vectors of the upper and lower nodes, and the value range is [0,1], which is used to characterize the initial strength of the causal relationship between the two.

[0393] Edge-blocking rules:

[0394] In the following three cases, no related edges will be established to completely block the transmission path of non-causal information at the structural level:

[0395] Reverse edges are prohibited: Upper-level nodes are not allowed to point to lower-level nodes, thus eliminating the reverse causal logic of selling points determining entities and value determining selling points;

[0396] Cross-layer connections are prohibited: The pixel parameter layer is not allowed to directly connect to the selling point layer or the value layer, nor is the entity layer allowed to directly connect to the value layer. All information must be passed up level by level. This rule can force the mixed features of the underlying scene (such as shooting background and ambient light and shadow) to be filtered through the entity layer first. Since the background features do not belong to any product entity node, they cannot be passed up through the entity layer, thereby preventing scene features from being misidentified as product selling points from the root.

[0397] Prohibit horizontal connections between nodes at the same level: Do not establish horizontal connections between nodes at the same level to avoid feature mixing and mutual interference between nodes at the same level.

[0398] Causal indicator matrix generation:

[0399] Based on the above connection rules, a causal indicator matrix matching the node size is generated. As a path mask for subsequent graph attention operations, it forces information to be transmitted only along legitimate causal paths at the computational level.

[0400] Matrix Dimension: The dimension is N×N, where N is the total number of all nodes in the ontology structure (number of dynamic nodes in the pixel parameter layer + number of nodes in the entity layer + number of nodes in the selling point layer + number of nodes in the value layer).

[0401] Element selection rules: For any element in the matrix If and only if there exists a point from the lower node j to the upper node. When there is a positive causal dependency edge, In all other cases (reverse edge, cross-layer edge, no edge, same-layer edge), the value is 0.

[0402] In subsequent graph attention feature aggregation, this matrix is ​​directly used as a mask factor for attention scores. Attention scores for non-causal paths are directly set to zero and do not participate in feature aggregation calculations, thus completely blocking the upward transmission of non-causal mixed information at the computational level.

[0403] Step 32: Extract the multimodal low-level features corresponding to the user-input promotional materials, attach all multimodal low-level features as leaf nodes to the pixel parameter layer, and construct a cross-modal causal heterogeneous graph according to the connection rules of the unidirectional causal dependency edges;

[0404] Step 32 converts the unstructured promotional materials input by the user into standardized feature nodes that can be accessed by causal ontology. Semantic matching is used to accurately mount the underlying features at the pixel parameter layer, and a complete cross-modal causal heterogeneous graph is constructed strictly following unidirectional causal dependency rules. This provides structured graph data input for subsequent causal perception feature aggregation. This step relies entirely on a pre-trained feature extraction network for inference operations, eliminating the need for real-time training. Single-sample processing latency is controlled within milliseconds, meeting the real-time requirements of online generation.

[0405] Multimodal low-level feature extraction from user input materials

[0406] For the two types of core materials input by the user (product reference images and auxiliary text descriptions), feature extraction is performed using pre-trained encoders in the corresponding domains to generate low-level feature vectors with semantic representation capabilities. The specific extraction process is as follows:

[0407] (1) Visual modality low-level feature extraction

[0408] Using the standardized product reference image (224×224×3, pixel values ​​normalized to [-1,1]) output from step 2 as input, multi-granularity visual feature extraction is performed using the CLIP-ViT-L / 14 visual encoder. This encoder is pre-trained on a large-scale image-text alignment dataset and has excellent cross-modal semantic representation capabilities.

[0409] Patch-level region feature extraction: The encoder divides the input image into 196 image patches evenly according to the 14×14 block rule. Convolution and self-attention operations are performed independently on each patch, and 196 768-dimensional region feature vectors are output. Each vector corresponds to the local visual semantics of a fixed spatial location in the image, which can accurately represent fine-grained visual information such as local parts, textures, and materials of the product.

[0410] Global image feature extraction: Perform global average pooling operation on all patch-level region features to obtain a 768-dimensional global image feature vector corresponding to a single image, which is used to characterize the overall visual attributes and appearance tone of the product;

[0411] Component label association: Combining the product component detection results (including component category labels and normalized bounding box coordinates) output by the pre-trained YOLOv8 object detection model, a mapping relationship between the features of each patch region and the corresponding component category is established, providing semantic anchors for subsequent entity mounting.

[0412] (2) Extraction of low-level features of text modalities

[0413] Taking the normalized auxiliary text description output from step 2 as input, semantic encoding is performed using the RoBERTa-wwm-ext-large Chinese pre-trained language model. This model has been fine-tuned in the field of e-commerce marketing and has higher semantic representation accuracy for texts describing product selling points and attributes.

[0414] Token-level fine-grained feature extraction: After performing word segmentation on the input text, the output layer feature vector corresponding to each token is extracted. The feature dimension of a single token is 1024-dimensional, which is used to represent the fine-grained semantics of individual entities and attribute words in the text.

[0415] Sentence-level global feature extraction: The output vector corresponding to the first mask token of the text is taken as the sentence-level global semantic feature with a dimension of 1024, which is used to represent the overall promotional theme and style of the entire text.

[0416] Entity phrase annotation: Combining a pre-trained named entity recognition model, token fragments corresponding to product entities and attribute phrases are extracted from the text, and their semantic categories are annotated to provide a matching basis for subsequent entity attachment.

[0417] Unified cross-modal feature dimensions:

[0418] Since the feature dimensions output by different modal encoders differ (visual features are 768-dimensional and text features are 1024-dimensional), all low-level features need to be uniformly mapped to 768 dimensions through a linear projection layer to maintain consistency with the feature dimensions of nodes at each level in the ontology structure, thus meeting the dimensionality compatibility requirements of subsequent graph attention operations.

[0419] The formula for calculating dimension mapping is:

[0420] ;

[0421] in, The low-level feature vector of the original input; The learnable linear projection matrix has a dimension of 768×1024 for text features and 768×768 for visual features (visual features are only calibrated in the feature space and the dimension is not changed). This is the bias vector, with a dimension of 768. This is a standardized feature vector with unified dimensions, fixed at 768 dimensions.

[0422] The parameters of the linear projection layer are jointly optimized with the overall causal network during the model training phase and remain fixed during the inference phase. After mapping, all visual and textual low-level features are unified to 768 dimensions and serve as candidate leaf nodes for the pixel parameter layer.

[0423] Mounting and matching of low-level feature nodes to pixel parameter layers

[0424] After unifying all dimensions, all bottom-level feature nodes are assigned to the pixel parameter layer, serving as leaf nodes of the ontology structure. Using a semantic similarity matching algorithm, each bottom-level feature node is attached to the entity layer node with the closest semantic similarity, strictly adhering to the unidirectional causal connection rule from the pixel parameter layer to the entity layer. The specific matching process is as follows:

[0425] Matching feature preparation: For each low-level feature node, extract its corresponding semantic label vector:

[0426] Visual patch features: The associated part category label is extracted and encoded into a 768-dimensional label semantic vector by the BERT-base encoder;

[0427] Text token features: The corresponding entity / attribute phrase label is extracted and encoded into a 768-dimensional label semantic vector through a homologous BERT-base encoder.

[0428] Similarity calculation: The cosine similarity between the label semantic vector of each low-level feature and the initial semantic vector of all nodes in the entity layer is calculated one by one.

[0429] Mounting determination and rules:

[0430] The entity layer node with the highest similarity and a value greater than or equal to a preset threshold of 0.7 is selected as the upper-level associated node of the underlying feature node and attached to the lower layer of the corresponding entity node. A one-way causal dependency edge is established between the pixel parameter layer and the entity layer.

[0431] If the similarity between the feature and all entity layer nodes is less than 0.7, then it is uniformly attached to the general attribute node of the entity layer to ensure that all underlying features are included in the causal transmission link and avoid loss of feature information.

[0432] All mounting operations are strictly limited to pixel parameter layer nodes pointing to entity layer nodes. Reverse connections are not allowed, nor are direct connections across layers to the selling point layer or value layer allowed. This fully complies with the unidirectional causal dependency rule.

[0433] Structured construction of cross-modal causal heterogeneous graphs

[0434] After mounting all the underlying feature nodes, and combining them with the predefined ontology hierarchy, a complete cross-modal causal heterogeneous graph is finally constructed, outputting structured graph data, which specifically includes the following components:

[0435] The node set consists of four parts: dynamically generated pixel parameter layer nodes, predefined entity layer nodes, predefined selling point layer nodes, and predefined value layer nodes. All nodes carry a 768-dimensional feature vector, as well as corresponding semantic labels, hierarchical attributes, and other meta-information.

[0436] Directed Adjacency Matrix: Generating Binary Directed Adjacency Matrices Based on One-Way Causal Dependency Rules The matrix has dimensions N×N, where N is the total number of nodes; matrix elements The rule for determining the value of is: if and only if there exists a valid one-way causal edge pointing from the lower node j to the upper node i, In all other cases (reverse edge, cross-layer edge, same-layer edge, no edge), the value is 0. Valid edges include only three types: pixel parameter layer to entity layer, entity layer to selling point layer, and selling point layer to value layer.

[0437] Edge weight initialization: For each legal directed edge, the cosine similarity between the feature vectors of the two nodes above and below is used as the initial edge weight, with a value range of [0,1], representing the initial strength of the causal relationship between the two nodes; the edge weight will be dynamically updated during the subsequent graph attention aggregation process.

[0438] Causal indicator matrix update: Incorporate the newly added pixel parameter layer dynamic nodes to update the causal indicator matrix generated in step 3.1. This ensures that the matrix dimension matches the total number of nodes, and sets all positions corresponding to non-causal paths to zero, serving as a path mask for subsequent feature aggregation.

[0439] The final cross-modal causal heterogeneous graph fully carries the multimodal information of the input material and the predefined causal transmission rules. All information can only be transmitted step by step along the unidirectional causal path from bottom to top, blocking the upward transmission of mixed information in the scene from both structural and computational levels.

[0440] Step 33: Use a multi-layer causal perceptual graph attention network to perform message passing and feature aggregation along the causal forward direction on the cross-modal causal heterogeneous graph, and block the reverse and cross-layer information passing through the causal indicator factor mask to obtain the preliminary aggregated embedding vector corresponding to each selling point semantic node;

[0441] Step 33 uses a multi-layer causal perception graph attention network to strictly perform message passing and feature aggregation along the bottom-up positive causal path. By using the causal indicator factor matrix as a path mask, it completely blocks the transmission of non-causal information in the reverse and cross-layer modes at the computational level, so that the selling point layer nodes only aggregate the effective information of their subordinate entities and pixel features, and finally obtain the preliminary aggregated embedding vector corresponding to each selling point semantic node.

[0442] This step employs a two-layer stacked causal perceptual graph attention network. The two layers have the same structure but independent parameters, corresponding to a two-level progressive aggregation of pixel parameter layer → entity layer and entity layer → selling point layer, respectively, to ensure that information is strictly transmitted level by level without any jumps across layers.

[0443] The computational flow of attention in a single-layer causal perceptual graph:

[0444] The attention operation of a single-layer causal perception graph strictly follows the principle of prioritizing positive causal paths and setting the mask of non-causal paths to zero. It consists of four core steps: linear transformation of node features, calculation of unnormalized attention scores, normalization of attention weights, and weighted aggregation of features, as detailed below:

[0445] (1) Linear transformation of nodal features

[0446] First, a shared linear space mapping is performed on all the input node features to map the original features to a high-dimensional feature space, thereby improving the nonlinear expressive power of the network.

[0447] Attention score calculation under causal mask constraints

[0448] For any target node Iterate through all lower-level neighbor nodes. The unnormalized attention scores for both are calculated. A causal indicator factor is introduced as a mask during the calculation, directly setting the scores of non-causal paths to zero to ensure that only positive causal edges participate in the attention calculation. The calculation formula is as follows:

[0449] ;

[0450] in, For the target node The eigenvectors after linear transformation; For lower-level neighbor nodes The eigenvectors after linear transformation; This is a vector concatenation operation;

[0451] This is a learnable attention vector used to map concatenated features to a single-valued raw attention score.

[0452] It is a linear rectified activation function with leakage, and the negative slope is preset to 0.2 to introduce nonlinearity;

[0453] These are causal indicator factors, corresponding to the target nodes in the causal indicator factor matrix. with neighboring nodes Element value: when there exists a bottom-up pointer When the positive causal edge is, =1, this path participates in attention calculation; when there are no positive causal edges (reverse edges, cross-layer edges, no edges), =0, the score for this path is set to zero and will not be included in subsequent calculations;

[0454] For nodes Relative to the target node The unnormalized attention score.

[0455] For the target node The attention scores of all valid lower-level neighbors are normalized using softmax to obtain the final attention weights, ensuring that the sum of the weights of all valid neighbors is 1.

[0456] The features of all lower-level neighbor nodes are weighted and summed according to attention weights, and then the updated features of the target node are obtained after non-linear activation. The calculation formula is as follows:

[0457] ;

[0458] in, This is a weighted aggregation result of the features of the lower-level neighbors;

[0459] ELU (Exponential Linear Unit) is a non-linear activation function used to introduce non-linear expressive power and prevent the network from degenerating into a purely linear transformation. For the target node The output feature vector after single-layer aggregation.

[0460] Progressive aggregation of two-layer networks:

[0461] A two-layer graph attention network is stacked to perform progressive aggregation, strictly matching the hierarchical transmission order of the ontology structure to ensure that information is only passed upwards level by level without skipping across layers. The specific division of labor is as follows:

[0462] First layer aggregation: Pixel parameter layer → Solid layer

[0463] The above single-layer causal attention aggregation operation is performed using the initial features of all nodes in the output cross-modal causal heterogeneous graph as input.

[0464] Aggregation objective: All nodes in the entity layer receive feature information from their subordinate pixel parameter layer nodes, completing the aggregation of underlying multimodal features into entity semantics;

[0465] Operational constraints: Only positive causal edges from the pixel parameter layer to the entity layer are valid; all other edges are set to zero by the causal indicator factor mask.

[0466] Output: The features of all nodes in the entity layer are updated, incorporating information from all their subordinate visual and textual underlying features; the nodes in the selling point layer and value layer only receive preliminary input from the lower-level entity nodes and do not complete the final aggregation.

[0467] Second layer aggregation: Entity layer → Selling point layer

[0468] Using the full node update features from the first layer aggregation output as input, the same causal attention aggregation operation is repeatedly performed.

[0469] Aggregation goal: All nodes in the selling point layer receive feature information from their subordinate entity layer nodes, completing the convergence of entity semantics to selling point semantics;

[0470] Operational constraints: Only positive causal edges between "entity layer → selling point layer" are valid; all reverse and cross-layer paths are blocked by a mask.

[0471] Output: The features of all nodes in the selling point layer are finally updated. Each selling point node fully integrates the information of all its subordinate entities and all modal underlying features, forming a preliminary selling point aggregation representation.

[0472] The two-layer network uses its own independent linear transformation matrix and attention vector parameters, and is jointly optimized during the training phase to adapt to the feature aggregation requirements of different levels.

[0473] After completing two layers of progressive aggregation operations, feature vectors corresponding to all selling point layer nodes are extracted from the final full node feature matrix according to the node level index, forming a preliminary aggregated embedding vector set of selling point nodes.

[0474] The embedding has integrated information from all entities under the corresponding selling point and multimodal underlying features, but it has not yet completed the decoupling and separation of essential attributes and scene confounding factors. It is only a preliminary aggregation representation, which serves as the input for the subsequent decoupling process in step 3.4.

[0475] Step 34: Using a learnable split matrix, each initial aggregated embedding vector is split into essential selling point factors and scenario confounding factors. Combined with contrastive learning decoupling loss, optimization constraints are applied to make the essential factor features of the same selling point in different scenarios approach consistency and maximize the differences in scenario confounding factor features. After convergence, the essential selling point factor set corresponding to all selling points is extracted.

[0476] Step 34 involves stripping away background and environmental factors at the feature level, retaining only the pure selling points strongly related to the product's attributes. The overall solution is based on a mature linear projection decomposition and contrastive learning framework, with simple and controllable technical implementation and clear boundaries between training and inference.

[0477] Step 34 includes the following steps:

[0478] Step 341: Two-factor splitting based on learnable linear projection

[0479] Feature splitting is achieved through a pre-trained learnable linear projection layer (i.e., the learnable splitting matrix described in the claims), which is a mature application of existing fully connected layers and has no complex network structure.

[0480] The input is a preliminary aggregated embedding vector for a single selling point, with a fixed dimension of 768.

[0481] The linear projection layer maps the input vector into two feature vectors, each with a dimension of 384: one is the essential selling point factor, which carries the core and stable selling point information such as the product's physical attributes and functional characteristics, and is not affected by changes in the external scene; the other is the scene confounding factor, which carries scene interference information unrelated to the product itself, such as the shooting background, lighting environment, and matching objects.

[0482] The parameters of this linear projection layer are optimized during the model training phase, and the weights are fixed during the inference phase. The splitting can be completed by performing only one matrix multiplication operation, which is extremely efficient.

[0483] Step 342: Decoupling Constraints Based on Contrastive Learning (Executed during the training phase)

[0484] The decoupling constraint only takes effect during the model training phase. By constructing contrastive sample pairs and optimizing the projection layer parameters in conjunction with the loss function, the two types of factors after splitting satisfy the preset semantic boundaries. This is a common application of existing contrastive learning techniques in causal decoupling scenarios.

[0485] Constructing the comparison sample pairs:

[0486] During training, multiple sets of samples of the same product in different shooting scenarios and display environments are taken and processed through steps 3.1 to 3.3 above to obtain multiple sets of preliminary aggregated embeddings corresponding to the same selling point; each pair of embeddings of the same selling point in different scenarios constitutes a positive sample pair, which generates two sets of essential factors and two sets of confounding factors.

[0487] At the same time, negative sample pairs are randomly selected from the essential factors of different selling points to enhance the feature differentiation between different selling points and avoid feature confusion of the essential factors of different selling points.

[0488] 2) Decoupling loss function

[0489] The optimization objective of the training is to learn the decoupling loss through contrastive learning. The total loss is composed of a weighted average of the essential factor similarity loss and the confounding factor difference loss. The formula for the total loss is:

[0490] ;

[0491] in, To compare the total loss value of decoupling during learning; , This represents the essential factor vectors of the same selling point in two different sets of scenario samples. This is the essential factor similarity loss function, used to constrain the essential factor features of the same selling point to be as consistent as possible in different scenarios; This represents the scene confounding factor vector for the same selling point in two different sets of scene samples. This is the confounding factor difference loss function, used to constrain the maximization of confounding factor feature differences under different scenarios; The balancing coefficient is used to adjust the weight ratio of the two types of losses. Its value ranges from 0.5 to 2.0, and in this embodiment, it is set to 1.0.

[0492] The essential factor similarity loss is calculated by subtracting the cosine similarity from 1, using the following formula:

[0493] ;

[0494] in, It is the inner product of two essential factor vectors; These are the L2 norms (modulus) of the two vectors, respectively.

[0495] The smaller the loss value, the higher the similarity of the essential factors of the same selling point in the two scenarios, indicating that the essential factors are not affected by changes in the scenario.

[0496] The confounding factor difference loss is calculated using the negative value of the Euclidean distance, as shown in the formula:

[0497] ;

[0498] in, Let be the Euclidean distance between two confounding factor vectors;

[0499] The smaller the loss value, the greater the difference in confounding factors between the two scenarios, indicating that scenario-related information is completely carried by confounding factors and does not penetrate the essential factor space.

[0500] For negative sample pairs across selling points, an additional negative sample constraint is added to the similarity loss: the feature distance of the essential factors of different selling points is increased, and the loss calculation logic is reversed with that of positive samples. Based on this, the split matrix parameters are optimized by backpropagation.

[0501] (3) Convergence criterion

[0502] End-to-end backpropagation optimization is performed using the AdamW optimizer. When the decoupling loss decreases by less than 1×10 over 10 consecutive training epochs... -5 When the average cosine similarity of the essential factors of the same selling point across different scenarios is stable above 0.9, the decoupling is considered to have converged, and the parameters of the split matrix are fixed.

[0503] Step 343: In the inference generation stage, contrastive learning and loss calculation are not performed. Feature splitting is completed only through the fixed learnable split matrix. Scene confounding factors are directly discarded, and the essential factors corresponding to all selling points are extracted to form the essential factor set of selling points corresponding to the current input material.

[0504] Output specifications: Each selling point corresponds to a 384-dimensional essential factor vector, and the essential factors of all selling points together constitute the essential factor set of selling points;

[0505] This factor set retains only the essential attribute information of the product's selling points, completely eliminating the interference of mixed factors such as shooting scene and environmental background, and can serve as an unbiased semantic basis for subsequent generation guidance.

[0506] Step 35: Concatenate and linearly map the essential selling point factor set with the style semantic features input by the user to generate a target multimodal joint conditional embedding vector for guiding generation.

[0507] Step 35 combines the essential selling point factor set with the user-specified style semantic features, and generates a target multimodal joint conditional embedding vector with unified dimensions and complete semantics through feature concatenation and linear projection fusion, which serves as the guiding condition for the subsequent diffusion generation module.

[0508] Style semantic feature preprocessing

[0509] First, the style description text input by the user (such as minimalist business style, Chinese trendy festive style, etc.) is standardized and encoded to obtain style semantic features that can be used for fusion:

[0510] Using the BERT-base pre-trained model, which is the same as the text encoding in step 3.2, semantic encoding is performed on the style description text input by the user, and the output vector of the mask bit is taken as the initial style feature;

[0511] The initial style features are calibrated into 768-dimensional standardized style semantic features by using a fixed linear projection layer, thus aligning them with the feature space of the selling point features.

[0512] This encoding process is a mature and general-purpose text encoding technology. The model weights are fixed during the inference phase, and the encoding latency of a single text is in the millisecond range.

[0513] Multi-source feature splicing

[0514] Discrete selling point essential factors and style semantic features are sequentially concatenated along the channel dimension to form a high-dimensional concatenated vector that integrates multi-source semantics. The specific concatenation logic is as follows:

[0515] Selling point factor sequence concatenation: All essential selling point factors output in step 3.4 are concatenated sequentially according to the preset selling point priority order to form a global selling point representation vector. If the product has a total of M selling points, and each essential selling point factor has a dimension of 384, then the concatenated global selling point representation vector will have a dimension of M×384.

[0516] Cross-modal feature concatenation: The global selling point representation vector and style semantic features are directly concatenated along the channel dimension to obtain the final high-dimensional concatenated vector. The total dimension of the concatenated vector is M×384+768.

[0517] The splicing operation adopts the method of connecting the first and last features along the feature dimension without making any non-linear transformation, thus preserving the original semantic information of the two types of features completely and avoiding information loss.

[0518] Linear mapping fusion

[0519] By using a learnable fully connected linear projection layer, feature fusion and dimensionality calibration are performed on the high-dimensional concatenated vector to generate a target multimodal joint conditional embedding vector of uniform dimension.

[0520] The input dimension of the linear projection layer is exactly the same as the total dimension of the spliced ​​vector (M×384+768 dimensions), and the output dimension is fixed at 768 dimensions.

[0521] The operation logic is standard matrix multiplication plus bias operation. Through linear transformation, the multi-source and scattered features are mapped to a unified feature space. Semantic fusion is completed while reducing dimensionality, so that the final vector carries both the essential semantic information of selling points and style semantic information.

[0522] The parameters of this linear projection layer are jointly optimized with the overall network during the model training phase, and the weights are fixed during the inference phase. The fusion can be completed with only one matrix operation, resulting in extremely low computational overhead.

[0523] The final output target multimodal joint conditional embedding vector is a dense feature vector with a fixed 768 dimensions, which perfectly matches the input dimension of the subsequent multi-granularity cross-modal conditional decoupling and hierarchical injection module. It can be directly input into the diffusion generation link to guide the generation of promotional material images.

[0524] In an alternative embodiment, after obtaining the essential selling point factor set in step 34 and before generating the target multimodal joint conditional embedding vector in step 35, the method further includes:

[0525] Based on the historical feedback preprocessing dataset and the essential factor set of selling points, selling point perception processing variables, user feedback result variables, and three types of confounding covariates (audience, channel, and scenario) are constructed for each user feedback sample. The standardized observation dataset is obtained by splitting it according to the single selling point dimension.

[0526] Based on a pre-trained multimodal conditional result prediction network, under the premise of fixing all confounding covariates and non-target selling point features, counterfactual feedback results are generated where only the target selling point perception state is reversed, and are paired one-to-one with real samples to form a real-counterfactual paired sample set.

[0527] A causal forest model is used to fit the heterogeneous processing effect of each selling point. Multiple causal trees are constructed with the maximization of the difference in processing effect between nodes as the node splitting criterion. Discretization, aggregation and normalization are performed according to three dimensions: audience, channel and scenario, and a three-dimensional selling point dissemination weight matrix is ​​output.

[0528] Based on the target audience, target channel and target scenario corresponding to the current generated task, the propagation weight value corresponding to each selling point is indexed from the three-dimensional selling point propagation weight matrix, and the propagation weight value is used to perform factor-wise weighted adjustment on the essential factors of the corresponding selling point.

[0529] In step 35, the weighted and adjusted essential selling point factor set is concatenated and linearly mapped and fused with the style semantic features input by the user to generate a target multimodal joint conditional embedding vector for guiding generation.

[0530] In this embodiment, step 4: inputting the target multimodal joint conditional embedding vector used for guidance into the trained image-text joint embedding guided promotional material generation model to obtain promotional material images includes:

[0531] Before startup, the initial state initialization and the pre-calculation of two types of guiding conditions are completed. All operations are performed based on the trained network with fixed weights, without parameter updates.

[0532] Initial hidden noise initialization: Sample and generate a random Gaussian noise matrix that conforms to the standard normal distribution. The spatial size is consistent with the hidden space feature, which is 32×32×4. This matrix serves as the initial noisy hidden space feature for the diffusion iteration, corresponding to the maximum time step t=999 of the diffusion process.

[0533] Pre-calculation of conditional features: The target multimodal joint conditional embedding vector output in step 3 is input into the conditional decoupling subunit of the multi-granularity cross-modal conditional decoupling and hierarchical injection module. Three types of orthogonal subspace representations, namely global semantic embedding, layout style embedding, and element detail embedding, are calculated in one go and can be reused in subsequent full iteration steps without repeated calculation.

[0534] Pre-calculation of layout constraints: The standardized layout reference image output from step 2 is input into the layout prior encoding and injection module. After mask extraction, multi-scale encoding, resolution and channel alignment, a 16×16×256 layout constraint feature is obtained. The style layout layer is fixedly injected in the full iteration step, without the need for repeated calculation in each step.

[0535] Multi-step iterative diffusion denoising operation:

[0536] The DDIM sampler is used to perform deterministic iterative denoising. The total number of sampling steps is preset to 50, and the time step starts from 999 and decreases by a fixed step size to gradually remove noise from the latent space features. The computational process of each iteration is completely consistent, as follows:

[0537] Current time step encoding generation: Based on the time step index t corresponding to the current iteration step, a 256-dimensional time step embedding vector is generated through sinusoidal position encoding, and input into all residual attention blocks of the hierarchical denoising UNet backbone network.

[0538] Dynamic conditional injection weight calculation: The current time step is embedded into the gating network of the input hierarchical gating injection subunit, and three weight values ​​in the range of 0 to 1 are dynamically generated: semantic injection weight, style injection weight, and detail injection weight, which are adapted to the control requirements of the current denoising stage.

[0539] Hierarchical conditional feature injection: Combining dynamic injection weights, global semantic embedding, layout style embedding, and element detail embedding are injected into the semantic control layer, style layout layer, and detail control layer of the hierarchical denoising UNet backbone network through a cross-attention mechanism, respectively, to generate dynamic conditional enhancement features at the corresponding levels.

[0540] Layout constraint feature injection: The layout constraint features obtained from the previous calculation are injected into the input features of the style layout layer by adding the residuals, and together with the style conditions, they constrain the denoising process of the intermediate layer.

[0541] Single-step denoising operation: The hierarchical denoising UNet backbone network, based on the latent space features of the current step, performs a complete forward propagation under the joint guidance of temporal step encoding, three-level dynamic conditional enhancement features, and layout constraint features, and outputs the noise matrix predicted in the current step.

[0542] Latent feature update: Based on the update formula of DDIM sampling, combined with the predicted noise, the parameters of the current time step and the next time step, the latent space features after one step of denoising are calculated and used as the input for the next iteration.

[0543] Repeat the above process until all 50 iterations are completed to obtain fully denoised latent space features.

[0544] Latent space feature decoding and pixel image output:

[0545] The denoised latent space features obtained through iteration are input into the decoder subunit of the latent space encoder-decoder module to perform pixel restoration:

[0546] The decoder gradually upsamples and restores the 32×32×4 latent space features through multi-level transposed convolution upsampling and residual operation, and finally outputs 256×256×3 normalized pixel features with a value range of [-1,1].

[0547] Perform inverse normalization of pixel values ​​to linearly map the features in the [-1,1] interval to the standard image pixel range of [0,255], and convert them into an 8-bit RGB three-channel image.

[0548] Optional post-processing optimization: Perform slight edge sharpening and color correction on the output image to improve visual quality, ultimately resulting in promotional material images that can be directly used for distribution.

[0549] The following examples further illustrate this application in detail. It is understood that these examples do not constitute any limitation on this application.

[0550] Step 1: Obtain a trained promotional material generation model guided by image-text joint embedding.

[0551] This model consists of four parts: a multi-granularity cross-modal conditional decoupling and hierarchical injection module, a format prior encoding and injection module, a hierarchical denoising UNet backbone network, and a latent space encoding and decoding module. It is obtained through two-stage progressive training.

[0552] The first stage involves pre-training the latent space encoder / decoder module, using a total of 1.2 million training samples (general images + e-commerce promotional materials), with a total of 100 training epochs, a batch size of 32, and an initial learning rate of 4.5 × 10⁻⁻⁶. 5 After convergence, all parameters are frozen.

[0553] The second phase of joint training covers the remaining three modules, with a total of 60,000 mobile phone category-specific promotional materials as training samples. The training consists of 80 rounds, a batch size of 16, and an initial learning rate of 1×10⁻⁻⁻⁶. 4 After convergence, the final generative model is obtained.

[0554] The core fixed parameters of the model are: latent space feature size 32×32×4, and the resolutions of the three levels of the hierarchical UNet are 8×8 (semantic control layer), 16×16 (style layout layer), and 32×32 (detail control layer), respectively. The basic dimension of conditional embedding is 768.

[0555] Step 2: Obtain promotional materials input by the user

[0556] Users upload three types of input data: a real photo of the mobile phone product with a white background (PNG format, original resolution 800×800), a reference layout poster image (JPG format, original resolution 750×1000), and auxiliary text description "lightweight business style, 6.7-inch ultra-clear screen, 67W fast charging" (48 characters in total).

[0557] After standardized preprocessing: both types of images are uniformly scaled to 224×224 resolution RGB three-channel format, and pixel values ​​are normalized to the range of [-1,1]; the text is cleaned, completed and formatted, and outputs normalized semantic text.

[0558] Step 3: Obtain the target multimodal joint conditional embedding vector

[0559] A four-level causal semantic ontology structure is constructed: 25 entity layer nodes (including component and material nodes such as screen, battery, and mid-frame) are preset for the mobile phone category, 18 selling point layer nodes (including selling point nodes such as lightweight, high-definition display, and fast charging) are preset, and 4 value layer nodes are preset; only the adjacent levels are set with unidirectional causal dependency edges from bottom to top, and there are no related edges in the reverse direction or across layers.

[0560] Extract and mount multimodal low-level features: 196 768-dimensional patch region features are extracted on the visual side, and token-level and global semantic features are extracted on the text side. All of them are uniformly mapped to 768 dimensions. Using a cosine similarity of 0.7 as the mounting threshold, all feature nodes are matched and mounted to the corresponding entity nodes to complete the construction of a cross-modal causal heterogeneous graph.

[0561] Two-layer progressive feature aggregation: The forward aggregation of “pixel parameter layer → entity layer” and “entity layer → selling point layer” is completed sequentially through two layers of causal perceptual graph attention network. Non-causal paths are blocked by masks, and the 768-dimensional preliminary aggregated embedding vector corresponding to each selling point is output.

[0562] Decoupling of essential selling point factors: Through a learnable split matrix, each 768-dimensional initial embedding is split into two 384-dimensional vectors, namely the essential selling point factor and the scenario confounding factor; the scenario confounding factor is directly discarded to obtain the essential factor set corresponding to all 18 selling points.

[0563] The final embedding is generated by fusion: all essential selling points and 768-dimensional style semantic features are concatenated by channel, and after fusion and dimensionality reduction by a linear projection layer, the final 768-dimensional target multimodal joint conditional embedding vector is output.

[0564] Step 4: Generate promotional material images

[0565] Standard Gaussian noise with an initial size of 32×32×4 is used as the initial noisy latent space feature for diffusion iteration; three types of orthogonal subspace conditions and layout constraint features of 16×16×256 are obtained through pre-calculation and are reused throughout the entire iteration process.

[0566] The DDIM sampler is used to perform 50-step iterative denoising: each step generates a corresponding time step code and dynamically calculates the three-level condition injection weights; the three types of conditions, global semantics, layout style, and element details, are injected into the corresponding 8×8, 16×16, and 32×32 layers of UNet respectively, and the layout constraint features are injected into the style layout layer to complete the single-step denoising operation.

[0567] After completing all 50 iterations, the denoised 32×32×4 latent features are input into the decoder to restore a 256×256 resolution RGB three-channel image, with pixel values ​​mapped to the standard range of 0~255. The final output is a promotional image that can be directly used for e-commerce channel distribution.

[0568] Although the present invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention are within the scope of protection claimed by the present invention.

Claims

1. A method for intelligently generating promotional materials based on image and text joint embedding, characterized in that, The intelligent generation method for promotional materials based on image and text joint embedding includes: Step 1: Obtain a trained promotional material generation model guided by image-text joint embedding; Step 2: Obtain the promotional materials input by the user; Step 3: Obtain the target multimodal joint conditional embedding vector for guiding generation based on the promotional materials input by the user; Step 4: Input the target multimodal joint conditional embedding vector used for guidance into the trained image-text joint embedding guided promotional material generation model to obtain promotional material images.

2. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 1, characterized in that, The promotional material generation model guided by the combined embedding of text and images includes: A multi-granularity cross-modal conditional decoupling and hierarchical injection module is used to receive the target multi-modal joint conditional embedding vector, combine it with the current time step encoding generated during the diffusion iteration process, perform multi-granularity semantic decomposition and orthogonal decoupling processing on the joint embedding vector, and generate dynamic conditional enhancement features corresponding to different generation levels. The layout prior encoding and injection module is used to extract the layout prior mask from the acquired user-input promotional materials, perform convolutional encoding and resolution alignment processing on the layout prior mask, and generate layout constraint features. The hierarchical denoising UNet backbone network is used to receive initial noisy latent space features, diffusion time step encoding, dynamic conditional enhancement features and layout constraint features as generation guidance conditions. After multi-level condition-guided diffusion denoising operation, it outputs the denoised latent space features. The latent space encoding and decoding module is used to receive latent space features, decode them to restore the image data in pixel space, and output the final promotional material image. During the training phase, the latent space encoding and decoding module is also used to perform encoding compression on the original promotional material image and output standardized latent space features for model training.

3. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 2, characterized in that, The multi-granularity cross-modal conditional decoupling and hierarchical injection module includes a conditional decoupling subunit and a hierarchical gated injection subunit; wherein... The conditional decoupling subunit is used to receive the target multimodal joint conditional embedding vector, and to map the joint embedding into three subspace vectors of semantic representation, style representation and detail representation through a three-way parallel projection network. Orthogonal constraint processing is performed on the three subspace vectors to output global semantic embedding, layout style embedding and element detail embedding. The hierarchical gated injection subunit is used to receive the current time step encoding of global semantic embedding, layout style embedding and element detail embedding and diffusion iteration process, calculate the dynamic injection weights of the corresponding time steps of receiving global semantic embedding, layout style embedding and element detail embedding respectively, and inject global semantic embedding, layout style embedding and element detail embedding into the hierarchical denoising UNet backbone network respectively through cross attention mechanism, and generate dynamic conditional enhancement features of the corresponding level by combining the dynamic injection weights.

4. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 3, characterized in that, The layout prior encoding and injection module includes a mask extraction subunit, a layout encoding subunit, and a feature alignment and fusion subunit; wherein... The mask extraction subunit is used to perform region semantic segmentation on the acquired user input promotional materials, identify three types of functional areas in the materials: product display area, title copy area and white space area, and generate a binary layout prior mask. The layout coding subunit is used to receive the layout prior mask, extract multi-scale layout structure features through multi-layer convolutional network and residual unit, and enhance the feature weight of core functional area and suppress the feature weight of white space area through spatial attention mechanism, and output the enhanced layout feature map. The feature alignment and fusion subunit is used to perform resolution upsampling and channel dimension alignment processing on the enhanced layout feature map to generate layout constraint features that match the resolution of the hierarchical denoising UNet backbone network.

5. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 4, characterized in that, The hierarchical denoising UNet backbone network includes a semantic control layer, a style layout layer, and a detail control layer, with the three layers connected sequentially from low to high feature resolution. The semantic control layer includes a residual attention block at the end of the downsampling path, multiple bottleneck residual attention blocks, and a residual attention block at the front end of the upsampling path, corresponding to the lowest resolution feature dimension of the network. It is used to access the global semantic corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, and to perform denoising feature operation of the global semantic dimension in combination with diffusion time step encoding, and output the semantically constrained deep features to the style layout layer. The style layout layer includes a residual attention block in the middle of the downsampling path and a residual attention block in the middle of the upsampling path, corresponding to the resolution feature dimension in the network. It is used to simultaneously access the layout style corresponding dynamic condition enhancement features output by the multi-granularity cross-modal condition decoupling and hierarchical injection module, as well as the layout constraint features output by the layout prior encoding and injection module. Combined with diffusion time step encoding, it performs denoising feature operation on the style layout dimension and outputs the mid-layer features after style and layout constraints to the detail control layer. The detail control layer includes a residual attention block at the front end of the downsampling path and a residual attention block at the end of the upsampling path, corresponding to the highest resolution feature dimension of the network. It is used to access the element detail corresponding to the dynamic condition enhancement features output by the multi-granularity cross-modal conditional decoupling and hierarchical injection module, and to perform denoising feature operations on the pixel detail dimension in combination with diffusion time step encoding, and finally output the denoised latent space features.

6. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 5, characterized in that, The latent space encoding / decoding module includes an encoder subunit and a decoder subunit; The encoder subunit is used to receive the original promotional material image during the model training phase, and to perform dimensionality compression processing on the image through a multi-level convolutional downsampling unit and a residual operation unit, mapping the image data in the pixel space to a low-dimensional latent space, and outputting standardized latent space features as the training input of the hierarchical denoising UNet backbone network. The decoder subunit is used to receive the denoised latent space features output by the hierarchical denoising UNet backbone network during the material generation stage. It performs dimension restoration processing on the latent features through multi-level transposed convolutional upsampling units and residual operation units, mapping the low-dimensional latent space data back to the pixel space and outputting the final promotional material image.

7. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 6, characterized in that, The promotional material generation model guided by the joint embedding of text and images is obtained through iterative training using a multi-objective joint loss function. The total loss function during the training process is as follows: ； in, For the total loss, For layered noise prediction loss, For hierarchical cross-modal alignment loss, Due to layout constraints, For conditional orthogonal constraint loss, , , These are the preset balancing weight hyperparameters; The hierarchical noise prediction loss is: ； in, , , The mean squared error of noise prediction for the semantic control layer, style layout layer, and detail control layer are respectively. , , Preset weights for the corresponding levels; The hierarchical cross-modal alignment loss is: ； in, , , These are the global average pooling vectors representing the output features of the semantic control layer, style layout layer, and detail control layer, respectively. , , These are global semantic embedding, layout style embedding, and element detail embedding, respectively. The cosine similarity function; The layout constraint loss is: ； in, To generate salient subject masks corresponding to latent features, For layout prior mask, The function for calculating the intersection-union ratio; The loss due to the orthogonal constraint is: ； in, The inner product of the global semantic embedding vector and the layout style embedding vector; The inner product of the global semantic embedding vector and the element detail embedding vector; The inner product of the layout style embedding vector and the element details embedding vector.

8. The intelligent generation method for promotional materials based on image and text joint embedding as described in claim 7, characterized in that, Step 3 includes: Step 31: Construct a four-level hierarchical multi-layer causal semantic ontology structure. The multi-layer causal semantic ontology structure includes, from bottom to top, a pixel parameter layer, an entity layer, a selling point layer, and a value layer. Only one-way causal dependency edges are set between nodes of adjacent levels from bottom to top. No association edges are set between nodes of the reverse level and non-adjacent levels. This blocks the transmission path of non-causal mixed information from the ontology structure level. Step 32: Extract the multimodal low-level features corresponding to the user-input promotional materials, attach all multimodal low-level features as leaf nodes to the pixel parameter layer, and construct a cross-modal causal heterogeneous graph according to the connection rules of the unidirectional causal dependency edges; Step 33: Use a multi-layer causal perceptual graph attention network to perform message passing and feature aggregation along the causal forward direction on the cross-modal causal heterogeneous graph, and block the reverse and cross-layer information passing through the causal indicator factor mask to obtain the preliminary aggregated embedding vector corresponding to each selling point semantic node; Step 34: Using a learnable split matrix, each initial aggregated embedding vector is split into essential selling point factors and scenario confounding factors. Combined with contrastive learning decoupling loss, optimization constraints are applied to make the essential factor features of the same selling point in different scenarios approach consistency and maximize the differences in scenario confounding factor features. After convergence, the essential selling point factor set corresponding to all selling points is extracted. Step 35: Concatenate and linearly map the essential selling point factor set with the style semantic features input by the user to generate a target multimodal joint conditional embedding vector for guiding generation.

9. A smart device for generating promotional materials based on image and text co-embedding, characterized in that, The intelligent promotional material generation device based on image and text joint embedding includes: The promotional material generation model acquisition module is used to acquire a trained promotional material generation model guided by image and text joint embedding. A promotional material acquisition module, which is used to acquire promotional materials input by the user; An embedding vector acquisition module is used to acquire a target multimodal joint conditional embedding vector for guiding generation based on promotional materials input by the user. The promotional material image acquisition module is used to input the target multimodal joint conditional embedding vector used for guided generation into the trained image-text joint embedding guided promotional material generation model, thereby acquiring promotional material images.

Images