Ad creative matching system based on multi-modal content generation

Abstract

The application discloses an advertisement creative matching system based on multi-modal content generation and concretely relates to the technical field of electric digital data processing, comprising a cross-modal time axis anchoring module, a two-way mapping matrix is established, word-frame pairs with a confidence lower than a threshold value are set to zero, and an anchoring pair set is obtained; a semantic hierarchy stripping module, text word elements are disassembled into three layers, three-layer mutual information difference values are calculated to mark metaphorical word elements, and a semantic constraint list is generated; a visual symbol axis indexing module, image frame attribute values are calculated along a pre-extracted symbol axis and mapped to discrete intervals, and a visual symbol label tuple is generated; a cultural fingerprint constraint module, a matching regional culture taboo vector and a brand symbol vector are retrieved, and a semantic guardrail is generated; and a cross-modal creative generation module, joint features and the semantic guardrail are input into a generator hidden space, dimension pruning is performed according to the semantic guardrail, only dimension components with a dot product greater than zero are reserved, and a constrained advertisement creative sample is output.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and more specifically, to an advertising creative matching system based on multimodal content generation. Background Technology

[0002] With the rapid development of the digital advertising industry, the intelligent generation of advertising creative content has become a key technology direction in the field of advertising production. In recent years, with the continuous progress of multimodal data processing technology and generative artificial intelligence, the automatic generation method of advertising content based on multimodal fusion has gradually become a research hotspot, providing a new technical path for improving the generation efficiency and personalization level of advertising creative content.

[0003] Existing technical solutions primarily achieve automatic advertising content generation through multimodal feature extraction and fusion. For example, unified features are extracted from raw multimodal data, and after semantic fusion and similarity enhancement, a fused representation is obtained. Then, combined with user feedback, a generative network outputs preliminary creative content. By calculating the matching degree, a market-adaptive adjustment vector is generated, and the creative content is iteratively optimized. Alternatively, cross-modal temporal anchoring is used to associate text and images at the temporal level, and metaphorical hierarchy decomposition and symbol axis annotation are performed on this basis.

[0004] However, in practical use, it still has some shortcomings. For example, in terms of temporal alignment of multimodal data, most existing solutions use coarse timestamp matching or global feature alignment, failing to establish a precise bidirectional mapping relationship at the word and frame levels. This results in insufficient accuracy of the correspondence between text and image in the time dimension, affecting the consistency of subsequent creative generation. In terms of text semantic parsing, existing solutions lack a systematic hierarchical parsing mechanism for the numerous metaphorical and symbolic expressions in advertising texts. They usually only perform general semantic feature extraction, making it difficult to accurately identify the cultural connotations carried by metaphorical words. In terms of cultural compliance constraints, existing solutions usually perform cultural adaptability verification or post-processing filtering after creative generation. This post-constraint method cannot avoid culturally taboo content from the source during the generation process. Moreover, the identification of cultural taboos is often based only on word matching or single-modal information, lacking the ability to deeply constrain semantic and visual multimodal collaboration. In terms of controlling the generation process, existing solutions lack a mechanism to actively prune and constrain the generation direction at the latent space level of the generator, making it difficult to ensure that the generated content always conforms to the brand tone and regional cultural norms while maintaining creative diversity. To address the above problems, this invention proposes an advertising creative matching system based on multimodal content generation. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an advertising creative matching system based on multimodal content generation, which solves the problems mentioned in the background art through the following solutions.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an advertising creative matching system based on multimodal content generation, including a cross-modal time axis anchoring module: receiving a text sequence and an image sequence, establishing a bidirectional mapping matrix based on the time step of the word in the text sequence and the timestamp of the frame in the image sequence, setting the word-frame pairs with confidence levels below a preset threshold to zero, and obtaining the corresponding anchoring pair set;

[0007] Semantic layer stripping module: Decomposes the text lexical units in the anchor pair set into surface meaning layer, metaphorical meaning layer and deep semantic layer, calculates the mutual information difference between the three layers of vectors, marks the lexical units with mutual information difference exceeding the threshold as metaphorical lexical units, and generates a list of semantic constraints including the position of metaphorical lexical units and the corresponding three layers of vectors.

[0008] Visual symbol axis indexing module: For image frames in the anchor pair set, calculate symbol attribute values ​​along the symbol axis, map the values ​​to the corresponding discrete intervals, and generate visual symbol label tuples;

[0009] Cultural fingerprint constraint module: retrieves regional cultural taboo vectors and brand symbol vectors that simultaneously match the metaphor word metavectors and visual symbol tag tuples in the semantic constraint list, synthesizes high-dimensional constraint vectors and uses them as semantic guardrails;

[0010] Cross-modal creative generation module: After aligning the text word vectors and image frame feature vectors in time, concatenate them into joint features. Input the joint features and semantic guardrails into the latent space of the generator. In the latent space, perform dimensional pruning on the generation path according to the semantic guardrails: only retain the dimensional components whose dot product with the semantic guardrail vector is greater than zero, and output at least one advertising creative sample that integrates text and image and is constrained by the semantic guardrails.

[0011] Preferably, in the cross-modal time axis anchoring module, the time step of the word in the text sequence is calculated based on the standard Chinese broadcast speech rate, and each word corresponds to a reference timestamp; the timestamp of the image frame is the display timestamp of each frame in the original video or an incremental timestamp allocated according to the input order; the row index of the bidirectional mapping matrix corresponds to the position of the text word, the column index corresponds to the position of the image frame, and the matrix element value is the cosine similarity between the corresponding word feature and the image frame feature.

[0012] Preferably, in the semantic layer stripping module, the surface meaning layer extracts the literal meaning vectors of word elements through a Chinese pre-trained language model; the metaphorical meaning layer generates escape vectors based on a metaphor dictionary specific to the advertising domain, and unregistered word elements obtain approximate escape vectors by retrieving the labeled metaphorical word elements with the highest semantic similarity through a general metaphor embedding model; the deep semantic layer generates hypersensory vectors based on a knowledge graph in the advertising and marketing domain.

[0013] Preferably, in the semantic layer stripping module, the mutual information difference between the three layers of vectors is obtained by calculating the mutual information between the surface meaning layer and the metaphorical meaning layer, and the mutual information between the surface meaning layer and the deep semantic layer respectively using the Gaussian kernel density estimation method, and then taking the absolute value of the difference; when the mutual information difference exceeds a preset threshold, the corresponding word is determined to be a metaphorical word.

[0014] Preferably, in the visual symbol axis indexing module, the symbol axis is a set of axes pre-extracted from the historical advertising creative dataset using non-negative matrix decomposition, including at least the yin-yang axis, the geometric axis, and the hue axis; each axis has an independent discrete value range, and the values ​​of the image frame on each axis are combined into visual symbol label tuples after discrete mapping.

[0015] Preferably, in the visual symbol axis indexing module, the pre-extraction of symbol axes includes: extracting visual features from historical advertising images and performing non-negative translation processing to form a feature matrix; setting the number of basis vectors and then iteratively updating the basis matrix and coefficient matrix through non-negative matrix decomposition; performing orthogonalization processing on the basis matrix to obtain multiple mutually orthogonal symbol axes; manually semantically summarizing the coefficient values ​​of each axis to label the axis type; and using the equal-frequency discretization method to divide the discrete interval.

[0016] Preferably, in the cultural fingerprint constraint module, the regional cultural taboo vector is retrieved from a pre-built regional cultural taboo sub-database; and the brand symbol vector is retrieved from a pre-built brand symbol sub-database.

[0017] Preferably, in the cultural fingerprint constraint module, the high-dimensional constraint vector is synthesized as follows: the metaphor word vector is weighted and fused and then projected to the same dimension as the generator's latent space to obtain the semantic taboo vector; the embedded vector is retrieved to obtain the visual taboo vector; the semantic taboo vector and the visual taboo vector are weighted and synthesized into a comprehensive taboo vector; the brand symbol vector of the target brand is subtracted from the product of the constraint strength coefficient and the comprehensive taboo vector to obtain the semantic guardrail vector.

[0018] Preferably, in the cultural fingerprint constraint module, the semantic taboo vector is obtained by projecting the features of taboo words extracted by a pre-trained language model onto the same dimension as the generator's latent space; the visual taboo vector is obtained by projecting the visual features of taboo patterns or symbols extracted onto the same dimension as the generator's latent space; and the brand symbol vector is obtained through multimodal feature fusion training.

[0019] Preferably, in the cross-modal creative generation module, the joint feature is generated as follows: the three-layer semantic vectors are weighted and fused and then projected onto the generator latent space dimension to obtain text features; the visual features are projected onto the generator latent space dimension to obtain visual features; and the text features and visual features are fused to output a joint feature vector.

[0020] The technical effects and advantages of this invention are as follows:

[0021] 1. This invention establishes a bidirectional mapping matrix based on the time step of words in the text sequence and the timestamp of frames in the image sequence through a cross-modal time axis anchoring module. It also forces word-frame pairs with confidence scores below a preset threshold to be set to zero, thereby achieving precise alignment between word and frame levels in the time dimension. This effectively filters out noise pairs with weak temporal correlation, providing a high-precision multimodal alignment foundation for subsequent semantic parsing, symbol indexing, and creative generation. It significantly improves the temporal consistency of text and image content during the advertising creative generation process.

[0022] 2. This invention decomposes text lexical units into surface meaning layer, metaphorical meaning layer and deep semantic layer through a semantic layer stripping module. It combines mutual information difference to achieve accurate identification of metaphorical lexical units. It also uses a visual symbol axis indexing module to discretize and index image frames along mutually orthogonal symbol axes. Finally, it uses a cultural fingerprint constraint module to retrieve and match the regional cultural taboo vector and brand symbol vector of the metaphorical lexical unit vector and the visual symbol label tuple. It synthesizes a high-dimensional constraint vector as a semantic guardrail, realizing deep cultural adaptation of semantic and visual multimodal collaboration.

[0023] 3. This invention uses a cross-modal creative generation module to input joint features and semantic guardrails into the latent space of the generator. In the latent space, the generation path is dimensionally pruned according to the semantic guardrails, that is, only the dimensional components whose dot product with the semantic guardrail vector is greater than zero are retained. At the same time, the generator is forcibly prohibited from exploring latent dimensions that are close to the taboo vector direction. This achieves full compliance and controllability of the generated content without sacrificing the diversity of creativity, effectively balancing the richness and security of advertising creativity. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0025] Figure 2 This is a schematic diagram of the cross-modal time axis anchoring of the present invention;

[0026] Figure 3 This is a schematic diagram of semantic hierarchy stripping and metaphor determination in this invention;

[0027] Figure 4 This is a schematic diagram of the visual symbol axis index of the present invention;

[0028] Figure 5 This is a schematic diagram illustrating the cultural constraints and creative generation of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, the advertising creative matching system based on multimodal content generation includes a cross-modal time axis anchoring module: receiving text sequences and image sequences, establishing a bidirectional mapping matrix based on the time step of the words in the text sequence and the timestamp of the frames in the image sequence, setting word-frame pairs with confidence levels below a preset threshold to zero, and obtaining the corresponding anchoring pair set.

[0031] like Figure 2 As shown, it should be specifically noted that in the cross-modal time axis anchoring module, the time step of the word in the text sequence is calculated based on the standard Chinese broadcast speech rate, and each word corresponds to a base timestamp; the timestamp of the image frame is the display timestamp of each frame in the original video or an incremental timestamp allocated according to the input order; the row index of the bidirectional mapping matrix corresponds to the position of the text word, the column index corresponds to the position of the image frame, and the matrix element value is the cosine similarity between the corresponding word feature and the image frame feature.

[0032] It should be further explained that the cross-modal time axis anchoring module takes the original input advertising text sequence and reference image sequence of the system, outputs a set of structured anchor pairs, and provides aligned lexical and frame data to the subsequent semantic level stripping module and visual symbol axis indexing module.

[0033] The system receives text and image sequences via a multi-source data acquisition device. For text, it supports three input methods: API input, structured file import, and front-end interface input. For images, it supports three input methods: video file upload, ordered image set import, and access to the built-in media library. A unified timeline benchmark is constructed for both types of input data to eliminate timing discrepancies caused by differences in input formats.

[0034] Text side: The input is a 15-second short video advertisement script text. The BERT-base Chinese word segmenter is used for word segmentation and lexical segmentation, resulting in a lexical sequence of length N. Based on the standard Chinese broadcast speech rate of 3.5 words / second, the time interval for each lexical is calculated: The start time of each word element seconds, end time Seconds, take the midpoint of the interval as the base timestamp of the word. The time accuracy is preserved to 0.1 seconds.

[0035] On the image side: The input is an advertising reference storyboard video or raw video. An equal-interval frame extraction strategy is adopted to extract keyframes at a sampling rate of 1 frame / second, resulting in an image frame sequence of length M. Each frame carries a corresponding physical timestamp, which is the display timestamp (PTS) of the corresponding frame in the original video. It is directly read from the video file container by FFmpeg, and the unit is seconds, which is consistent with the time base of text words. If the input is a static image set, timestamps are assigned sequentially in increments of 1 second according to the input order to ensure temporal continuity.

[0036] The CLIP ViT-L / 14 pre-trained model is used to extract features from two types of data: on the text side, the hidden state at the corresponding position of each word is extracted from the word dimension output of the last layer of the CLIP text encoder as a 768-dimensional text feature vector; on the image side, the output of the global pooling layer of the visual encoder is extracted as a 768-dimensional visual feature vector. All feature vectors are subjected to L2 normalization to eliminate the interference of vector magnitude on similarity calculation.

[0037] Construct a bidirectional mapping matrix Q of size N×M, where the row index corresponds to the text word position and the column index corresponds to the image frame position. The value of each element in the matrix is ​​the cosine similarity between the corresponding word feature and the image frame feature, calculated using the following formula:

[0038]

[0039] in, Let i be the feature vector of the i-th text word. Let j be the feature vector of the j-th frame image. These are the L2 norms (vector magnitudes) of the text feature vector and the image feature vector, respectively. The element values ​​range from [0,1]. The higher the value, the stronger the semantic association between the corresponding word and the corresponding frame, i.e., the higher the anchoring confidence.

[0040] Optimization was performed on a validation set of 1000 manually labeled alignment relationships for advertisements. A confidence threshold τ = 0.65 was pre-set. All elements of the bidirectional mapping matrix were traversed. If an element value M[i][j] < 0.65, the element was forcibly set to zero to filter out semantically weak noise pairs. After traversal, all non-zero elements were collected into triplets (i,j,M[i][j]) to form the final anchor pair set S. a .

[0041] After completing the temporal anchoring, the system inputs all text word sequences in the anchored set into the semantic hierarchy stripping module to perform refined semantic analysis on the numerous metaphorical and symbolic expressions present in the advertising text.

[0042] Semantic layer stripping module: Decomposes text lexical units in the anchor pair set into surface meaning layer, metaphorical meaning layer and deep semantic layer, calculates the mutual information difference between the three layers of vectors, marks lexical units with mutual information difference exceeding the threshold as metaphorical lexical units, and generates a list of semantic constraints including the position of metaphorical lexical units and the corresponding three layers of vectors.

[0043] like Figure 3 As shown, it should be specifically explained that in the semantic layer stripping module, the surface meaning layer extracts the literal meaning vector of word elements through a Chinese pre-trained language model; the metaphorical meaning layer generates escape vectors based on an advertising domain-specific metaphor dictionary, and unregistered word elements obtain approximate escape vectors by retrieving the labeled metaphorical word elements with the highest semantic similarity through a general metaphor embedding model; the deep semantic layer generates hypersensory vectors based on a knowledge graph in the advertising and marketing domain.

[0044] In the semantic layer stripping module, the mutual information difference between the three layers of vectors is obtained by calculating the mutual information between the surface meaning layer and the metaphorical meaning layer, and the mutual information between the surface meaning layer and the deep semantic layer respectively using the Gaussian kernel density estimation method, and then taking the absolute value of the difference; when the mutual information difference exceeds the preset threshold, the corresponding word is determined to be a metaphorical word.

[0045] It should be further explained that the semantic hierarchy stripping module constructs a three-layer semantic vector space, determines metaphorical lexical units through mutual information difference quantization, inputs the text lexical sequence in the anchor pair set, and outputs a structured list of semantic constraints, which serves as the semantic input for the subsequent cultural fingerprint constraint module.

[0046] The semantic hierarchy stripping module generates a 1024-dimensional semantic vector with three levels for each anchor word, and all vectors are subjected to L2 normalization.

[0047] Surface meaning layer: Inputting lexical units into the RoBERTa-large Chinese pre-trained model, taking the output of the last hidden state at the corresponding lexical unit position as the literal meaning vector, directly representing the basic literal semantics of the lexical unit, without the need for additional domain fine-tuning.

[0048] Metaphorical Meaning Layer: Escape vectors are generated based on a metaphor dictionary specific to the advertising field. The dictionary data is derived from the top 1000 commercial advertising copy on Douyin and Tmall platforms over the past 5 years. After manual annotation, 12,000 pairs of "literal meaning-metaphorical meaning" words are obtained, covering 8 major advertising categories: beauty and personal care, food and beverage, 3C digital appliances, clothing and shoes, household goods, maternal and infant products, automobiles and travel, and health care. The metaphor embedding vectors are trained using the Word2Vec algorithm with a training window size of 5 and a vector dimension of 1024. The escape vector is the 1024-dimensional word embedding vector of the metaphorical meaning corresponding to the word element. If the word element matches the dictionary, the corresponding escape vector is directly read. For words not in the dictionary, the Word2Vec general metaphor embedding model trained based on publicly available Chinese metaphor corpus is used to retrieve the labeled metaphor words with the highest semantic similarity, and the corresponding escape vector is used as the approximate escape vector of the word not in the dictionary.

[0049] Deep Semantic Layer: Generates hyperordinate semantic vectors based on a knowledge graph from the advertising and marketing domain. The knowledge graph includes 180,000 entity nodes and 320,000 hyperordinate hierarchical relationships, covering three major entity categories: product categories, consumption scenarios, and emotional attributes. The TransE algorithm is used to train the knowledge graph embedding, with a margin parameter set to 1.0 and a vector dimension of 1024. For each lexical unit, entity linking and matching against the knowledge graph entities are performed: the 1024-dimensional RoBERTa semantic vector of the lexical unit is extracted, and its cosine similarity is calculated with the TransE embedding vectors of each entity in the knowledge graph. The entity with the highest similarity is selected. If the similarity is greater than or equal to a preset threshold of 0.7, the match is considered successful. The threshold of 0.7 is calibrated using a grid search on the entity linking annotation test set in the advertising domain, representing the optimal value for both precision and recall, ensuring the accuracy and coverage of entity links. If the match fails, the first-level category entity in the knowledge graph that is semantically closest to the word is selected as the matching result. After matching is completed, the corresponding third-level superordinate entities are recursively retrieved and weighted by weights of "first level 0.5, second level 0.3, third level 0.2" to obtain the deep superordinate semantic vector of the word.

[0050] The Gaussian kernel density estimation method is used to calculate the mutual information between each pair of vectors in the three layers. The bandwidth of the kernel function is automatically determined by the Silverman criterion.

[0051] Calculate the joint probability density and marginal probability density of the surface meaning layer vector and the metaphorical meaning layer vector, and substitute them into the mutual information formula to obtain the surface-metaphorical mutual information. ;

[0052] Similarly, the mutual information between the surface semantic layer and the deep semantic layer is calculated. ;

[0053] The formula for calculating mutual information is unified as follows: ;

[0054] Define mutual information difference in, Let p(x) be the joint probability density function of X and Y, p(y) be the marginal probability density function, and p(y) be the marginal probability density function of the metaphor vector, describing the probability density of two sets of vectors simultaneously taking corresponding values. This refers to the mutual information between the surface meaning layer and the metaphorical meaning layer. It refers to the mutual information between the surface semantic layer and the deep semantic layer.

[0055] Optimization was performed using a metaphor recognition validation set, with a preset difference threshold δ=0.3. This difference threshold was obtained by grid search calibration on the advertising metaphor annotation test set, representing the optimal value for both precision and recall, effectively distinguishing between metaphorical and non-metaphorical words. When a word is identified as a metaphorical word, meaning that the metaphorical meaning deviates significantly from the literal meaning, it needs to be included in the scope of semantic constraint control.

[0056] To further explain, the probability density is calculated using the Gaussian kernel density estimation method, and the specific steps are as follows:

[0057] Prepare three sets of samples, taking all the words to be judged as units:

[0058] Surface sample set: Each element corresponds to a 1024-dimensional surface meaning vector of a word;

[0059] Metaphorical sample set: Each element corresponds to a 1024-dimensional metaphorical meaning layer vector of a word;

[0060] Joint sample set: The surface vector of each word is concatenated with the metaphor vector to obtain a 2048-dimensional joint vector, and each element corresponds to a pair of word vectors.

[0061] Gaussian kernel density estimation is performed on the surface sample set and the metaphor sample set respectively: a multidimensional Gaussian kernel function is superimposed on each vector sample in the set as the center; the bandwidth of the kernel function is determined adaptively by the Silverman criterion; the kernel function values ​​of all samples are averaged to obtain the marginal probability density function p(x) of the surface vector and the marginal probability density function p(y) of the metaphor vector respectively.

[0062] For all lexics marked as metaphors, a structured list of semantic constraints is generated. Each record includes the following fields: the lexicographic position index in the text sequence, the original lexicographic text, a three-layer semantic vector, mutual information difference, and metaphor confidence score. The list is sorted in ascending order by lexicographic position and is directly passed to the downstream cultural fingerprint constraint module.

[0063] While performing semantic hierarchical parsing on the text side, the system simultaneously inputs the image frames in the anchor pair set into the visual symbol axis indexing module to complete the symbolic encoding on the visual side.

[0064] Visual symbol axis indexing module: For image frames in the anchor pair set, calculate symbol attribute values ​​along the symbol axis, map the values ​​to the corresponding discrete intervals, and generate visual symbol label tuples.

[0065] like Figure 4 As shown, it should be specifically noted that in the visual symbol axis indexing module, the symbol axis is a set of axes pre-extracted from the historical advertising creative dataset using non-negative matrix decomposition, including at least the yin-yang axis, the geometric axis, and the hue axis; each axis has an independent discrete value range, and the values ​​of the image frame on each axis are combined into visual symbol label tuples after discrete mapping.

[0066] In the visual symbol axis indexing module, the pre-extraction of symbol axes includes: extracting visual features from historical advertising images and performing non-negative translation processing to form a feature matrix; setting the number of basis vectors and then iteratively updating the basis matrix and coefficient matrix through non-negative matrix decomposition; performing orthogonalization processing on the basis matrix to obtain multiple mutually orthogonal symbol axes; manually semantically summarizing the coefficient values ​​of each axis to label the axis type; and using the equal-frequency discretization method to divide the discrete interval.

[0067] It should be further explained that the visual symbol indexing module takes as input the image frame sequence in the anchor pair set and outputs a visual symbol label tuple and the corresponding embedding vector, which serves as the visual input to the downstream cultural fingerprint constraint module. The specific implementation steps are as follows:

[0068] The symbolic axis set was pre-trained offline, including collecting 120,000 high-quality advertising images from the ZCOOL commercial advertising material library and the ADCM public advertising dataset, covering eight major advertising categories: beauty and personal care, food and beverage, 3C digital home appliances, clothing and footwear, household goods, maternal and infant products, automobiles and travel, and health care. Data cleaning was performed to remove duplicate images and low-quality images with a resolution lower than 512×512 to obtain training images. The images were uniformly scaled to 224×224 pixels and the pixel values ​​were normalized to the [0,1] interval.

[0069] The ResNet-50 pre-trained model is used to extract the average pooling layer features of each image, resulting in 2048-dimensional visual features. A non-negative translation process (subtracting the global feature minimum) is performed on all features to ensure that all feature elements are non-negative, forming a feature matrix X∈R. 112000×2048 , where R represents the set of real numbers.

[0070] The number of basis vectors is set to K=3, and the optimization objective is to minimize the reconstruction error. The constraints are (Identity matrix); The basis matrix W and coefficient matrix H are iteratively updated using a multiplicative update rule. After each iteration, Gram-Schmidt orthogonalization is performed on W to ensure that the three basis vectors are pairwise orthogonal and semantically non-overlapping; the iteration is completed after 1000 iterations or the reconstruction error is less than [value missing]. The time convergence eventually yields W∈R 2048×3 That is, 3 orthogonal symbol axes a.

[0071] For each axis, the 100 images with the highest coefficient values ​​were selected for manual semantic summarization and labeled as the Yin-Yang axis (corresponding to brightness contrast and light intensity), the geometric axis (corresponding to the hardness / softness of the composition lines), and the hue axis (corresponding to the warm or cool color tendency). The values ​​of each axis were normalized to the [0,1] interval and divided into 3 discrete intervals using the equal frequency discretization method, corresponding to 3 levels of labels, such as the Yin-Yang axis corresponding to low darkness, neutral, and high brightness.

[0072] For each input image frame, the same preprocessing and feature extraction process as in the offline stage is used to obtain 2048-dimensional non-negative visual features, which are then projected onto three pre-trained symbolic axes. The continuous attribute values ​​of each axis are calculated using the following formula:

[0073]

[0074] in, , represents the continuous attribute value of the k-th symbol axis. The value is a real number, and the magnitude of the value reflects the strength of the corresponding frame image in the corresponding symbol dimension. k=1,2,3 correspond to the three pre-trained symbol axes: the yin-yang axis, the geometric axis, and the hue axis, respectively. is a 2048-dimensional non-negative visual feature vector of the i-th frame image. It is obtained by non-negative translation after extracting the average pooling layer features from the ResNet-50 pre-trained model. It is a high-dimensional numerical representation of the visual content of the image. The basis vector of the k-th symbol axis has a dimension of 2048 and is obtained through non-negative matrix decomposition combined with orthogonalization training. It is the standard direction vector corresponding to the symbol semantics. The three basis vectors are pairwise orthogonal and respectively represent the three types of symbol semantics: light and dark, composition lines, and color warmth and coolness.

[0075] Each continuous attribute value is mapped to a corresponding discrete interval to obtain the corresponding label, which is then combined into a visual symbol label tuple, "tag". vis =(Yin Yang tag, Geometric tag, Hue tag).

[0076] The pre-trained label embedding layer transforms discrete label tuples into 512-dimensional visual label embedding vectors. The discrete label tuples consist of three types of discrete labels corresponding to the image: product category, consumption scenario, and visual style. The label embedding layer is pre-trained based on a million-level image annotation dataset in the advertising domain, with the goal of semantic alignment with the CLIPViT-L / 14 visual feature space. Mean pooling is performed on multiple label embeddings of the same image to output a unified 512-dimensional visual label embedding vector, which is consistent with the latent space dimension of the generator and used for subsequent culturally constrained similarity retrieval.

[0077] After generating the text-side semantic constraint list and the visual-side symbol labels, the two types of data are simultaneously input into the cultural fingerprint constraint module and merged to generate the pre-construction semantic guardrail.

[0078] Cultural fingerprint constraint module: retrieves regional cultural taboo vectors and brand symbol vectors that simultaneously match the metaphor word metavectors and visual symbol tag tuples in the semantic constraint list, synthesizes high-dimensional constraint vectors and uses them as semantic guardrails.

[0079] like Figure 5 As shown, it should be specifically noted that in the cultural fingerprint constraint module, the regional cultural taboo vector is retrieved from the pre-built regional cultural taboo sub-database; the brand symbol vector is retrieved from the pre-built brand symbol sub-database.

[0080] In the cultural fingerprint constraint module, the high-dimensional constraint vector is synthesized as follows: the metaphor word vector is weighted and fused and then projected to the same dimension as the generator's latent space to obtain the semantic taboo vector; the embedded vector is retrieved to obtain the visual taboo vector; the semantic taboo vector and the visual taboo vector are weighted and synthesized into a comprehensive taboo vector; the brand symbol vector of the target brand is subtracted from the product of the constraint strength coefficient and the comprehensive taboo vector to obtain the semantic guardrail vector.

[0081] In the cultural fingerprint constraint module, the semantic taboo vector is obtained by projecting the features of taboo words extracted by a pre-trained language model onto the same dimension as the generator's latent space; the visual taboo vector is obtained by projecting the visual features of taboo patterns or symbols extracted onto the same dimension as the generator's latent space; and the brand symbol vector is obtained through multimodal feature fusion training.

[0082] It should be further explained that this module constructs a cultural fingerprint vector library of the same dimension as the generator's latent space, integrates semantic and visual constraints to generate semantic guardrails, and embeds compliance constraints into the generation process in advance to avoid illegal content from the generation path level.

[0083] This module takes a list of semantic constraints, a visual tag embedding vector, a target geographic identifier, and a target brand identifier as input, and outputs a 512-dimensional semantic guardrail vector and a comprehensive taboo vector, which serve as the constraint basis for downstream generation modules. The specific implementation steps are as follows:

[0084] A 512-dimensional cultural fingerprint vector library is pre-built. All vectors are mapped to 512 dimensions consistent with the generator's latent space through a fully connected layer to ensure complete dimensionality matching. This includes two sub-libraries:

[0085] Regional Cultural Taboo Sub-database: Covering 32 major countries / regions worldwide, each region includes a semantic taboo vector set and a visual taboo vector set. The data comes from local advertising regulations, folk culture research reports, and historical cases of illegal advertising. The semantic taboo vectors are obtained by projecting the features extracted from taboo words using RoBERTa, while the visual taboo vectors are obtained by projecting the visual features extracted from taboo patterns and symbols. Each region includes 50-200 taboo vectors, covering three taboo dimensions: religion, folklore, and law.

[0086] Brand Symbol Sub-library: Covering 200+ mainstream commercial brands, each brand corresponds to a 512-dimensional brand symbol vector. The data comes from brand VI specification manuals, historical official advertising materials, and brand official promotional copy. It is obtained through multimodal feature fusion training and represents the brand's unified tone and positive semantic direction.

[0087] To further explain, the RoBERTa feature extraction and projection process involves the following steps: performing Chinese word segmentation on a single taboo word, consistent with the preceding text, and inputting the resulting word sequence into the RoBERTa-large Chinese pre-trained model; extracting the hidden state outputs of all word positions in the last layer of the model, performing mean pooling to obtain a 1024-dimensional sentence-level initial feature vector; linearly transforming the 1024-dimensional initial feature vector into a 512-dimensional vector through a fully connected linear layer; and performing L2 normalization on the projected 512-dimensional vector to eliminate differences in vector magnitude and ensure consistency in subsequent cosine similarity calculations.

[0088] To further explain, the multimodal feature fusion training splits the modality for three types of data sources. For text, the RoBERTa-large model is used, extracting sentence-level mean pooling features with a dimension of 1024. For visual features, the CLIP ViT-L / 14 model is used, extracting global pooling output features with a dimension of 768. The two types of single-modal features are projected to a unified 512 dimension through independent fully connected linear layers, consistent with the dimensionality standards of the cultural fingerprint vector library and the generator's latent space. Then, a weighted fusion is performed with a weight of 0.4 for text and 0.6 for visual features to obtain the initial brand vector. A triplet of "anchor sample - positive sample from the same brand - negative sample from different brands" is constructed. The projection layer parameters are optimized using a triplet loss function, making multimodal features from the same brand closer together in the vector space and features from different brands farther apart, thus enhancing the brand discriminative power and semantic representation ability of the vectors. After training, L2 normalization is performed on the final vector for each brand.

[0089] For the current generation task, matching taboo vectors are retrieved from both semantic and visual dimensions:

[0090] For each metaphorical lexical unit in the semantic constraint list, the corresponding 1024-dimensional three-layer vector is weighted and fused according to the weights of "surface layer 0.3, metaphor 0.4, and deep layer 0.3", and then projected to 512 dimensions through a fully connected layer. Cosine similarity is calculated in the semantic taboo vector set of the target region, and the average of the top 3 similarity vectors is taken to obtain the semantic taboo vector t. sem ∈R 512 .

[0091] For the input 512-dimensional visual label embedding vector, cosine similarity is calculated in the visual taboo vector set of the target region. The average of the top 3 similarity vectors is taken to obtain the visual taboo vector t. vis ∈R 512 .

[0092] According to the formula Synthesize the taboo vector, where β is the semantic weight coefficient, which is set to 0.5 by default. After synthesis, perform L2 normalization.

[0093] Retrieve the 512-dimensional brand symbol vector b corresponding to the target brand and perform L2 normalization.

[0094] According to the formula Calculate the semantic guardrail vector, where α is the constraint strength coefficient, which is 0.8 by default; perform L2 normalization on g to obtain the final 512-dimensional semantic guardrail vector.

[0095] Preset forbidden angle threshold The taboo angle threshold is determined by searching the grid of the ad violation generation test set. It is the optimal value that combines the effect of violation blocking and the creativity of generation. If the angle between the generated direction vector and the comprehensive taboo vector t is less than 30°, it is judged as a high-risk violation direction and must be forcibly blocked during the generation process.

[0096] Cross-modal creative generation module: After aligning the text word vectors and image frame feature vectors in time, concatenate them into joint features. Input the joint features and semantic guardrails into the latent space of the generator. In the latent space, perform dimensional pruning on the generation path according to the semantic guardrails: only retain the dimensional components whose dot product with the semantic guardrail vector is greater than zero, and output at least one advertising creative sample that integrates text and image and is constrained by the semantic guardrails.

[0097] It should be specifically noted that in the cross-modal creative generation module, the joint feature is generated as follows: the three-layer semantic vectors are weighted and fused and then projected onto the generator's latent space dimension to obtain text features; the visual features are projected onto the generator's latent space dimension to obtain visual features; and the text features and visual features are fused to output a joint feature vector.

[0098] It should be further explained that the cross-modal creative generation module performs dimensional pruning and taboo direction decay in the generator's latent space, restricting the generation path from the bottom layer of the latent space to achieve creative generation under strong constraints. The inputs are the anchor pair set, the semantic constraint list, the visual symbol label tuple, the semantic guardrail vector g, and the comprehensive taboo vector t. The output is a constrained multimodal advertising creative sample. The specific implementation steps are as follows:

[0099] Iterate through each word-frame pair (i,j) in the anchor pair set and generate joint features according to a unified rule:

[0100] The 1024-dimensional fused semantic vector of the i-th word (0.3 for the surface layer + 0.4 for the metaphor layer + 0.3 for the deep layer) is projected onto a 512-dimensional text feature through a dedicated fully connected layer on the text side. The dedicated fully connected layer is a single-layer linear transformation structure with an input dimension of 1024 and an output dimension of 512. Its parameters are trained independently of the visual projection layer and are used to map the text semantic space to the generator latent space to achieve dimensional alignment.

[0101] Take the 768-dimensional CLIP visual features of the j-th frame and project them into 512-dimensional visual features through a dedicated fully connected layer;

[0102] After adding the text features and visual features element by element, a 512-dimensional joint feature vector is output through a single linear fusion layer, which is completely consistent with the latent space dimension of the generator.

[0103] The joint features of all anchor pairs are sorted in ascending order of timestamp to obtain the temporal joint feature sequence, which is used as the conditional embedding input of the generator.

[0104] We employ a latent diffusion model (LDM) for the advertising domain, fine-tuned based on Stable Diffusion v1.5, as the base generator. The model is based on the open-source Stable Diffusion v1.5 and uses a high-quality commercial advertising image dataset of 120,000 images covering 8 major advertising categories. We adopt a low-rank adaptation (LoRA) parameter fine-tuning strategy, only fine-tuning the attention layer parameters of the denoising UNet network, and freezing the backbone parameters of the VAE encoder / decoder and text encoder. With latent space image reconstruction accuracy and text-visual semantic alignment as joint optimization objectives, the latent space dimension of the model is fixed at 512 after training to adapt to the visual style and semantic requirements of advertising creative generation.

[0105] The latent space dimension is fixed at 512, the denoising steps are 50 by default, and the DDIM sampler is used. The temporal joint feature sequence is used as a conditional embedding and input into the cross attention layer of the diffusion model to guide the generated content to align with the input temporal sequence.

[0106] After each round of noise prediction is completed and before the latent vector is updated, a two-level pruning operation is performed to ensure that constraints apply throughout the entire generation process. Specifically:

[0107] Forward dimension preservation: Let the current latent vector be z∈R 512 Calculate the dot product dimension by dimension ;like Then let Only the dimension components that are consistent with the positive direction of the semantic guardrail are retained to ensure that the generation direction conforms to the brand's positive tone and avoids deviating from the brand style.

[0108] Taboo direction decay: Calculate the cosine angle between the current latent vector z and the combined taboo vector t. ;like Then calculate the projection component of the latent vector in the tabu direction. Multiply the projected components by an attenuation coefficient γ=0.01, and update the latent vector as follows: This forces the compression of feature intensity in forbidden directions, preventing the generator from exploring in unconventional directions.

[0109] After completing all 50 denoising iterations, the constrained latent vector sequence is obtained. The latent vector sequence is then upsampled and restored to a 512×512 resolution advertising visual image sequence using an LDM decoder. At the same time, the corresponding advertising copy is generated using a GPT-2 text decoder based on the temporal joint feature sequence. Finally, three sets of differentiated multimodal advertising creative samples are output, each set including temporally matched visual content and copy content.

[0110] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0111] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An advertising creative matching system based on multimodal content generation, characterized in that, include: Cross-modal time axis anchoring module: Receives text sequence and image sequence, establishes a bidirectional mapping matrix based on the time step of the word in the text sequence and the timestamp of the frame in the image sequence, sets the word-frame pair with confidence below a preset threshold to zero, and obtains the corresponding anchoring pair set; Semantic layer stripping module: Decomposes the text lexical units in the anchor pair set into surface meaning layer, metaphorical meaning layer and deep semantic layer, calculates the mutual information difference between the three layers of vectors, marks the lexical units with mutual information difference exceeding the threshold as metaphorical lexical units, and generates a list of semantic constraints including the position of metaphorical lexical units and the corresponding three layers of vectors. Visual symbol axis indexing module: For image frames in the anchor pair set, calculate symbol attribute values ​​along the symbol axis, map the values ​​to the corresponding discrete intervals, and generate visual symbol label tuples; Cultural fingerprint constraint module: retrieves regional cultural taboo vectors and brand symbol vectors that simultaneously match the metaphor word metavectors and visual symbol tag tuples in the semantic constraint list, synthesizes high-dimensional constraint vectors and uses them as semantic guardrails; Cross-modal creative generation module: After aligning the text word vectors and image frame feature vectors in time, concatenate them into joint features. Input the joint features and semantic guardrails into the latent space of the generator. In the latent space, perform dimensional pruning on the generation path according to the semantic guardrails: only retain the dimensional components whose dot product with the semantic guardrail vector is greater than zero, and output at least one advertising creative sample that integrates text and image and is constrained by the semantic guardrails.

2. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the cross-modal time axis anchoring module, the time step of the word in the text sequence is calculated based on the standard Chinese broadcast speech rate, and each word corresponds to a base timestamp; the timestamp of the image frame is the display timestamp of each frame in the original video or an incremental timestamp allocated according to the input order; the row index of the bidirectional mapping matrix corresponds to the position of the text word, the column index corresponds to the position of the image frame, and the matrix element value is the cosine similarity between the corresponding word feature and the image frame feature.

3. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the semantic layer stripping module, the surface meaning layer extracts the literal meaning vectors of word elements through a Chinese pre-trained language model; the metaphorical meaning layer generates escape vectors based on an advertising domain-specific metaphor dictionary, and unregistered word elements obtain approximate escape vectors by retrieving the labeled metaphorical word elements with the highest semantic similarity through a general metaphor embedding model; the deep semantic layer generates hypersensory vectors based on a knowledge graph in the advertising and marketing domain.

4. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the semantic layer stripping module, the mutual information difference between the three layers of vectors is obtained by calculating the mutual information between the surface meaning layer and the metaphorical meaning layer, and the mutual information between the surface meaning layer and the deep semantic layer respectively using the Gaussian kernel density estimation method, and then taking the absolute value of the difference; when the mutual information difference exceeds the preset threshold, the corresponding word is determined to be a metaphorical word.

5. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the visual symbol axis indexing module, the symbol axis is a set of axes pre-extracted from the historical advertising creative dataset using non-negative matrix decomposition, including at least the yin-yang axis, the geometric axis, and the hue axis; Each axis has an independent discrete value range, and the values ​​of the image frame on each axis are combined into a visual symbol label tuple after discrete mapping.

6. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the visual symbol axis indexing module, the pre-extraction of symbol axes includes: extracting visual features from historical advertising images and performing non-negative translation processing to form a feature matrix; setting the number of basis vectors and then iteratively updating the basis matrix and coefficient matrix through non-negative matrix decomposition; performing orthogonalization processing on the basis matrix to obtain multiple mutually orthogonal symbol axes; manually semantically summarizing the coefficient values ​​of each axis to label the axis type; and using the equal-frequency discretization method to divide the discrete interval.

7. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the cultural fingerprint constraint module, the regional cultural taboo vector is retrieved from the pre-built regional cultural taboo sub-database; the brand symbol vector is retrieved from the pre-built brand symbol sub-database.

8. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the cultural fingerprint constraint module, the high-dimensional constraint vector is synthesized as follows: the metaphor word vector is weighted and fused and then projected to the same dimension as the generator's latent space to obtain the semantic taboo vector; the embedded vector is retrieved to obtain the visual taboo vector; the semantic taboo vector and the visual taboo vector are weighted and synthesized into a comprehensive taboo vector; the brand symbol vector of the target brand is subtracted from the product of the constraint strength coefficient and the comprehensive taboo vector to obtain the semantic guardrail vector.

9. The advertising creative matching system based on multimodal content generation according to claim 8, characterized in that: In the cultural fingerprint constraint module, the semantic taboo vector is obtained by projecting the features of taboo words extracted by a pre-trained language model onto the same dimension as the generator's latent space; the visual taboo vector is obtained by projecting the visual features of taboo patterns or symbols extracted onto the same dimension as the generator's latent space; and the brand symbol vector is obtained through multimodal feature fusion training.

10. The advertising creative matching system based on multimodal content generation according to claim 1, characterized in that: In the cross-modal creative generation module, the joint feature is generated as follows: the three-layer semantic vectors are weighted and fused and then projected onto the generator's latent space dimension to obtain text features; the visual features are projected onto the generator's latent space dimension to obtain visual features; and the text features and visual features are fused to output a joint feature vector.

Images