Advertisement content generation method based on multi-modal fusion

Abstract

The invention discloses a multi-modal fusion-based advertisement content generation method, which comprises the following steps of: firstly, extracting unified features of original multi-modal data, and performing semantic fusion and similarity enhancement to obtain fusion representation; and outputting the initial creativity by the generative network in combination with user feedback, calculating a matching degree, generating a market adaptation adjustment vector, performing iterative optimization on the creativity, and performing optimization according to a prediction effect score, thereby finally generating high-precision and personalized advertisement content. According to the invention, the personalized level, market suitability and generation efficiency of the advertisement content are obviously improved, the trial and error cost is reduced, and efficient and intelligent advertisement creation is realized.

Description

Technical Field

[0001] This invention belongs to the field of advertising generation technology, and in particular relates to a method for generating advertising content based on multimodal fusion. Background Technology

[0002] Content creation, a core component of digital marketing, is crucial for enhancing brand influence and user engagement. As consumer behavior becomes increasingly diverse, advertising needs to precisely align with the preferences and needs of the target audience to stand out in a highly competitive market. Traditional ad generation methods often rely on single-modal data analysis or human experience, making it difficult to fully capture the complex interplay between user emotions, product characteristics, and brand tone. When faced with multi-dimensional information integration, this approach often suffers from a lack of systematic understanding, resulting in insufficiently targeted creative output. Especially in dynamic market environments, existing solutions struggle to effectively integrate multi-modal information such as text, images, audio, and video, limiting the innovation and appeal of ad content.

[0003] The core challenge lies in achieving deep semantic fusion and personalized creative generation from multimodal data. First, feature extraction and integration of multimodal data face technical bottlenecks. Information from different modalities is heterogeneous; for example, the visual features of images and the semantic information of text differ significantly in their expression, making effective fusion difficult within a unified content representation space. This results in poor cross-modal consistency in advertising content. For instance, a brand's advertisement might emphasize a high-end feel in its copywriting, but fail to convey a consistent emotional atmosphere through color schemes or music choices. Second, personalized creative generation based on fused content requires dynamic optimization by combining historical data, user feedback, and market trends. Existing systems often cannot accurately predict the effects of different modal combinations. For example, in advertisements designed for young users, it's difficult to determine which color combinations and music styles will maximize attention. This lack of predictive ability leaves creators without data-driven guidance in creative decisions, increasing trial-and-error costs.

[0004] Therefore, how to achieve semantic fusion of multimodal data in a unified content representation space, and based on this, provide personalized creative suggestions and effect predictions, has become a key issue in the field of advertising content generation. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an advertising content generation method based on multimodal fusion, thereby resolving the issues present in the prior art.

[0006] To achieve the above objectives, the present invention provides a method for generating advertising content based on multimodal fusion, comprising: The process involves: acquiring raw multimodal data related to the advertising content; extracting features from the raw multimodal data to obtain a unified feature vector set; performing semantic fusion on the unified feature vectors to obtain a semantic representation matrix; performing similarity judgment on the semantic representation matrix; enhancing the semantic representation matrix based on the similarity judgment result to obtain an enhanced fusion representation; acquiring user feedback data; generating a preliminary personalized creative plan through a generative network based on the enhanced fusion representation and user feedback data; evaluating the preliminary personalized creative plan to obtain a preliminary matching degree; acquiring indicators based on the preliminary matching degree and the preliminary personalized creative plan to obtain a market adaptation adjustment vector; adjusting the preliminary personalized creative plan based on the market adaptation adjustment vector; evaluating the adjusted preliminary personalized creative plan to obtain a predicted performance score; and optimizing the adjusted preliminary personalized creative plan based on the predicted performance score to obtain optimized advertising content.

[0007] Optionally, the process of feature extraction from the original multimodal data includes: Feature vectors of the original multimodal data are obtained by extracting features from the original multimodal data using a feature extractor; wherein the original data includes text data, image data, audio data, and video data. The feature vectors of the multimodal data are dimensionally determined, and based on the dimension determination result, the feature vectors of the multimodal data are dimensionally reduced to obtain a unified feature vector set.

[0008] Optionally, the process of obtaining the semantic representation matrix includes: The unified feature vector set is encoded using the Transformer model to obtain encoded text semantic vectors and image visual vectors. A multi-head attention mechanism is then used to perform cross-modal fusion of the encoded text semantic vectors and image visual vectors to obtain a fused feature vector. The fused feature vector is evaluated, and based on the evaluation result, a new fused feature vector is generated to obtain an optimized fused feature vector. A cross-modal semantic representation matrix is ​​constructed based on the optimized fused feature vector. The cross-modal semantic representation matrix is ​​then decomposed to obtain a low-dimensional semantic representation matrix. The low-dimensional semantic representation matrix is ​​then classified and semantic consistency is evaluated to obtain the semantic representation matrix.

[0009] Optionally, the process of obtaining the enhanced fusion representation includes: Initial semantic features are obtained from the semantic representation matrix, the association strength of the initial semantic features is extracted, the association strength is judged, the fusion weight parameters are updated according to the judgment result, and an optimized weight set is obtained. The semantic representation matrix is ​​weighted and fused through the optimized weight set to obtain a preliminary fused representation. The preliminary fused representation is similar to the original representation, the similarity evaluation result is judged, and an enhanced fused representation is generated according to the judgment result.

[0010] Optional, the process of generating preliminary personalized creative solutions includes: The process involves acquiring historical user feedback data, cleaning the data, extracting user behavior patterns and preference features to create user profiles, generating multi-dimensional feature vectors based on these profiles to obtain enhanced representations, processing the enhanced fusion representations and enhanced representations using a generative adversarial network, generating preliminary personalized creative solutions, and matching these solutions with user profiles to obtain a preliminary matching degree.

[0011] Optionally, the process of obtaining the market adaptation adjustment vector includes: Based on the initial matching degree of the preliminary personalized creative solutions, an initial dataset is generated. The initial dataset is then filtered to obtain a set of high matching degree indicators. Trend extraction is performed on the set of high matching degree indicators to obtain market trend features. The market trend features are then judged. Based on the judgment results, an initial adjustment vector is generated according to the market trend features. The initial adjustment vector is then iteratively optimized to obtain an optimized adjustment vector. Based on the optimized adjustment vector, a market-adaptive adjustment vector is obtained.

[0012] Optionally, the process of obtaining the predicted performance score includes: Based on the market adaptation adjustment vector, user behavior features and market trend features are obtained. Feature extraction and dimensionality reduction are performed on the user behavior features and market trend features to obtain a simplified feature set. Gradient descent optimization is performed on the simplified feature set to obtain an optimized adjustment vector. The optimized adjustment vector is predicted through a long short-term memory network to obtain a prediction performance score.

[0013] Optional, optimized ad content acquisition processes include: If the predicted performance score is higher than the preset threshold, the preliminary personalized creative solution with the highest score is selected as the final ad content. If the predicted performance score is lower than the preset threshold, a backtracking optimization mechanism is triggered, and the fusion representation step is re-executed. This step extracts features from multi-source data and generates a data fusion representation using principal component analysis. Based on the data fusion representation, a generative adversarial network (GAN) algorithm is used to optimize and generate a new version of the ad content. For the new version, a preset performance evaluation standard is used to calculate the predicted performance score. If the predicted performance score of the new version is still lower than the preset threshold, the optimization iteration count is updated, and the fusion representation step is re-executed. Finally, a threshold judgment logic compares the final predicted performance score with the preset threshold, outputting the final ad content that meets the conditions.

[0014] On the other hand, the present invention also provides an advertising content generation system based on multimodal fusion for performing the above-described method.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention discloses a technical solution for multimodal advertising content generation and optimization. Addressing the problems of insufficient multimodal data fusion, low personalization, poor market adaptability, and high trial-and-error costs in traditional advertising creation, this invention generates highly efficient personalized advertising content through multimodal data collection, feature extraction, and semantic fusion, combined with market trends and user feedback. The invention acquires multi-source inputs such as text, images, audio, and video through a pre-set multimodal data collection module, generates a unified feature vector set using a feature extractor, and then performs cross-modal semantic fusion using a Transformer model to generate a consistent semantic representation matrix. If the similarity is insufficient, iterative optimization is used to adjust the weights and enhance the fused representation. Based on historical user feedback, a generative adversarial network is used to generate an initial creative scheme, and the scheme is updated by extracting adaptation vectors from a dynamic market trend database. A long short-term memory network predicts the multimodal combination effect, ensuring that the final advertising content is output after the prediction score reaches a certain level. Simulation tests are used to verify the degree of cost reduction. This invention significantly improves the personalization level, market adaptability, and generation efficiency of advertising content, reduces trial-and-error costs, and achieves efficient and intelligent advertising creation. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an advertising content generation method based on multimodal fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the advertising content generation system based on multimodal fusion according to an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] like Figure 1 As shown, the advertising content generation method based on multimodal fusion in this embodiment may specifically include: S101. Obtain raw input data from text, image, audio and video sources through a preset multimodal data acquisition module, process the features of each modality using a feature extractor, and obtain a unified feature vector set.

[0020] The system acquires raw input data from text, image, audio, and video data sources using a pre-defined multimodal data acquisition module, resulting in a multimodal raw dataset. A feature extractor performs word segmentation and word embedding on the text data in the multimodal raw dataset to generate text feature vectors. The feature extractor also performs convolutional neural network processing on the image data in the multimodal raw dataset to generate image feature vectors. Furthermore, the feature extractor performs short-time Fourier transform processing on the audio data in the multimodal raw dataset to generate audio feature vectors. Finally, the feature extractor performs frame extraction and temporal feature analysis on the video data in the multimodal raw dataset to generate video feature vectors. If the dimension of the unified feature vector set exceeds a preset threshold, principal component analysis (PCA) is used to reduce the dimensionality of the unified feature vector set, resulting in a dimensionality-reduced feature vector set.

[0021] Specifically, multimodal data processing has wide applications in the field of intelligent content analysis, and can integrate multiple data sources such as text, images, audio, and video to improve the comprehensiveness and accuracy of information understanding.

[0022] For example, to acquire text, image, audio, and video sources, web crawlers can be used to collect user comments and related images from social media platforms to form an initial dataset. Text data may include user reviews of a product, image data may consist of product photos or user-uploaded images, and audio and video can be obtained from advertising content for relevant software products or recommended short videos.

[0023] The generation of advertising content can be based on relevant scenario content, such as in a short video recommendation system. Advertising content can be displayed through short video recommendations, and multimodal data processing in such systems can achieve more accurate content collection. Focusing on the short video recommendation scenario, word segmentation and word embedding of text data are fundamental to feature extraction.

[0024] Specifically, suppose a short video platform collects user comments as text data, with comments like "This video is hilarious, highly recommended." The feature extractor first segments the text into words such as "this," "video," "too," and "hilarious." Then, a pre-trained word embedding model (such as BERT) maps each word to a high-dimensional vector, for example, generating a 768-dimensional vector for each word. These vectors capture semantic relationships; for example, "hilarious" and "interesting" are close in the vector space. This processing preserves the semantic information of the text, providing high-quality features for subsequent fusion and helping the recommendation system identify user sentiment. Convolutional neural network processing of image data is used to extract visual features.

[0025] In one embodiment, the cover image of the short video may include a landscape or a person. A convolutional neural network (such as ResNet) processes the image, extracting low-level features such as edges and textures, as well as high-level features such as objects and scenes, ultimately generating a 2048-dimensional image feature vector.

[0026] For example, landscape images might generate vectors containing semantics such as "mountain" and "lake." This feature extraction helps recommendation systems understand the visual themes of videos, such as recognizing that landscape videos are more popular with a certain type of user, thereby optimizing recommendation performance. Short-time Fourier transform processing of audio data is used to analyze sound features.

[0027] For example, the background music in a short video might be pop or classical. The feature extractor converts the audio signal into a spectrogram using a short-time Fourier transform, extracting frequency distribution and rhythmic features to generate a 512-dimensional audio feature vector. Fast-paced music may correspond to high-energy features, making it suitable for recommendations to users who prefer dynamic content. This processing can uncover the emotional atmosphere of the audio, enhancing the personalization of recommendations. Frame extraction and temporal feature analysis of video data capture dynamic information.

[0028] In one possible implementation, the short video is decomposed into an image sequence of 25 frames per second. The temporal relationship between frames is analyzed by a 3D convolutional neural network to generate a 1024-dimensional video feature vector. The video feature vector includes images from different frames and represents the feature vectors of the frame images in the time series.

[0029] For example, the temporal features of dance videos may reflect the rhythm and fluidity of the movements. This feature extraction can identify the dynamic patterns of videos, helping recommendation systems differentiate between action-oriented and narrative videos to meet different user preferences.

[0030] When the dimensionality of the unified feature vector set is too high, principal component analysis is used to reduce dimensionality.

[0031] For example, a 256-dimensional vector can be reduced to 128 dimensions while retaining 95% of the information variance. Dimensionality reduction results in a more compact set of feature vectors, reducing computational complexity while preserving key information, thus improving the real-time performance and accuracy of recommendation systems.

[0032] It's important to note that each step in multimodal processing is closely interconnected, forming a complete technical chain from data acquisition to feature extraction and dimensionality reduction. This method can significantly improve the accuracy and user experience of recommendation systems, meeting personalized needs.

[0033] S102. Apply the Transformer model to perform semantic fusion operation based on the unified feature vector set, integrate text semantics and image visual features, and determine a cross-modal consistent semantic representation matrix.

[0034] A Transformer model is used to encode a unified set of feature vectors, generating encoded text semantic vectors and image visual vectors. A multi-head attention mechanism is then used to perform a cross-modal fusion operation on the encoded text semantic vectors and image visual vectors to obtain a fused feature vector. If the modal consistency of the fused feature vector is lower than a preset threshold, the fusion operation is re-executed by adjusting the attention weights to obtain an optimized fused feature vector. Based on the optimized fused feature vector, a cross-modal semantic representation matrix is ​​constructed, ensuring that the matrix dimension is aligned with the feature vectors. Matrix factorization is used to reduce the dimensionality of the cross-modal semantic representation matrix, resulting in a low-dimensional semantic representation matrix. A clustering algorithm is then used to classify the low-dimensional semantic representation matrix, assessing cross-modal semantic consistency and generating the final semantic representation result.

[0035] The Transformer model encodes a unified set of feature vectors. Text semantic vectors are encoded using the Transformer's encoder, combined with a self-attention mechanism, to enhance context-sensitive semantic representations and generate more globally semantically relevant encoded vectors. Image visual vectors are processed by a visual Transformer to capture the spatial relationships between different regions in the image, generating more structured visual encoded vectors. This encoding method effectively preserves semantic information between modalities, laying the foundation for cross-modal fusion. Audio and video feature vectors are processed by their respective Transformer models to generate corresponding audio text semantic vectors and temporal image visual vectors for the video.

[0036] For example, when performing cross-modal fusion using a multi-head attention mechanism, text semantic vectors and image visual vectors can be input into the same multi-head attention module. The attention mechanism generates a fused feature vector by calculating the correlation between text and image features.

[0037] For example, if a product review mentions "stylish appearance," and the corresponding image displays unique design lines, the attention mechanism will strengthen the connection between the two, generating a fusion vector that reflects the consistency between the text and the image. If the modal consistency of the fusion vector is lower than a preset threshold, such as 0.8, the attention weights are dynamically adjusted to redistribute the weight ratios of text and image features, thus optimizing the fusion effect.

[0038] Specifically, when constructing a cross-modal semantic representation matrix, the fused feature vectors are arranged according to a preset dimension to form a matrix.

[0039] For example, if the fused vector is 512-dimensional, the matrix dimension can be set to 512×N, where N is the number of data samples. Matrix factorization techniques, such as singular value decomposition, preserve the main information and reduce the matrix dimension to 256×N, generating a low-dimensional semantic representation matrix. This dimensionality reduction method reduces computational complexity while preserving core semantic information.

[0040] In one possible implementation, the K-means clustering algorithm is used to classify the low-dimensional semantic representation matrix and determine cross-modal semantic consistency.

[0041] For example, samples in the matrix can be divided into two categories: "positive reviews and aesthetically pleasing images" and "negative reviews and average images." By analyzing the clustering results, a final semantic representation can be generated for subsequent tasks such as sentiment analysis or recommendation systems. This classification method can effectively distinguish semantic relationships between different modalities and improve data processing efficiency.

[0042] S103. If the similarity in the semantic representation matrix is ​​lower than the preset threshold, the fusion weight parameters are adjusted, and an enhanced fusion representation is obtained through iterative optimization.

[0043] Initial semantic features are obtained from the semantic representation matrix, and the correlation strength between features is determined by similarity calculation. If the similarity calculation result is lower than a preset threshold, the fusion weight parameters are updated according to a weight adjustment strategy. The updated fusion weight parameters are processed through an iterative optimization process to obtain an optimized weight set. The semantic representation matrix is ​​weighted and fused using the optimized weight set to generate a preliminary fused representation. The similarity of the preliminary fused representation is evaluated to determine whether it meets the optimization objective function. If the similarity evaluation result does not meet the optimization objective function, the iterative optimization process is repeated to obtain an enhanced fused representation. The final semantic representation matrix is ​​generated from the enhanced fused representation, and the output result is determined.

[0044] For example, in cross-modal semantic fusion scenarios, when obtaining initial semantic features from the semantic representation matrix, a set of feature vectors can be formed by extracting semantic information from text and images. Suppose we are processing a product recommendation system where the text data is the product description, such as "red dress, suitable for summer wear," and the image data is the product image. The initial semantic features can be keyword vectors from the text (such as "red" and "dress") and color and shape feature vectors from the image. After obtaining these features, the semantic relevance between text and image is measured by calculating the correlation strength between the features. One possible implementation uses cosine similarity to evaluate the angle between the text vector and the image vector, determining whether they are semantically aligned.

[0045] For example, the word "red" in the text description matches the main color features in the image with a similarity of 0.9, which is higher than the preset threshold of 0.8, indicating a strong initial association.

[0046] In one embodiment, if the similarity is below a preset threshold, such as only 0.6, the fusion weight parameters need to be updated using a weight adjustment strategy. The weight adjustment can be based on the gradient descent principle, dynamically increasing the weights of highly relevant features and reducing the influence of irrelevant features.

[0047] For example, the initial weights might be evenly distributed, but if it's found that the image's texture features are irrelevant to the text, the weight of texture features is reduced to 0.1, while the weight of color features is increased to 0.7. The updated weight set is further processed through an iterative optimization process to generate a weight set that better conforms to semantic consistency. The iterative optimization can set a maximum number of iterations, such as 10, evaluating the similarity improvement after weight adjustment each time, until stability is reached.

[0048] Specifically, when using the optimized weight set to perform weighted fusion of the semantic representation matrix, the text and image feature vectors can be summed according to their weights to generate a preliminary fused representation.

[0049] For example, if the text vector is [0.5, 0.3, 0.2], the image vector is [0.4, 0.6, 0.1], and the weight set is [0.7, 0.2, 0.1], the resulting fused vector is used for subsequent evaluation. The initial fused representation needs to be evaluated for similarity to determine if it meets the optimization objective function. For example, the objective function might require the Euclidean distance between the fused representation and the ideal semantic vector to be less than 0.5. If this requirement is not met, such as a distance of 0.8, the iteration process is repeated, adjusting the weights or introducing more contextual features until an enhanced fused representation is generated.

[0050] In one embodiment, the enhanced fusion representation is used to generate the final semantic representation matrix, ensuring that the output is dimensionally aligned with the input features.

[0051] For example, the fusion representation can be mapped to a 10×10 matrix, where each row represents a semantic dimension, used for classification or ranking tasks in subsequent recommendation systems. This method, through multiple rounds of optimization, ensures that the semantic information of text and images is fully fused in the matrix, improving the semantic consistency of the recommendation system.

[0052] It should be noted that the combination of weight adjustment and iterative optimization can effectively reduce semantic bias between modalities, making the final matrix more accurately reflect cross-modal information.

[0053] S104. Obtain the enhanced fusion representation and combine it with historical user feedback data. Use generative adversarial networks to generate preliminary personalized creative solutions and determine the preliminary matching degree of the creative solutions.

[0054] Historical user feedback data is acquired and cleaned to extract user behavior patterns and preference features, resulting in a user profile. Based on the user profile, a pre-defined enhanced representation model is fused to generate a multi-dimensional feature vector, yielding an enhanced representation. A generative adversarial network (GAN) is then used as input to generate an initial personalized creative solution. Using pre-defined evaluation metrics, the matching degree between the initial personalized creative solution and the user profile is calculated, resulting in a matching evaluation result. If the matching evaluation result is lower than a pre-defined threshold, the GAN parameters are adjusted, and a new personalized creative solution is generated.

[0055] For example, extracting behavioral patterns and preference characteristics from historical user feedback data can be achieved by analyzing user interaction records. Suppose a social media platform collects user browsing, likes, and comments data from the past 30 days. The data cleaning process includes removing invalid data, such as duplicate clicks or abnormally high-frequency actions. After cleaning, the time distribution of users browsing specific content is extracted; for example, users tend to browse short videos rather than long articles, with an average browsing time of 15 seconds, and a preference for entertainment content accounting for 60%. Through cluster analysis, user behavior patterns are identified, such as "nighttime active" users being concentrated between 22:00 and 02:00, and preferring light and humorous content. These patterns are further refined into user profiles, including features such as age, interest tags, and active time periods.

[0056] In one possible implementation, when generating multidimensional feature vectors by fusing a pre-defined enhanced representation model, a deep learning model can be used to process user profiles.

[0057] For example, interest tags and behavioral patterns from user profiles can be input into a pre-trained semantic embedding model to generate a 128-dimensional feature vector.

[0058] Preferably, during fusion, contextual information, such as popular trends in the user's region, can be incorporated to enhance the semantic expressiveness of the feature vector.

[0059] For example, if a user profile shows a preference for music-related content, the model can increase the weights of relevant features based on recent popular music festival events to generate more targeted multidimensional vectors.

[0060] Specifically, when using generative adversarial networks to generate initial personalized creative solutions, the enhanced representation is used as input, and the generator attempts to produce content ideas that match the user's preferences.

[0061] For example, based on the aforementioned user profile and enhanced representation fusion, the generator might output a 15-second short video script, featuring a humorous music mashup, catering to users' nighttime viewing habits. The discriminator then evaluates whether the creative concept aligns with user preferences. Suppose the generator initially outputs a script leaning towards educational content; the discriminator identifies a low match with the user profile and triggers parameter adjustments.

[0062] For example, when evaluating the matching degree of an initial personalized creative solution, multi-dimensional indicators can be set, including content relevance, predicted user interaction value, and creative novelty. Assuming a preset threshold of 0.8, the initial creative solution's matching degree is 0.6, which does not meet the threshold. Based on the evaluation results, the system adjusts the loss function of the generative adversarial network, increasing the weight of user preference features, such as increasing the probability of generating entertainment content. After regeneration, the creative solution becomes a short music-related video, and the matching degree improves to 0.85, meeting the threshold requirement.

[0063] In one possible implementation, if the matching degree is still below the threshold, it can be further optimized through a feedback loop.

[0064] For example, by analyzing users' recent actual interaction data with similar creative ideas, the generative adversarial network's generation strategy can be adjusted, such as increasing the proportion of humorous elements or shortening the video length to less than 10 seconds, to better suit user habits.

[0065] Preferably, the system can introduce diverse generation paths and try different creative solutions, such as animation or live-action montage, to ensure the diversity of output while maintaining a high degree of relevance to the user profile.

[0066] It's important to note that dynamic updates to user profiles are crucial. For example, if a user recently starts paying attention to fitness-related content, the system needs to promptly incorporate this new preference into the profile and adjust the feature vectors and generation strategy accordingly.

[0067] For example, the generator can output a mashup of fitness music videos, combined with user entertainment preferences, to create a short 15-second humorous fitness recommendation ad. This dynamic adjustment ensures the timeliness and personalization of the creative solutions.

[0068] S105. For creative solutions with a high initial matching degree, relevant indicators are extracted from the dynamic market trend database to obtain the market adaptation adjustment vector.

[0069] Matching indices for creative solutions are retrieved from a dynamic market trend database to generate an initial dataset. This dataset is then filtered through index analysis to obtain a set of high-matching indices. Trend extraction techniques are used to process this set of high-matching indices to determine market trend characteristics. If these market trend characteristics match a preset adaptation threshold, a preliminary adjustment vector is generated based on these characteristics. This preliminary adjustment vector is iteratively processed using a solution optimization algorithm to obtain an optimized adjustment vector. Finally, the optimized adjustment vector is compared with market adaptation requirements to determine the final adaptation vector.

[0070] For example, when obtaining matching metrics for creative solutions from a dynamic market trend database, an initial dataset can be formed by crawling user browsing records and purchase data from e-commerce platforms in real time. Suppose an e-commerce platform targets personalized recommendation creatives, and its database contains metrics such as user click-through rate, purchase conversion rate, and browsing time. The initial dataset might include 10 million user interaction records, covering browsing preferences and price sensitivity for clothing products. During data cleaning, invalid records such as duplicate clicks or abnormal data are removed to ensure data quality.

[0071] In one possible implementation, cluster analysis can be used when screening a set of highly matching indicators.

[0072] For example, based on the K-means algorithm, user behavior data is grouped by click-through rate (CTR) and conversion rate. A set of metrics with a CTR higher than 80% and a conversion rate higher than 5% is selected, yielding approximately 100,000 highly relevant data points. This data reflects users' high acceptance of specific creative solutions, such as click preferences for personalized promotional banners.

[0073] Specifically, trend extraction techniques can be achieved through time series analysis.

[0074] For example, analyzing high-relevance data from the past six months can extract market trend characteristics, such as users preferring minimalist clothing recommendation pages or showing a preference for limited-time discounts during promotional activities. Assuming we find that 60% of users click on limited-time discount-related creatives during promotional activities, this trend characteristic can be defined as "limited-time promotion preference."

[0075] For example, when a market trend characteristic matches a preset adaptation threshold, the threshold can be set to "trend coverage exceeds 50%". If the "limited-time promotion preference" coverage reaches 60%, an initial adjustment vector is generated. The adjustment vector may include weights for creative elements, such as "increase discount tag visibility by 20%" or "increase recommendation frequency by 10%".

[0076] In one possible implementation, the scheme optimization algorithm can use a genetic algorithm to iteratively process the initial adjustment vector.

[0077] For example, by simulating natural selection, multiple rounds of optimization can be performed by adjusting the parameter combinations of the vector (such as discount tag color and recommendation frequency) to generate an optimized vector. The optimized vector might increase the visibility of the discount tag by 30% and adjust the recommendation frequency to 15%, thus better aligning with user preferences.

[0078] Specifically, when comparing and optimizing the adjustment vectors to meet market adaptation requirements, A / B testing can be used for verification.

[0079] For example, distribute the optimized creative to 100,000 users and observe the improvement in click-through rate. If the click-through rate of the optimized creative increases from 80% to 85%, it indicates that the final adaptation vector is effective.

[0080] It should be noted that this comparison method ensures that creative solutions are highly aligned with market trends and optimizes the user experience.

[0081] For example, the above process can be applied to personalized recommendation scenarios. Suppose a user recently browsed sneakers; the system generates a recommendation page containing limited-time discounts based on trend characteristics, significantly increasing the user's willingness to buy. This approach ensures the accuracy and market relevance of the recommended content by dynamically adjusting the creative scheme.

[0082] S106. Update creative solutions by adjusting vectors through market adaptation, apply long short-term memory networks to predict the effects of multimodal combinations, and determine the prediction effect score.

[0083] Market-adaptive data is acquired, and user behavior and market trend features are extracted from multi-source datasets to generate an initial feature set. The initial feature set is processed using feature extraction techniques, and dimensionality reduction is achieved using principal component analysis to obtain a simplified feature set. Based on the simplified feature set, adjustment vectors are constructed, and gradient descent is applied to optimize the vector parameters, generating an optimized adjustment vector. A long short-term memory network is used to train the optimized adjustment vector and multimodal data to generate a multimodal combined prediction model. The multimodal combined prediction model processes the input data to generate prediction results of the multimodal combined effect. If the prediction results meet the conditions after comparison with preset thresholds, the prediction results are weighted to obtain an effect score. In one possible implementation, acquiring market-adaptive data is the starting point for extracting user behavior features and market trend features from multi-source datasets.

[0084] For example, data can be extracted from user browsing history, purchase history, and social media interaction data on e-commerce platforms to generate an initial feature set. User behavior features may include user click frequency, dwell time, and purchase conversion rate, while market trend features cover product popularity and competitor price changes. Suppose an e-commerce platform observes that the number of views for a certain type of product has increased by 20% in the past 30 days, while the purchase conversion rate is only 2%. This data constitutes the initial feature set. This multi-source data collection can comprehensively reflect market dynamics and user preferences, providing a rich foundation for subsequent analysis.

[0085] Specifically, when feature extraction techniques process the initial feature set, principal component analysis is used for dimensionality reduction.

[0086] For example, the initial feature set might contain hundreds of features, such as user age, purchase time, and product category. Principal component analysis (PCA) can compress these features into a few principal components, such as user purchase intent and market trends, while retaining over 90% of the data variance. Hypothetical analysis reveals a strong correlation between user purchase intent and price sensitivity to products and promotional activities. This dimensionality reduction effectively reduces data redundancy and improves the efficiency of subsequent modeling.

[0087] In one embodiment, an adjustment vector is constructed based on a reduced feature set.

[0088] For example, for a smart home product, a simplified feature set might show that users are price-sensitive and prefer promotional activities. The adjustment vector could include two dimensions: "reducing the price by 10%" and "increasing the frequency of promotions." When optimizing the vector parameters using the gradient descent algorithm, assuming an initial learning rate of 0.01, the optimal combination of price reduction and promotion frequency is found through multiple iterations. This optimization method ensures that the adjusted vector better reflects actual market demand.

[0089] For example, a Long Short-Term Memory (LSTM) network can be used to train optimized adjustment vectors and multimodal data to generate a predictive model. Multimodal data might include textual data from user reviews, visual features of product images, and time series data from sales. Suppose that during training, a high frequency of the keyword "ease of use" in user reviews is found to be correlated with sales growth; the model will then strengthen the weight of this feature. LSM networks can capture long-term dependencies in time series data, such as the impact of promotional activities on sales over the next two weeks, thus generating a more accurate predictive model.

[0090] Specifically, when processing input data, the multimodal combined prediction model generates prediction results of the multimodal combined effect.

[0091] For example, the model predicts that a certain promotional campaign mix will increase sales by 15% and improve user satisfaction scores by 0.5 points over the next 7 days. If the prediction results meet the conditions after comparison with preset thresholds (such as a 10% increase in sales), the results are weighted.

[0092] For example, if sales growth has a weight of 0.6 and user satisfaction has a weight of 0.4, the calculated performance score is 0.6 × 15 + 0.4 × 0.5 = 9.2. This weighted approach can comprehensively evaluate market performance across different dimensions, providing a more comprehensive basis for decision-making.

[0093] In one possible implementation, the generation of performance scores can provide e-commerce platforms with clear optimization directions.

[0094] For example, based on a high score of 9.2, the platform can prioritize price adjustments and promotional mix strategies. This approach, through multi-dimensional data analysis and model training, ensures that the adjustment plans are highly aligned with market trends, providing reliable support for business decision-making.

[0095] S107. If the predicted performance score exceeds the threshold, output the final ad content version; otherwise, backtrack to the fusion representation step to re-optimize and obtain optimized ad content.

[0096] If the predicted performance score is higher than the preset threshold, the highest-scoring version is retrieved from the ad content version library and designated as the final ad content. If the predicted performance score is lower than the preset threshold, evaluation metrics are retrieved from the performance evaluation criteria, triggering a backtracking optimization mechanism and re-executing the fusion representation step. Through the fusion representation step, features are extracted from multi-source data, and principal component analysis is used to generate a data fusion representation. Based on the data fusion representation, a generative adversarial network algorithm is used to optimize and generate a new version of the ad content. For the new version of the content, the preset performance evaluation criteria are used to calculate the predicted performance score. If the predicted performance score of the new version is still lower than the preset threshold, the optimization iteration count is updated, and the fusion representation step is re-executed. Through threshold judgment logic, the final predicted performance score is compared with the preset threshold, and the final ad content that meets the conditions is output.

[0097] For example, in ad content optimization scenarios, the predicted performance score directly determines the final ad content selection. Suppose an e-commerce platform needs to select ad content for new product promotion, with a preset threshold of 80 points. If the predicted performance score is 85 points, higher than the threshold, the highest-scoring version is extracted from the ad content version library. For instance, a short video ad displaying "New Product Launch, Limited-Time Discount" is directly selected as the final content. This method ensures rapid identification of high-performing content, improving campaign efficiency.

[0098] Specifically, if the predicted performance score is 75, which is below the threshold, a backtracking optimization mechanism is triggered. Backtracking optimization obtains metrics from the performance evaluation criteria, such as click-through rate and conversion rate, and re-enters the fusion representation step. The fusion representation step extracts features from multi-source data (such as user browsing history, purchase preferences, and market competitor analysis). Principal component analysis (PCA) algorithms reduce the dimensionality of high-dimensional features, generating a simplified data fusion representation.

[0099] For example, two key features can be extracted from user behavior data: "preference for fashion items" and "attention to discount information." This dimensionality reduction approach preserves key information and reduces the complexity of subsequent processing.

[0100] In one embodiment, a generative adversarial network (GAN) is used to optimize ad content based on data fusion representation. The generator attempts to generate new ad copy, such as "Trendy New Products, Limited-Time Offer," while the discriminator evaluates its match with user preferences. Assuming that after generating the new ad version, the performance evaluation shows a predicted click-through rate of 3.5%, higher than the old version's 2.8%, the new version enters the performance evaluation phase, calculating a new predicted performance score. If the new score is 82, meeting the threshold, it is output as the final ad content; if it is still 78, below the threshold, the iteration count is updated, such as from the first iteration to the second, and the fusion representation step is executed again.

[0101] For example, performance evaluation criteria might include metrics such as user engagement rate and dwell time. Suppose an ad's predicted engagement rate is 4% and dwell time is 30 seconds, an improvement over the old version's 3% and 25 seconds, indicating the new content is more engaging for users. This multi-dimensional evaluation ensures comprehensive content optimization.

[0102] It should be noted that an upper limit should be set for the number of iterations, such as a maximum of 3 iterations, to avoid wasting resources. If the final score still does not meet the target, a second-best version can be selected from the repository as a compromise.

[0103] Specifically, the threshold judgment logic compares the final score with the 80-point threshold and outputs advertising content that meets the conditions.

[0104] For example, the final ad content might be a combination of animated images and text with coupons, with a predicted score of 83. This approach, through multiple rounds of optimization and rigorous screening, ensures that the output ad content both aligns with user preferences and improves campaign performance. The entire process, driven by data and optimized through models, significantly enhances the targeting and appeal of the ad content.

[0105] S108. Calculate the potential trial-and-error cost index based on the optimized advertising content, use the simulation test module to verify the degree of cost reduction, and judge the overall generation efficiency.

[0106] Initial ad content data is acquired from an ad content database using data acquisition tools. Key features are extracted using text parsing methods to obtain a first feature set. If the completeness of the first feature set exceeds a preset threshold, a linear regression algorithm is used to analyze the feature set and determine a preliminary indicator of trial-and-error cost. Based on the preliminary indicator, a Monte Carlo simulation method is used to construct a test module, generating simulated test results to obtain the cost fluctuation range. If the cost fluctuation range is below the preset threshold, the cost reduction is calculated by comparing the simulated test results with historical data. Based on the cost reduction, a decision tree algorithm is used to adjust the optimization strategy, generating a second optimization strategy set. The ad content is iteratively adjusted using the second optimization strategy set to generate a second ad content dataset. Based on the second ad content dataset, the generation efficiency is recalculated to obtain the final efficiency evaluation result.

[0107] For example, when obtaining initial advertising content data from an advertising content database, multimodal information containing text, images, and user interaction data can be extracted using data collection tools.

[0108] Specifically, assuming an e-commerce platform's advertising content database contains product descriptions, user reviews, and click-through rate data, data collection tools can use web scraping technology to extract advertising text and user behavior data from the database over the past 30 days. Text parsing methods focus on extracting key features, such as keywords like "discount," "limited-time offer," and sentiment-related terms like "great value." Natural language processing (NLP) techniques are then used to analyze the text's sentiment score and keyword frequency, forming the first feature set. A completeness threshold can be set to 80%, meaning the feature set must cover more than 80% of the core advertising information.

[0109] In one possible implementation, if the completeness of the first feature set meets the standard, a linear regression algorithm can be used to analyze the feature set and determine a preliminary indicator of trial and error cost.

[0110] For example, for a promotional ad, the feature set includes keyword occurrence frequency, user click-through rate, and display duration. Linear regression can analyze the impact of these features on ad conversion rate and derive a trial-and-error cost metric, such as a conversion cost of 50 yuan per thousand impressions. Based on this, the Monte Carlo simulation method can be used to build a test module to simulate the ad's performance under different placement scenarios. Assuming 1000 simulated placements, if the cost fluctuation range is controlled between 40-60 yuan, which is 10% lower than the preset threshold, it indicates that the cost is controllable.

[0111] Specifically, when comparing simulation test results with historical data, the extent of cost reduction can be calculated.

[0112] For example, historical data shows that the average conversion cost is 60 yuan, while simulation results show that the cost drops to 45 yuan after optimization, a reduction of 25%. Based on this, the decision tree algorithm can adjust the optimization strategy and generate a second set of optimization strategies.

[0113] For example, decision tree analysis shows that adding the keyword "limited-time offer" and adjusting the timing of ad placement can improve conversion rates, so the strategy set includes more precise ad placement times and keyword combinations.

[0114] For example, a second set of optimized strategies is used to adjust ad content, generating a second ad content dataset. Assume the adjusted ad text emphasizes "24-hour limited-time offer" and optimizes delivery times for highly active user groups. When recalculating generation efficiency, improvements in click-through rate and conversion rate can be used for evaluation.

[0115] For example, after optimization, the click-through rate increased from 2% to 3.5%, and the conversion rate increased from 1% to 1.8%, indicating a significant improvement in the final efficiency evaluation results. These adjustments ensure that ad content is more attractive, while reducing trial-and-error costs and improving the accuracy of ad targeting.

[0116] like Figure 2 As shown, this invention provides an advertising content generation system based on multimodal fusion, which mainly includes: A multimodal data acquisition module is used to acquire raw input data from text, image, audio, and video sources through a preset multimodal data acquisition module; The feature extraction module is used to process the features of each modality using a feature extractor to obtain a unified set of feature vectors; The semantic fusion module is used to perform semantic fusion operations by applying the Transformer model based on a unified feature vector set, fusing text semantics and image visual features to determine a cross-modal consistent semantic representation matrix; The weight optimization module is used to adjust the fusion weight parameters if the similarity in the semantic representation matrix is ​​lower than a preset threshold, and obtain an enhanced fusion representation through iterative optimization. The creative generation module is used to obtain enhanced fusion representations and combine them with historical user feedback data to generate preliminary personalized creative solutions using generative adversarial networks, and to determine the initial matching degree of the creative solutions. The market adaptation module is used to extract relevant indicators from the dynamic market trend database for creative solutions with a high initial matching degree, and obtain the market adaptation adjustment vector. The effect prediction module is used to update creative solutions by adjusting vectors for market adaptation, apply long short-term memory networks to predict the effects of multimodal combinations, and determine the predicted effect score. The cost verification module is used to output the final ad content version if the predicted performance score exceeds the threshold; otherwise, it backtracks to the fusion representation step to re-optimize and obtain optimized ad content. Based on the optimized ad content, the potential trial-and-error cost index is calculated, and the simulation test module is used to verify the degree of cost reduction and judge the overall generation efficiency.

[0117] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating advertising content based on multimodal fusion, characterized in that, include: Obtain raw multimodal data related to the advertising content, extract features from the raw multimodal data, and obtain a unified set of feature vectors; The unified feature vector is semantically fused to obtain a semantic representation matrix. Similarity is assessed on the semantic representation matrix, and the matrix is ​​enhanced based on the similarity assessment results to obtain an enhanced fused representation. User feedback data is acquired, and a preliminary personalized creative plan is generated through a generative network based on the enhanced fused representation and user feedback data. The preliminary personalized creative plan is evaluated to obtain a preliminary matching degree. Indicators are acquired based on the preliminary matching degree and the preliminary personalized creative plan to obtain a market adaptation adjustment vector. The preliminary personalized creative plan is adjusted based on the market adaptation adjustment vector, and the adjusted preliminary personalized creative plan is evaluated to obtain a predicted performance score. Based on the predicted performance score, the adjusted preliminary personalized creative plan is optimized to obtain optimized advertising content.

2. The method according to claim 1, characterized in that, The process of feature extraction from the original multimodal data includes: Feature vectors of the original multimodal data are obtained by extracting features from the original multimodal data using a feature extractor; wherein the original data includes text data, image data, audio data, and video data. The feature vectors of the multimodal data are dimensionally determined, and based on the dimension determination result, the feature vectors of the multimodal data are dimensionally reduced to obtain a unified feature vector set.

3. The method according to claim 1, characterized in that, The process of obtaining the semantic representation matrix includes: The unified feature vector set is encoded using the Transformer model to obtain encoded text semantic vectors and image visual vectors. A multi-head attention mechanism is then used to perform cross-modal fusion of the encoded text semantic vectors and image visual vectors to obtain a fused feature vector. The fused feature vector is evaluated, and based on the evaluation result, a new fused feature vector is generated to obtain an optimized fused feature vector. A cross-modal semantic representation matrix is ​​constructed based on the optimized fused feature vector. The cross-modal semantic representation matrix is ​​then decomposed to obtain a low-dimensional semantic representation matrix. The low-dimensional semantic representation matrix is ​​then classified and semantic consistency is evaluated to obtain the semantic representation matrix.

4. The method according to claim 1, characterized in that, The process of obtaining the enhanced fusion representation includes: Initial semantic features are obtained from the semantic representation matrix, the association strength of the initial semantic features is extracted, the association strength is judged, the fusion weight parameters are updated according to the judgment result, and an optimized weight set is obtained. The semantic representation matrix is ​​weighted and fused through the optimized weight set to obtain a preliminary fused representation. The preliminary fused representation is similar to the original representation, the similarity evaluation result is judged, and an enhanced fused representation is generated according to the judgment result.

5. The method according to claim 1, characterized in that, The process of generating a preliminary personalized creative solution includes: The process involves acquiring historical user feedback data, cleaning the data, extracting user behavior patterns and preference features to create user profiles, generating multi-dimensional feature vectors based on these profiles to obtain enhanced representations, processing the enhanced fusion representations and enhanced representations using a generative adversarial network, generating preliminary personalized creative solutions, and matching these solutions with user profiles to obtain a preliminary matching degree.

6. The method according to claim 1, characterized in that, The process of obtaining the market adaptation adjustment vector includes: Based on the initial matching degree of the preliminary personalized creative solutions, an initial dataset is generated. The initial dataset is then filtered to obtain a set of high matching degree indicators. Trend extraction is performed on the set of high matching degree indicators to obtain market trend features. The market trend features are then judged. Based on the judgment results, an initial adjustment vector is generated according to the market trend features. The initial adjustment vector is then iteratively optimized to obtain an optimized adjustment vector. Based on the optimized adjustment vector, a market-adaptive adjustment vector is obtained.

7. The method according to claim 1, characterized in that, The process of obtaining the prediction performance score includes: Based on the market adaptation adjustment vector, user behavior features and market trend features are obtained. Feature extraction and dimensionality reduction are performed on the user behavior features and market trend features to obtain a simplified feature set. Gradient descent optimization is performed on the simplified feature set to obtain an optimized adjustment vector. The optimized adjustment vector is predicted through a long short-term memory network to obtain a prediction performance score.

8. The method according to claim 1, characterized in that, The optimized process for acquiring ad content includes: If the predicted performance score is higher than the preset threshold, the preliminary personalized creative scheme with the highest score is selected as the final ad content. If the predicted performance score is lower than the preset threshold, a backtracking optimization mechanism is triggered, and the fusion representation step is re-executed. Through the fusion representation step, features are extracted from multi-source data, and a data fusion representation is generated using principal component analysis. Based on the data fusion representation, a generative adversarial network algorithm is used to optimize and generate a new version of the ad content. For the new version of the content, a preset performance evaluation standard is used to calculate the predicted performance score. If the predicted performance score of the new version is still lower than the preset threshold, the number of optimization iterations is updated, and the fusion representation step is re-executed. Through threshold judgment logic, the final predicted performance score is compared with the preset threshold, and the final ad content that meets the conditions is output.

9. An advertising content generation system based on multimodal fusion, characterized in that, Used to perform the method described in any one of claims 1-8.

Images