Advertising copy generation method, device, equipment and medium based on multimodal fusion

Abstract

The present invention discloses a method, device, equipment and medium for generating advertising copy based on multimodal fusion, which relates to the field of advertising and marketing technology. The method comprises: obtaining multimodal data of a target video, wherein the multimodal data comprises visual data, auditory data and related metadata of the video, and extracting related data from a local knowledge base; preprocessing the multimodal data and the local knowledge base data, and converting them into a feature form that can be used for analysis. The present invention semantically calibrates the multimodal input with the help of a local knowledge base to ensure that the generated content strictly follows the constraints of domain knowledge, solves the problem of deviation caused by the traditional model's reliance on implicit knowledge, eliminates the ambiguity of multimodal information, realizes the collaborative semantic generation of text, images and structured data, enriches the content dimension, and provides an efficient solution for the implementation of intelligent generation technology in vertical fields.

Description

Technical Field

[0001] The present invention relates to the field of advertising and marketing technology, and in particular to a method, device, equipment and medium for generating advertising copy based on multimodal fusion. Background Art

[0002] In the past, advertising copywriting relied primarily on manual creation. This approach was not only inefficient and time-consuming, but also highly dependent on the creator's experience and creativity, making it difficult to ensure copywriting stability and consistency. For example, for large companies with multiple product lines or advertising campaigns in different regions, manual creation was difficult to generate sufficient and fully qualified copywriting in a short period of time.

[0003] With the development of natural language processing technology, some automated copy generation technologies have emerged. However, early automated copy generation was mostly based on simple template filling or rule matching. The generated copy lacked flexibility and innovation, making it difficult to meet diverse market needs. These technologies often only process single-modal data, such as generating copy based solely on text information, and are unable to fully utilize multimodal information such as images and audio to enrich the copy content and enhance its appeal.

[0004] While multimodal fusion technology has made progress in recent years, its application in advertising copy generation remains challenging. Existing multimodal fusion models often struggle to achieve deep semantic fusion when processing data from different modalities, resulting in weak correlation between the generated copy and the data from each modality. Furthermore, most of these models lack effective utilization of domain-specific knowledge, making it impossible to generate highly targeted advertising copy tailored to the characteristics and needs of different industries. For example, in e-commerce, finance, healthcare, and other fields, advertising copy must strictly adhere to industry standards and expertise, but existing technologies struggle to meet these requirements.

[0005] Current ad copy generation methods rarely integrate deeply with local knowledge bases. Local knowledge bases contain a wealth of valuable resources, including company-specific information, historical ad copy, and product data. However, existing technologies are unable to fully tap into and utilize these resources to optimize the copy generation process. This makes it difficult for the generated copy to reflect a company's brand characteristics and historical experience, and it also fails to adapt promptly to market changes and adjustments to a company's marketing strategy. Summary of the Invention

[0006] The present invention aims to at least partially address one of the technical issues in the aforementioned technologies. To this end, one objective of the present invention is to propose a method for generating advertising copy based on multimodal fusion, which can generate advertising copy by collecting and fusing multimodal data, thereby improving the matching, relevance, and quality of the advertising copy and video.

[0007] The second object of the present invention is to provide a multimodal fusion advertising copy generation device.

[0008] A third object of the present invention is to provide a computer device.

[0009] A fourth object of the present invention is to provide a computer-readable storage medium.

[0010] To achieve the above objectives, a first embodiment of the present invention proposes a method for generating advertising copy based on multimodal fusion, comprising the following steps:

[0011] Acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract associated data from a local knowledge base;

[0012] Preprocessing the multimodal data and local knowledge base data to convert them into feature forms that can be used for analysis;

[0013] Construct a knowledge base feature index, align the preprocessed local knowledge base data with the multimodal data, and unify them into a feature vector space of the same dimension;

[0014] Based on the pre-processed multimodal data and local knowledge base data, the multimodal fusion model is vertically adapted, and the fusion feature vector is output through the multimodal fusion model;

[0015] Using the fused features, the copy generation model generates the initial advertising copy;

[0016] Evaluate and optimize the initial advertising copy, and generate optimized advertising copy based on brand requirements, platform rules, and user feedback;

[0017] Adapt the optimized ad copy based on the common needs of different platforms' communication characteristics to generate ad copy versions that are suitable for multiple platforms;

[0018] Integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles;

[0019] The vertical adaptation of the multimodal fusion model includes:

[0020] The CLIP model is used to extract visual semantic embeddings from image data in the local knowledge base. , historical copywriting data generates structured semantic vectors through the T5 model , extracting speech style parameters from digital human data and posture feature matrix , forming a knowledge base feature set ;

[0021] Through a trainable cross-modal mapping matrix , the video multimodal features of multimodal data Knowledge base features with local knowledge base data Project to the unified feature space and calculate the cosine similarity matrix , identify highly relevant seed data in the knowledge base , where the seed data The similarity is greater than the feature matching threshold Knowledge base entries;

[0022] Perform targeted fine-tuning on the pre-trained multimodal fusion model and copywriting generation model, and introduce knowledge base constraint loss ,in is the cross entropy loss, is the model output distribution, is the prior distribution of knowledge base seed data, is the constraint weight coefficient;

[0023] Run cross-validation iterations on the model.

[0024] Preferably, the acquiring of multimodal data of the target video includes:

[0025] Extracting raw video frame sequences from video files as visual data;

[0026] Separate the audio stream from the video as auditory data;

[0027] Obtain metadata about the time, location, and / or hashtags of a video;

[0028] The local knowledge base includes:

[0029] Image data and its visual feature labels, historical advertising copy data and its semantic labels, digital human model data, and associated video clips in the video material library and their content labels.

[0030] Preferably, the preprocessing of the multimodal data and the local knowledge base data includes:

[0031] Perform image enhancement and noise reduction on visual data and extract key frames;

[0032] Perform audio noise reduction and speech endpoint detection on auditory data and convert speech into text;

[0033] Classify and organize metadata to remove redundant or invalid information;

[0034] Extract visual features from image data obtained from the local knowledge base;

[0035] Perform content analysis and feature extraction on video data;

[0036] Perform keyword extraction and semantic analysis on copywriting data;

[0037] Extract posture and expression features from digital human data.

[0038] Preferably, the vertical adaptation of the multimodal fusion model based on the pre-processed multimodal data and the local knowledge base data, and outputting the fusion feature vector through the multimodal fusion model, includes:

[0039] Perform feature extraction on visual data, auditory data and metadata respectively to obtain their respective feature vectors;

[0040] Convert the pre-processed images, videos, texts, and digital human feature information obtained from the local knowledge base into feature vectors;

[0041] Input the feature vectors of each modality into the multimodal fusion model based on the Transformer architecture;

[0042] The knowledge base gating mechanism is introduced in the Transformer encoder layer, through the learnable gating vector Dynamically select the fusion weight of video features and knowledge base features, where is the learnable weight matrix, is the Sigmoid function, To multiply elements by element, the high-frequency features in the knowledge base are activated preferentially;

[0043] Through the multi-head attention mechanism, the weights of each modal feature vector are dynamically allocated to achieve the fusion of multimodal data and output the fused feature vector; the contrast loss function is used

[0044] Enhance the model's ability to discriminate knowledge base features, where is the multimodal fusion feature vector, is a positive sample, is a negative sample, is the temperature parameter;

[0045] According to the characteristics and weights of the data in the local knowledge base, the weight distribution in the multi-head attention mechanism is adjusted.

[0046] Preferably, the generating of the initial advertising copy by using the fusion feature and the copy generation model includes:

[0047] Input the fused feature vector into the Transformer-based copywriting generation model;

[0048] Set the parameters of the copywriting generation model;

[0049] Build a knowledge base-guided decoding mechanism. During the decoding phase of the T5 model, dynamically generate decoding prefix hints based on the high-frequency structures of similar documents in the knowledge base.

[0050] Design a multi-round cross-validation process;

[0051] The model generates initial advertising copy based on the input fusion features and set parameters;

[0052] The multi-round cross-validation process includes:

[0053] Output copy based on rough matching model ;

[0054] Retrieve the knowledge base through the BM25 algorithm The three historical texts with the closest semantics ;

[0055] Calculate the ROUGE-L score of the generated copy and the reference copy. If it is lower than the text similarity threshold, , then adjust the model decoding parameters in reverse;

[0056] Repeat the above steps until the ROUGE-L score reaches the target or the maximum number of iterations is reached, and then output the final copy. .

[0057] Preferably, the evaluating and optimizing the initial advertising copy, combining brand requirements, platform rules and user feedback to generate optimized advertising copy, includes:

[0058] Conduct compliance checks on initial ad copy based on the keywords, banned word lists, and brand tone requirements provided by the brand;

[0059] Adjust the format and content of your initial ad copy based on the common ground rules across platforms;

[0060] Collect user feedback data on the initial ad copy;

[0061] Based on the feedback data, the reinforcement learning algorithm is used to optimize the initial advertising copy and generate the optimized advertising copy;

[0062] In the optimization of the initial advertising copy using the reinforcement learning algorithm based on the feedback data, the policy gradient formula is used for calculation, and the formula is as follows:

[0063] ;

[0064] Where, are the parameters of the policy network, is the objective function, It's a state. It's action. Is in state Take action The probability distribution of From the moment The cumulative reward at the beginning, is the expectation operator, averaging over all possible states and actions.

[0065] Preferably, the optimized advertising copy is adapted according to the common needs of the dissemination characteristics of different platforms to generate advertising copy versions adapted to multiple platforms, including:

[0066] Analyze the common patterns of user group characteristics and content style preferences on different platforms;

[0067] Based on the common rules, the optimized advertising copy is streamlined, specific guiding elements are added, and the copy style is adjusted to be more interactive, generating advertising copy versions that are adapted to the common needs of each platform.

[0068] The second embodiment of the present invention provides an advertising copy generation device based on multimodal fusion, comprising:

[0069] A data acquisition module is used to acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract related data from a local knowledge base;

[0070] A data preprocessing module, configured to preprocess the multimodal data and local knowledge base data and convert them into a feature form that can be used for analysis;

[0071] The feature alignment module is used to construct a knowledge base feature index and align the preprocessed local knowledge base data with the multimodal data to unify them into a feature vector space of the same dimension.

[0072] The fusion analysis module is used to perform vertical adaptation of the multimodal fusion model based on the preprocessed multimodal data and the local knowledge base data, and output the fusion feature vector through the multimodal fusion model;

[0073] The copy generation module is used to generate initial advertising copy using the fusion features and the copy generation model;

[0074] The evaluation and optimization module is used to evaluate and optimize the initial advertising copy, combining brand requirements, platform rules and user feedback to generate optimized advertising copy;

[0075] The adaptation processing module is used to adapt the optimized advertising copy according to the common needs of the dissemination characteristics of different platforms and generate advertising copy versions that are suitable for multiple platforms;

[0076] The integrated output module is used to integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles.

[0077] A third embodiment of the present invention provides an electronic device, which is a physical device and includes:

[0078] a processor and a memory, wherein the memory is communicatively connected to the processor;

[0079] The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the above-mentioned advertising copy generation method based on multimodal fusion.

[0080] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored therein. When the computer program is executed by a processor, the method for generating advertising copy based on multimodal fusion as described above is implemented.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] The present invention uses the local knowledge base to perform semantic calibration on multimodal inputs to ensure that the generated content strictly follows the constraints of domain knowledge, thereby solving the deviation problem caused by the traditional model's reliance on implicit knowledge. The contrast loss function optimizes the feature space by pulling positive and pushing negative, so that the model focuses on strong relevant information in the knowledge base. The dynamic matching mechanism supports real-time updating of the knowledge base. Combined with the adaptive adjustment of temperature parameters, the model can quickly adapt to new products or marketing strategies, reduce cross-domain application costs, and enhance robustness to noise input and complex scenarios. The deep alignment of cross-modal fusion features and knowledge base eliminates multimodal information ambiguity, realizes the collaborative semantic generation of text, images, and structured data, and enriches the content dimension. The overall solution constructs an extensible framework of "multimodal fusion + domain knowledge enhancement", which significantly improves marketing effectiveness while ensuring compliance and professionalism, and provides an efficient solution for the implementation of intelligent generation technology in vertical fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A main flow chart of a method for generating advertising copy based on multimodal fusion provided by an embodiment of the present invention;

[0084] Figure 2 A schematic structural diagram of an advertising copy generation device based on multimodal fusion provided by an embodiment of the present invention;

[0085] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0087] The execution subject of the method of this embodiment is a terminal, which can be a mobile phone, tablet computer, PDA, notebook or desktop computer, etc. Of course, it can also be other devices with similar functions, and this embodiment is not limited.

[0088] See also Figure 1 The present invention provides an advertising copy generation method based on multimodal fusion, which is applied to generate copy subtitles for advertising videos, comprising:

[0089] Step 100: Acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract associated data from a local knowledge base.

[0090] Specifically, step 100 includes:

[0091] Step 110 : extracting an original video frame sequence from the video file as visual data.

[0092] Alternatively, you can use FFmpeg. First, import the target video file into a supported environment. The tool will decode the video, converting it from its original compression format, such as H.264 or MPEG-4, into a raw image sequence suitable for analysis. Due to the large number of frames in a complete video, key frames are typically extracted according to specific rules to improve subsequent processing efficiency. For example, a fixed frame rate of 10 frames per second is used, or scene change detection technology is used to intelligently identify and select frames that represent different scene changes.

[0093] These extracted video frames contain key visual information from the video and serve as the foundational data for subsequent visual feature analysis. By processing these frames, we can identify objects, scenes, actions, and other elements within the video, providing a visual basis for generating advertising copy. For example, we can determine the appearance and usage scenarios of products shown in the video, allowing us to accurately describe these elements in the copy.

[0094] Step 120: Separate the audio stream from the video as auditory data.

[0095] Optionally, you can also use FFmpeg and execute the command to separate the audio stream in its operating environment. The tool will extract the audio part of the video file from the video container and generate an independent audio file. The extracted audio may not match the subsequent processing requirements in terms of sampling rate, number of channels and other parameters. In this case, you need to use audio processing tools to convert the audio into a common and easy-to-process sampling rate, such as 16kHz, and an appropriate number of channels, such as mono or stereo.

[0096] The separated audio stream contains rich information, such as character dialogue, environmental sound effects, background music, etc. By analyzing the audio, we can obtain the key voice content in the video, understand the emotional tone conveyed by the audio, and judge the atmosphere created by the video based on the background music or sound effects. This information is crucial for generating advertising copy that fits the overall feeling of the video, allowing the copy to match the video audio in terms of emotional expression and atmosphere creation.

[0097] Step 130: Obtain the video's shooting time, location, and / or subject tag metadata.

[0098] Alternatively, if the video file itself contains Exif metadata, you can extract the shooting time and location information using a tool or software module specifically designed to read Exif data. These tools parse the video file and filter out the "EXIFDateTimeOriginal" field containing the shooting time and the latitude and longitude information associated with the shooting location.

[0099] Alternatively, regarding topic tags, one approach is to perform a preliminary analysis of the visual and auditory data extracted from the video, using image classification techniques to identify key objects in the video, convert the audio into text, and then extract keywords. The topic tags can then be generated based on these analysis results. Alternatively, if the video comes from a pre-set library, the library may have already annotated the video with topic tags when it was recorded, and the corresponding video tags can be directly obtained from the library. Alternatively, the appropriate topic tags can be determined by matching the video content with existing tags in the library using a specific algorithm based on their relevance.

[0100] The shooting time and location can add background information to the advertising copy, allowing the copy to be created in combination with specific time and location elements, enhancing the relevance and appeal of the copy to the video. Theme tags help to quickly identify the core content and focus of the video, making it convenient to highlight key information when generating advertising copy. In multimodal data fusion analysis, theme tags can serve as important semantic clues to assist in understanding the overall intent of the video, so that the generated advertising copy can more accurately fit the video theme.

[0101] Wherein, the local knowledge base includes:

[0102] Image data and its visual feature labels, historical advertising copy data and its semantic labels, digital human model data, and associated video clips in the video material library and their content labels.

[0103] Step 200: pre-process the multimodal data and local knowledge base data to convert them into feature forms that can be used for analysis.

[0104] Specifically, step 200 includes:

[0105] Step 210 , using histogram equalization and Gaussian filtering to perform image enhancement and noise reduction on the visual data, and extract key frames;

[0106] By removing background noise, equipment noise, etc. from the audio, the clarity and intelligibility of the speech are improved, providing high-quality audio data for subsequent speech recognition and analysis.

[0107] Step 220 , performing audio noise reduction on the auditory data by spectral subtraction, detecting speech endpoints by a zero-crossing rate-based method, and then converting the speech into text form by using a Transformer-based speech recognition model;

[0108] The zero-crossing rate refers to the number of times an audio signal crosses the zero level per unit time. The zero-crossing rate of a speech signal is usually lower than that of noise. The start and end positions of speech are determined by detecting changes in the zero-crossing rate.

[0109] By accurately locating the starting and ending positions of the speech, the speech and non-speech parts in the audio can be distinguished, reducing the computational complexity of subsequent speech recognition and improving recognition accuracy.

[0110] Step 230: classify and organize the metadata to remove redundant or invalid information.

[0111] Optionally, step 230 includes:

[0112] Step 231 , classify metadata according to its type, for example, classify the shooting time, location and other information into geographic time category, and classify the subject tags into semantic category. Different types of metadata can be stored and managed in different ways to facilitate subsequent query and use.

[0113] Step 232, check whether there is duplicate information in the metadata. If there is, only one copy is retained. Some information that is obviously wrong or irrelevant to the video content, such as incorrect shooting location records, meaningless tags, etc., is deleted.

[0114] Step 230 can make the metadata clearer and more organized, facilitating subsequent multimodal fusion analysis. Removing redundant and invalid information can reduce data storage space, improve data processing efficiency, and ensure that the metadata used in generating advertising copy is accurate and useful.

[0115] The step 200 further includes:

[0116] Step 240: extract visual features from the image data obtained from the local knowledge base;

[0117] Step 250, performing content analysis and feature extraction on the video data;

[0118] Step 260: extract keywords and perform semantic analysis on the copywriting data;

[0119] Step 270: Extract posture and expression features from the digital human data.

[0120] Step 300: construct a knowledge base feature index, align the preprocessed local knowledge base data with the multimodal data, and unify them into a feature vector space of the same dimension to prepare for the vertical adaptation of the multimodal fusion model.

[0121] Understandably, by matching multimodal inputs (user needs, product information, etc.) with local knowledge bases (such as industry terminology libraries, product attribute libraries, and historical high-quality copy libraries), it is ensured that the generated advertising copy strictly complies with the knowledge constraints of specific fields (such as product parameter accuracy and brand tone consistency). Different from the traditional solution that relies on general pre-trained models, the model is adapted to the unique characteristics of the local knowledge base (such as the company's private user portraits and regionalized marketing rules) through algorithm adjustment (such as contrastive learning and parameter fine-tuning). After multimodal fusion (text + image + structured data), cross-modal semantic conflicts are corrected through knowledge base matching. For example, when there is ambiguity between image labels and text descriptions, the authoritative product definitions in the knowledge base are used as the basis for calibration.

[0122] Step 400 : Based on the pre-processed multimodal data and the local knowledge base data, vertical adaptation is performed on the multimodal fusion model, and a fusion feature vector is outputted through the multimodal fusion model.

[0123] Specifically, step 400 includes:

[0124] Step 410 , performing feature extraction on the visual data, the auditory data, and the metadata to obtain respective feature vectors;

[0125] Step 420: converting the pre-processed images, videos, texts, and digital human feature information obtained from the local knowledge base into feature vectors;

[0126] Step 430: Input the feature vectors of each modality into a multimodal fusion model based on the Transformer architecture;

[0127] Among them, when building a multimodal fusion model based on the Transformer architecture, a deep learning framework (such as PyTorch or TensorFlow) is used to build the model structure, which includes multiple Transformer blocks. Each block has a multi-head attention mechanism and a feedforward neural network. The feature vectors of vision, hearing, and metadata are converted into vector representations of the same dimension through the embedding layer. At the same time, position encoding is added to each vector to mark its modal order. These processed feature vectors are then input into the input layer of the Transformer model.

[0128] It is understandable that the powerful sequence modeling and parallel processing capabilities of the Transformer architecture are used to uniformly process multimodal feature vectors, preparing for the subsequent fusion of information through the multi-head attention mechanism.

[0129] Step 430: Input the feature vectors of each modality into a multimodal fusion model based on the Transformer architecture;

[0130] Step 440: Introduce a knowledge base gating mechanism into the Transformer encoder layer, using a learnable gating vector Dynamically select the fusion weight of video features and knowledge base features, where is the learnable weight matrix, is the Sigmoid function, To multiply elements by element, the high-frequency features in the knowledge base are activated preferentially;

[0131] Step 450: Dynamically assign weights to each modality feature vector through a multi-head attention mechanism to achieve fusion of multimodal data (including local knowledge base data) and output fusion feature vectors; use contrast loss function

[0132] Enhance the model's ability to discriminate knowledge base features, where is the multimodal fusion feature vector, is a positive sample (highly relevant knowledge base feature), is a negative sample (randomly selected non-relevant features), is the temperature parameter;

[0133] Step 460: Adjust the weight distribution in the multi-head attention mechanism based on the characteristics and weights of the data in the local knowledge base.

[0134] Among them, when performing vertical adaptation on the multimodal fusion model, the steps include the following:

[0135] Step S1, feature extraction: Use the CLIP model to extract visual semantic embeddings from the image data in the local knowledge base , historical copywriting data generates structured semantic vectors through the T5 model , extracting speech style parameters from digital human data and posture feature matrix , forming a knowledge base feature set ;

[0136] Step S2, alignment screening: through a trainable cross-modal mapping matrix , the video multimodal features of multimodal data Knowledge base features with local knowledge base data Project to the unified feature space and calculate the cosine similarity matrix , identify highly relevant seed data in the knowledge base , where the seed data The similarity is greater than the feature matching threshold Knowledge base entries;

[0137] Step S3, constrained fine-tuning: perform targeted fine-tuning on the pre-trained multimodal fusion model (such as ViT-BERT) and the copywriting generation model (such as T5-base), introducing knowledge base constraint loss ,in is the cross entropy loss, is the model output distribution, is the prior distribution of knowledge base seed data, is the constraint weight coefficient, through Divergence forces model output features Close to the prior distribution of knowledge base seed data ;

[0138] Step S4, validation iteration: perform cross-validation iteration on the model.

[0139] Understandably, by retaining the general capabilities of the model, forcing the model to prioritize high-frequency features in the knowledge base (such as brand-specific selling points and common platform phrases), freezing the underlying visual / language encoder of the multimodal fusion model (such as ViT-BERT), and only fine-tuning the upper-level attention mechanism and the decoding layer parameters of the copy generation model (T5) to reduce computational costs.

[0140] In this embodiment, the contrast loss is used to force the model to move closer to the positive samples (matching knowledge base entries) and the fusion features in the feature space. The distance between the positive and negative samples (non-matching entries) is increased, forming "semantic separability". For example, in copywriting generation, the model prioritizes historically successful templates (positive samples) that are strongly relevant to user needs in the knowledge base, while suppressing irrelevant templates (negative samples), thereby improving the relevance of generated content.

[0141] Through large-scale negative samples ( ) mining, forcing the model to learn fine-grained differential features in the knowledge base (such as subtle differences in product models and emotional differences in copywriting styles). Unlike traditional matching algorithms (such as TF-IDF and rule matching), contrastive learning can capture nonlinear semantic relationships. For example, it can identify the opposition between the two styles of "high-end luxury" and "value for money" in the semantic space.

[0142] By temperature parameter Adaptive adjustment to balance the strictness and flexibility of matching. When the knowledge base contains a large number of entries, increase A certain degree of fuzzy matching can be allowed to avoid "matching failure" caused by excessive strictness; when precise matching is required (such as in the medical and financial fields), the fuzzy matching can be reduced. To strengthen strict distinction;

[0143] The input of the loss function is multimodal fusion features , prompting the model to align cross-modal information with knowledge base entries during the matching process, for example, ensuring that image features (such as product color), text features (such as "red promotion") and "festival marketing" entries in the knowledge base are consistent in the semantic space.

[0144] Step 500: Utilize the fusion features and generate initial advertising copy through a copy generation model.

[0145] Specifically, step 500 includes:

[0146] Step 510: Input the fused feature vector into the Transformer-based copywriting generation model.

[0147] Optionally, the fused feature vector output by the multimodal fusion model is used as input data, and a deep learning framework such as PyTorch or TensorFlow is selected to build and train a Transformer-based copy generation model.

[0148] Step 520 , setting parameters of the copywriting generation model, such as generation length, language style preference, etc.;

[0149] Optionally, set the parameters of the copy generation model based on actual needs. For the generation length parameter, determine the approximate word count for the generated copy based on the requirements of the advertising platform and the focus of the video content. For example, for ads placed on short video platforms, the copy might be set to 20-50 words; for video ads on e-commerce detail pages, the generated copy length can be set to 100-200 words. Regarding language style preference parameters, if the video is an advertisement for fashion products, the language style preference can be set to fashionable, trendy, and lively; if it is an advertisement for high-end business products, the style preference can be set to formal, professional, and stable. In the code, adjust these parameters by setting the corresponding variables.

[0150] Step 530: Build a knowledge base-guided decoding mechanism. During the decoding phase of the T5 model (e.g., the "pain point - solution - value" triple), dynamically generate decoding prefix hints based on the high-frequency structure of similar copywriting in the knowledge base.

[0151] Step 540, designing a multi-round cross-validation process;

[0152] In step 550 , the model generates initial advertising copy based on the input fusion features and set parameters.

[0153] Wherein, step 540 includes:

[0154] Step 541: Output the copy based on the rough matching model ;

[0155] Step 542: Search the knowledge base for the The three historical texts with the closest semantics ;

[0156] Step 543: Calculate the ROUGE-L score of the generated text and the reference text. If it is lower than the text similarity threshold, , then adjust the model decoding parameters in reverse;

[0157] Step 544: Repeat the above steps until the ROUGE-L score reaches the target or the maximum number of iterations is reached, and then output the final copy. .

[0158] Understandably, through multiple rounds of training-validation-parameter adjustment, we ensure that the model output is deeply matched with the knowledge base to avoid overfitting or underfitting.

[0159] Step 600: Evaluate and optimize the initial advertising copy, and generate an optimized advertising copy based on brand requirements, platform rules, and user feedback.

[0160] Specifically, step 600 includes:

[0161] Step 610: Based on the keywords and banned words list provided by the brand and the brand tone requirements, the initial advertising copy is checked for compliance, such as using a string matching algorithm to check whether the initial advertising copy contains these keywords and banned words;

[0162] Step 620 , adjusting the format and content of the initial ad copy based on basic rules such as character limits and topic usage specifications common to different platforms;

[0163] Step 630: Collect user feedback data on the initial advertising copy, such as click-through rate, conversion rate, and comment sentiment.

[0164] Step 640 , based on the feedback data, using a reinforcement learning algorithm to optimize the initial advertising copy to generate an optimized advertising copy;

[0165] In step 640, the initial advertisement copy is optimized using the policy gradient formula, which is as follows:

[0166] ;

[0167] Where, are the parameters of the policy network (such as neural network weights), is the objective function (e.g., the expectation of cumulative reward), is the state (such as the characteristics of the current ad copy, user feedback), It is an action (such as adjusting the keywords and structure of the copy), Is in state Take action The probability distribution of From the moment Initial cumulative rewards (such as click-through rate, conversion rate), is the expectation operator, averaging over all possible states and actions.

[0168] Understandable, state Including the characteristics of the current advertising copy (such as keywords, sentence structure, emotional tendency, etc.) and user feedback data (click-through rate, conversion rate, comment emotional tendency, etc.), this information can be encoded and converted into vector form as the input of the reinforcement learning algorithm. This includes adjusting the keywords, structure, and tone of the copy. For example, you can define a series of operations, such as replacing a keyword or changing the order of sentences.

[0169] Then use a neural network (such as a multilayer perceptron) to build a policy network ,in are the parameters of the network (weights and biases), and the input of the policy network is the state , the output is the probability distribution of taking each action in this state;

[0170] Then calculate the cumulative rewards based on indicators such as click-through rate and conversion rate ,For example, the weighted sum of click-through rate and conversion rate can be used as the reward at the current moment;

[0171] Then update the parameters of the policy network according to the policy gradient formula ,In actual implementation, Monte Carlo methods can be used to estimate the expectation,by sampling states and actions multiple times, calculating the approximate value of the gradient, and then using an optimization algorithm (such as,stochastic gradient descent) to update the parameters;

[0172] After the parameters of the policy network are updated, according to the current state , select the action with the highest probability through the policy network Adjust the advertising copy and generate optimized advertising copy.

[0173] In this embodiment, through the reinforcement learning algorithm, the advertising copy is continuously optimized according to user feedback to improve the effectiveness of the advertisement, such as click-through rate, conversion rate, etc., so that the advertising copy can better meet user needs and market requirements.

[0174] Step 700 , adapting the optimized advertising copy according to the common needs of the dissemination characteristics of different platforms to generate advertising copy versions that are adapted to multiple platforms.

[0175] Specifically, step 700 includes:

[0176] Step 710: Analyze the common patterns of user group characteristics and content style preferences on different platforms;

[0177] Optionally, for user group characteristics, statistical analysis can be used to find the overlaps in age, gender, etc. between different platforms to determine the main user group characteristics; for content style preferences, cluster analysis methods can be used to classify popular content on different platforms according to style to find common style characteristics. For example, it is found that popular content on multiple platforms tends to use concise and clear language and has a certain emotional resonance.

[0178] Step 720 , based on the common rules, the optimized advertising copy is streamlined, specific guiding elements are added, and the style of the copy is adjusted to be more interactive, thereby generating an advertising copy version that meets the common needs of each platform.

[0179] Optionally, first identify redundant information in the copy, such as repeated expressions and unnecessary modifiers, and then use a summary extraction algorithm from natural language processing technology to extract the core content of the copy. For example, long sentences can be split into short sentences and some descriptive but non-key words can be removed.

[0180] Then, analyze the common patterns for guiding user interaction and identify specific guidance elements that need to be added. Common guidance elements include calls to action (such as "Click to buy" and "Share with friends") and questions (such as "Do you like this product?"). Based on the content and objectives of the copy, add these guidance elements in appropriate locations. For example, add a call to action at the end of the copy to guide users to the next step.

[0181] Finally, based on the common preference for interactive content style, adjust the language expression and structure of the copy. In terms of language expression, use more friendly and colloquial vocabulary and tone to increase the sense of closeness with users. In terms of structure, you can use questions, set suspense, etc. to attract users' attention and stimulate their curiosity. For example, change the formal statement "This product has many advantages" to "Did you know? This product has many great features!"

[0182] Through the above steps, you can generate advertising copy versions that are adapted to the common needs of each platform, so that advertisements on different platforms can better attract users and improve communication effects.

[0183] Step 800: Integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles.

[0184] In this embodiment, semantic calibration of multimodal inputs is performed with the help of a local knowledge base to ensure that the generated content strictly follows domain knowledge constraints, thereby solving the problem of deviation caused by traditional models relying on implicit knowledge. The contrast loss function optimizes the feature space by pulling positive and pushing negative, so that the model focuses on strong relevant information in the knowledge base. The dynamic matching mechanism supports real-time updating of the knowledge base. Combined with adaptive adjustment of temperature parameters, the model can quickly adapt to new products or marketing strategies, reduce cross-domain application costs, and enhance robustness to noise input and complex scenarios. The deep alignment of cross-modal fusion features with the knowledge base eliminates multimodal information ambiguity, realizes collaborative semantic generation of text, images, and structured data, and enriches content dimensions. The overall solution builds an extensible framework of "multimodal fusion + domain knowledge enhancement", which significantly improves marketing effectiveness while ensuring compliance and professionalism, and provides an efficient solution for the implementation of intelligent generation technology in vertical fields.

[0185] Based on the above embodiments, Figure 2As shown, the present invention further provides an advertising copy generation device based on multimodal fusion, which is used to support the advertising copy generation method based on multimodal fusion of the above embodiment. The advertising copy generation device based on multimodal fusion includes:

[0186] A data acquisition module 11 is used to acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract related data from a local knowledge base;

[0187] A data preprocessing module 12 is used to preprocess the multimodal data and local knowledge base data and convert them into a feature form that can be used for analysis;

[0188] A feature alignment module 13 is used to construct a knowledge base feature index, align the pre-processed local knowledge base data with the multimodal data, and unify them into a feature vector space of the same dimension;

[0189] A fusion analysis module 14 is configured to perform vertical adaptation of the multimodal fusion model based on the preprocessed multimodal data and the local knowledge base data, so as to adapt the model to the domain characteristics of the local knowledge base, and output a fusion feature vector through the multimodal fusion model. The vertical adaptation includes feature extraction, alignment screening, constraint fine-tuning, and verification iterations.

[0190] The copy generation module 15 is used to generate initial advertising copy using the fusion features and the copy generation model;

[0191] Evaluation and optimization module 16, used to evaluate and optimize the initial advertising copy, and generate optimized advertising copy based on brand requirements, platform rules and user feedback;

[0192] Adaptation processing module 17, used to adapt the optimized advertising copy according to the common needs of the dissemination characteristics of different platforms, and generate advertising copy versions adapted to multiple platforms;

[0193] The integration output module 18 is used to integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles.

[0194] In this embodiment, semantic calibration of multimodal inputs is performed with the help of a local knowledge base to ensure that the generated content strictly follows domain knowledge constraints, thereby solving the problem of deviation caused by traditional models relying on implicit knowledge. The contrast loss function optimizes the feature space by pulling positive and pushing negative, so that the model focuses on strong relevant information in the knowledge base. The dynamic matching mechanism supports real-time updating of the knowledge base. Combined with adaptive adjustment of temperature parameters, the model can quickly adapt to new products or marketing strategies, reduce cross-domain application costs, and enhance robustness to noise input and complex scenarios. The deep alignment of cross-modal fusion features with the knowledge base eliminates multimodal information ambiguity, realizes collaborative semantic generation of text, images, and structured data, and enriches content dimensions. The overall solution builds an extensible framework of "multimodal fusion + domain knowledge enhancement", which significantly improves marketing effectiveness while ensuring compliance and professionalism, and provides an efficient solution for the implementation of intelligent generation technology in vertical fields.

[0195] Furthermore, the apparatus for generating advertising copy based on multimodal fusion may run the above-mentioned method for generating advertising copy based on multimodal fusion. For specific implementation, please refer to the method embodiment, which will not be described in detail here.

[0196] Based on the above embodiments, Figure 3 As shown, the present invention also provides an electronic device, the electronic device comprising:

[0197] At least one processor 22, at least one memory 21, a communication interface 23 and a communication bus 24, wherein the processor 22 is communicatively connected to the memory 21;

[0198] In this embodiment, the memory 21 can be implemented in any appropriate manner, for example, the memory 21 can be a read-only memory, a mechanical hard disk, a solid-state drive, or a USB flash drive, etc. The memory 21 is used to store at least one executable instruction executed by the processor;

[0199] In this embodiment, the processor 22 can be implemented in any appropriate manner. For example, the processor 22 can take the form of a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc.; the processor is used to execute the executable instructions to implement the above-mentioned advertising copy generation method based on multimodal fusion.

[0200] Based on the above embodiments, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating advertising copy based on multimodal fusion as described above is implemented.

[0201] Those skilled in the art will appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0202] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, equipment and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or units can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or equipment, which can be electrical, mechanical or other forms.

[0204] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0206] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only storage server, a random access storage server, a magnetic disk, or an optical disk.

[0207] In addition, it should be noted that the combination of the various technical features in this case is not limited to the combination described in the claims of this case or the combination described in the specific embodiments. All technical features recorded in this case can be freely combined or combined in any way unless there is a contradiction between them.

[0208] It should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many similar variations are possible. All variations directly derived from or associating with the present invention by those skilled in the art are intended to fall within the scope of protection of the present invention.

[0209] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating advertising copy based on multimodal fusion, characterized in that: The following steps are involved: Acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract associated data from a local knowledge base; Preprocessing the multimodal data and local knowledge base data to convert them into feature forms that can be used for analysis; Construct a knowledge base feature index, align the preprocessed local knowledge base data with the multimodal data, and unify them into a feature vector space of the same dimension; Based on the pre-processed multimodal data and local knowledge base data, the multimodal fusion model is vertically adapted, and the fusion feature vector is output through the multimodal fusion model; Using the fused features, the copy generation model generates the initial advertising copy; Evaluate and optimize the initial advertising copy, and generate optimized advertising copy based on brand requirements, platform rules, and user feedback; Adapt the optimized ad copy based on the common needs of different platforms' communication characteristics to generate ad copy versions that are suitable for multiple platforms; Integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles; The vertical adaptation of the multimodal fusion model includes: The CLIP model is used to extract visual semantic embeddings from image data in the local knowledge base. , historical copywriting data generates structured semantic vectors through the T5 model , extracting speech style parameters from digital human data and posture feature matrix , forming a knowledge base feature set ; Through a trainable cross-modal mapping matrix , the video multimodal features of multimodal data Knowledge base features with local knowledge base data Project to the unified feature space and calculate the cosine similarity matrix , identify highly relevant seed data in the knowledge base , where the seed data The similarity is greater than the feature matching threshold Knowledge base entries; Perform targeted fine-tuning on the pre-trained multimodal fusion model and copywriting generation model, and introduce knowledge base constraint loss ,in is the cross entropy loss, is the model output distribution, is the prior distribution of knowledge base seed data, is the constraint weight coefficient; Run cross-validation iterations on the model.

2. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The obtaining of multimodal data of the target video includes: Extracting raw video frame sequences from video files as visual data; Separate the audio stream from the video as auditory data; Obtain metadata about the time, location, and / or hashtags of a video; The local knowledge base includes: Image data and its visual feature labels, historical advertising copy data and its semantic labels, digital human model data, and associated video clips in the video material library and their content labels.

3. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The preprocessing of the multimodal data and the local knowledge base data includes: Perform image enhancement and noise reduction on visual data and extract key frames; Perform audio noise reduction and speech endpoint detection on auditory data and convert speech into text; Classify and organize metadata to remove redundant or invalid information; Extract visual features from image data obtained from the local knowledge base; Perform content analysis and feature extraction on video data; Perform keyword extraction and semantic analysis on copywriting data; Extract posture and expression features from digital human data.

4. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The method of vertically adapting the multimodal fusion model based on the pre-processed multimodal data and the local knowledge base data, and outputting a fusion feature vector through the multimodal fusion model, includes: Perform feature extraction on visual data, auditory data and metadata respectively to obtain their respective feature vectors; Convert the pre-processed images, videos, texts, and digital human feature information obtained from the local knowledge base into feature vectors; Input the feature vectors of each modality into the multimodal fusion model based on the Transformer architecture; The knowledge base gating mechanism is introduced in the Transformer encoder layer, through the learnable gating vector Dynamically select the fusion weight of video features and knowledge base features, where is the learnable weight matrix, is the Sigmoid function, To multiply elements by element, the high-frequency features in the knowledge base are activated preferentially; Through the multi-head attention mechanism, the weights of each modal feature vector are dynamically allocated to achieve the fusion of multimodal data and output the fused feature vector; the contrast loss function is used Enhance the model's ability to discriminate knowledge base features, where is the multimodal fusion feature vector, is a positive sample, is a negative sample, is the temperature parameter; According to the characteristics and weights of the data in the local knowledge base, the weight distribution in the multi-head attention mechanism is adjusted.

5. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The method of generating initial advertising copy by using the fusion features and the copy generation model includes: Input the fused feature vector into the Transformer-based copywriting generation model; Set the parameters of the copywriting generation model; Build a knowledge base-guided decoding mechanism. During the decoding phase of the T5 model, dynamically generate decoding prefix hints based on the high-frequency structures of similar documents in the knowledge base. Design a multi-round cross-validation process; The model generates initial advertising copy based on the input fusion features and set parameters; The multi-round cross-validation process includes: Output copy based on rough matching model ; Retrieve the knowledge base through the BM25 algorithm The three historical texts with the closest semantics ; Calculate the ROUGE-L score of the generated copy and the reference copy. If it is lower than the text similarity threshold, , then adjust the model decoding parameters in reverse; Repeat the above steps until the ROUGE-L score reaches the target or the maximum number of iterations is reached, and then output the final copy. .

6. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The evaluation and optimization of the initial advertising copy, combined with brand requirements, platform rules and user feedback, to generate optimized advertising copy, includes: Conduct compliance checks on initial ad copy based on the keywords, banned word lists, and brand tone requirements provided by the brand; Adjust the format and content of your initial ad copy based on the common ground rules across platforms; Collect user feedback data on the initial ad copy; Based on the feedback data, the reinforcement learning algorithm is used to optimize the initial advertising copy and generate the optimized advertising copy; In the optimization of the initial advertising copy using the reinforcement learning algorithm based on the feedback data, the policy gradient formula is used for calculation, and the formula is as follows: ; Where, are the parameters of the policy network, is the objective function, It's a state. It's action. Is in state Take action The probability distribution of From the moment The cumulative reward at the beginning, is the expectation operator, averaging over all possible states and actions.

7. The method for generating advertising copy based on multimodal fusion according to claim 1, characterized in that: The optimized advertising copy is adapted according to the common needs of the dissemination characteristics of different platforms to generate advertising copy versions that are adapted to multiple platforms, including: Analyze the common patterns of user group characteristics and content style preferences on different platforms; Based on the common rules, the optimized advertising copy is streamlined, specific guiding elements are added, and the copy style is adjusted to be more interactive, generating advertising copy versions that are adapted to the common needs of each platform.

8. An advertising copy generation device based on multimodal fusion, characterized in that: include: A data acquisition module is used to acquire multimodal data of a target video, wherein the multimodal data includes visual data, auditory data, and related metadata of the video, and extract related data from a local knowledge base; A data preprocessing module, configured to preprocess the multimodal data and local knowledge base data and convert them into a feature form that can be used for analysis; The feature alignment module is used to construct a knowledge base feature index and align the preprocessed local knowledge base data with the multimodal data to unify them into a feature vector space of the same dimension. The fusion analysis module is used to perform vertical adaptation of the multimodal fusion model based on the pre-processed multimodal data and the local knowledge base data, so that the model adapts to the domain characteristics of the local knowledge base, and outputs the fusion feature vector through the multimodal fusion model; The copy generation module is used to generate initial advertising copy using the fusion features and the copy generation model; The evaluation and optimization module is used to evaluate and optimize the initial advertising copy, combining brand requirements, platform rules and user feedback to generate optimized advertising copy; The adaptation processing module is used to adapt the optimized advertising copy according to the common needs of the dissemination characteristics of different platforms and generate advertising copy versions that are suitable for multiple platforms; An integrated output module is used to integrate the adapted advertising copy with the video to generate a complete video with corresponding advertising copy subtitles; The vertical adaptation of the multimodal fusion model includes: The CLIP model is used to extract visual semantic embeddings from image data in the local knowledge base. , historical copywriting data generates structured semantic vectors through the T5 model , extracting speech style parameters from digital human data and posture feature matrix , forming a knowledge base feature set ; Through a trainable cross-modal mapping matrix , the video multimodal features of multimodal data Knowledge base features with local knowledge base data Project to the unified feature space and calculate the cosine similarity matrix , identify highly relevant seed data in the knowledge base , where the seed data The similarity is greater than the feature matching threshold Knowledge base entries; Perform targeted fine-tuning on the pre-trained multimodal fusion model and copywriting generation model, and introduce knowledge base constraint loss ,in is the cross entropy loss, is the model output distribution, is the prior distribution of knowledge base seed data, is the constraint weight coefficient; Run cross-validation iterations on the model.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory, wherein the memory is communicatively connected to the processor; The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the advertising copy generation method based on multimodal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating advertising copy based on multimodal fusion according to any one of claims 1 to 7 is implemented.

Images