Text-driven advertisement video generation method based on emotion feedback

Abstract

The invention discloses a text-driven advertisement video generation method based on emotion feedback, relates to the technical field of digital advertisement and man-machine interaction, and aims to dynamically adjust advertisement content and improve individuation, interactivity and conversion rate by collecting emotion data such as facial expression, voice features and eye movement tracks of a user in real time. A text generation video model and a multi-mode emotion recognition technology are combined, and feature fusion and feedback processing are utilized, so that the advertisement video better conforms to the emotion of the user; multi-round feedback and self-adaptive optimization ensure that advertisements transmit core information and flexibly respond to user emotional states at the same time, attraction and participation degree are greatly improved, an innovative feature fusion algorithm further improves individuation and real-time response capability of generated videos, a brand new solution is provided for the field of digital advertisements, and the method is suitable for popularization and application. And the development of individuation and emotional interaction is promoted.

Description

Technical Field

[0001] The invention relates to the technical field of digital advertisement and human-computer interaction, and in particular to a text-driven advertisement video generation method based on emotional feedback. Background Art

[0002] In the digital advertising space, traditional video ads are usually pre-made fixed content that lacks flexibility and cannot be dynamically adjusted based on the audience's personalized needs and emotional state. This static advertising format is difficult to effectively attract user interest, resulting in limited audience engagement and advertising conversion rates. In addition, the audience's emotional state may change while watching an ad, but traditional advertising cannot respond to this in a timely manner, weakening the effectiveness of the ad.

[0003] With the rapid development of artificial intelligence and deep learning technologies, new possibilities have emerged, especially in generative models and multimodal emotion recognition technologies. Text-based generative models, such as GPT and diffusion models, have been widely used in the generation of image and video content. These technologies are able to generate diverse visual content based on input text to a certain extent. However, current applications in the advertising field are still mainly focused on generating static or single-style content, lacking the ability to dynamically respond to users' personalized needs and emotional feedback.

[0004] At the same time, the progress of affective computing technology has made it possible to obtain user emotional data in real time through sensors such as cameras, microphones, and eye tracking. However, existing technologies have not yet been able to organically combine multimodal emotion recognition with video generation models for dynamic generation and optimization of advertising videos, and there are problems such as insufficient real-time performance and poor feature fusion effects. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a text-driven advertising video generation method based on emotional feedback, comprising the following steps:

[0006] S1. The advertiser or content creator provides a piece of advertising copy T to clarify the core promotional information;

[0007] S2, convert the advertising copy T into the initial advertising video V0 through the text generation model;

[0008] S3, collecting multimodal emotional feedback data X fed back by users when watching the initial advertising video in real time through a multimodal sensor;

[0009] S4. Based on the deep learning model, the collected multimodal emotional feedback data X is feature extracted and fused to transform it into a high-dimensional feature vector F final ;

[0010] S5. According to the high-dimensional feature vector Ffinal and the advertising copy T, dynamically adjust the current video content and generate a new advertising video V t+1 ;

[0011] S6. The new advertising video V t+1 For users to watch, the multimodal sensor collects new multimodal emotional feedback data;

[0012] S7. Update the video generation strategy parameters θ through the adaptive optimization process t ;

[0013] S8, repeating steps S5 to S7 until the generated advertising video content maximizes the user's emotional engagement or reaches a preset number of cycles;

[0014] S9, the optimization process is terminated and the final advertising video V is output final ;

[0015] S10, the optimized final advertising video V final It will be put on the market and further analyzed and improved based on subsequent user feedback.

[0016] The technical solution further defined in the present invention is:

[0017] Furthermore, in step S2, the text generation model is set to Sora or CogVideoX-5b.

[0018] As described above, in the method for generating a text-driven advertising video based on emotional feedback, in step S3, the multimodal sensor includes a camera, a microphone, and an eye tracking device, and the multimodal emotional feedback data includes a facial expression image, a voice signal, and an eye tracking image, which are respectively denoted as X f , X v , and X g , get multimodal emotional feedback data X = {X f , X v , X g}.

[0019] As described above, in the text-driven advertising video generation method based on emotional feedback, step S4 specifically includes the following sub-steps:

[0020] S4.1. Input facial expression images, speech signals, and eye movement trajectory images into the deep learning model. Where N is the number of samples, H f , W f Represent the height and width of the facial expression image, C f Indicates the number of channels; voice signal Where T v represents the number of time steps, dv Dimensions for representing speech; eye-tracking images Among them, H g , W g Respectively represent the height and width of the eye movement track image, C g Indicates the number of channels;

[0021] S4.2. For image data X f and X g , use convolutional neural network to extract features and convert them into facial expression features and eye movement trajectory features, facial expression features where d′ f Represents the dimension of the extracted feature vector; eye movement trajectory features where d′ g Represents the dimension of the extracted feature vector;

[0022] S4.3. For the speech signal X v , which is converted into speech features using an automatic speech recognition model where d′ v Represents the dimension of the extracted feature vector;

[0023] S4.4. Use the multi-head attention mechanism to calculate and fuse different features to obtain a high-dimensional feature vector d a Represents the dimension of each sample.

[0024] As described above, in the text-driven advertising video generation method based on emotional feedback, in step S4.2, the convolutional neural network is set to VGG16 or VGG19; in step S4.3, the automatic speech recognition model is set to wav2vec2.0.

[0025] As described above, in the text-driven advertising video generation method based on emotional feedback, step S4.4 specifically includes the following sub-steps:

[0026] S4.4.1. Convert facial expression features F, speech features V, and eye movement track features G into query matrix, key matrix, and value matrix respectively:

[0027]

[0028] in, Represents the weight matrix that needs to be optimized during training, which is used to project the input features into the latent space of the attention mechanism; for the query matrix The feature dimension of each sample is d′ f , represents the linear transformation weight of the query matrix, where d a represents the dimension of the attention mechanism's factor space; for the key matrix The feature dimension of each sample is d′ v , represents the linear transformation weights of the key matrix; for the value matrix Represents the linear transformation weights of the value matrix; for the key matrix The feature dimension of each sample is d′ g , represents the linear transformation weights of the key matrix; for the value matrix Represents the linear transformation weights of the value matrix;

[0029] According to the above matrix, we can calculate F's attention to V and G respectively. The formula is as follows:

[0030]

[0031] For two attention matrices A F，V and A F，G Combine them by weighted average:

[0032]

[0033] Among them, α is a weight parameter;

[0034] S4.4.2. Find the attention matrices corresponding to V and G respectively and At the same time, the mean square combination method is used to obtain the attention matrix of an attention head:

[0035]

[0036] S4.4.3. Concatenate and project the outputs of multiple attention heads:

[0037] F 融合 =Concat(head1, head2,..., head h )W O

[0038] Among them, W O Represents the final projection matrix;

[0039] S4.4.4, the final fused feature is expressed as:

[0040] F final =MLP(LayerNorm(F 融合 )).

[0041] As described above, in the text-driven advertising video generation method based on emotional feedback, in step S5, a new advertising video is generated by the following formula:

[0042] Vt+1 =AdjustVideo( V t,F final , T)

[0043] Among them, V t+1 represents the new advertising video generated after the t+1th round of emotional feedback, V t represents the advertising video generated at the tth round of emotional feedback, F final Represents a high-dimensional feature vector; T represents the initial input advertising copy.

[0044] As described above, in the text-driven advertising video generation method based on emotional feedback, in step S7, after each round of feedback, the video generation strategy is updated:

[0045]

[0046] in, θ t+1 represents the generation strategy parameters updated after the t+1th round of feedback, which is used to determine the strategy for the next round of advertising video generation; θ t represents the generation strategy parameters used in the tth round of feedback to generate the current advertising video V t ; η represents the learning rate, which is used to control the step size of each gradient update; Denotes the loss function L(F final , T) is the gradient of the policy parameter θ, which is used to reflect the current generation strategy θ t Emotional feedback to users final and the response effect of advertising copy T; L(F final , T) represents the loss function, which is the objective function of the optimization process. final , T) improves the effect of generating advertising videos.

[0047] As described above, in the text-driven advertising video generation method based on emotional feedback, in step S4, the deep learning model includes the following working steps:

[0048] M1. Input a video sequence containing multiple frames, each frame is an image in the video, input these frames into the TimeSformer module, and output a feature map of the time dimension;

[0049] M2, extract facial expression features, voice features and eye movement trajectory features respectively, and fuse these features into a feature matrix for representing the multimodal information of the video;

[0050] M3, input the advertisement copy into the Batch Normalization layer, normalize the features in the advertisement copy, and output the normalized text features;

[0051] M4, processes the visual features of the video frame obtained in step M1 through the VisionTransformer module, divides the image into patches of fixed size, and processes these patches as input to generate the final representation of the image features;

[0052] M5, fusing the outputs of step M2, step M3 and step M4 to obtain a fused feature matrix;

[0053] M6, the fused feature matrix passes through the fully connected layer to convert the high-dimensional fused features into a format suitable for the generation task, which is used for video content generation, and the residual network is used in the fully connected layer; then it passes through the deconvolution layer to convert these features back to the format of the video frame, and through multiple deconvolution operations, the features are mapped to the pixel level to generate a new video frame;

[0054] M7, finally output a new video sequence.

[0055] As described above, in the text-driven advertising video generation method based on emotional feedback, step M4 specifically includes the following sub-steps:

[0056] M4.1. Input video dimensions (T, H, W, C), where T represents the number of frames, i.e., the time dimension; H represents the height of the frame, W represents the width of the frame, and C represents the number of channels of the frame;

[0057] M4.2. Split the input frame into fixed-size image patches, each of which is P×P in size, and flatten each patch to generate a fixed-size image patch of length P. 2 C; the flattened patch is mapped to a fixed dimension D through a linear transformation layer; a position code is added to each patch, which indicates the position of the patch in the original image. The position code has the same size as the dimension after the linear transformation, i.e. D, and is added to the representation of each patch;

[0058] M4.3. Process patch embeddings through a multi-head self-attention mechanism and calculate the relationship between them; the output of the multi-head self-attention mechanism passes through a feedforward neural network, residual connection, and layer normalization in turn;

[0059] M4.4. After passing through the Transformer encoder, the embeddings of all patches are recombined and passed through a fully connected layer to generate the final representation of the image features. The output dimension is (T, D), where T represents the time dimension, that is, the number of frames, and D represents the feature dimension.

[0060] The beneficial effects of the present invention are:

[0061] (1) In the present invention, by collecting the user's emotional feedback in real time and dynamically adjusting the video content, it is ensured that the advertisement is highly consistent with the user's emotional state. This personalized advertising experience greatly improves the user's participation and acceptance of the advertising content;

[0062] (2) In the present invention, by optimizing video content in real time, the video content is made closer to the needs and emotions of users, thereby effectively improving the attractiveness of advertisements, increasing click-through rates and conversion rates, and achieving better marketing effects;

[0063] (3) In the present invention, through feedback loop and adaptive optimization technology, the video generation strategy can be continuously improved in multiple rounds of feedback. This continuous optimization capability enables the advertising content to continuously adapt to the emotional changes of different users, thereby enhancing the effectiveness and competitiveness of the advertising;

[0064] (4) In the present invention, although the video content is adjusted according to the user's emotional feedback, it always revolves around the initially input advertising copy to ensure that the generated content does not deviate from the core promotional information of the advertisement. This ensures personalization while maintaining the overall strategic goal of the advertisement;

[0065] (5) In the present invention, by integrating multimodal emotional data such as facial expressions, voice features, and eye movement trajectories, the user's emotional response can be captured more comprehensively and accurately, thereby more accurately guiding the generation and adjustment of video content;

[0066] (6) In the present invention, a dynamic video generation mode based on emotional feedback is introduced, which brings a new interactive form to digital advertising. This innovative advertising form is expected to break the static limitations of traditional advertising and bring a more vivid and infectious advertising experience;

[0067] (7) In the present invention, advanced text-to-video technology is used to quickly generate initial video content that meets the theme of the advertisement, reducing the time and cost of advertisement production while providing flexible content adjustment capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0069] Figure 2 A schematic diagram of generating a video from text in an embodiment of the present invention;

[0070] Figure 3 It is a schematic diagram of feature extraction and processing flow in an embodiment of the present invention;

[0071] Figure 4 Schematic diagram of the framework of the deep learning model in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] This embodiment provides a method for generating a text-driven advertising video based on emotional feedback. Figure 1 As shown, the following steps are included

[0073] S1. The advertiser or content creator provides a piece of advertising copy T to clarify the core promotional information of the advertisement. The copy content includes key contents such as product description, target audience, marketing information, etc.

[0074] S2, such as Figure 2 As shown, the text generation model uses the latest text-generated video technology (such as Sora, CogVideoX-5b, etc.) based on the advertising copy T to convert the advertising copy T into a preliminary advertising video sequence, namely the initial advertising video V0.

[0075] S3, collecting multimodal emotional feedback data X provided by users when watching the initial advertisement video in real time through multimodal sensors; the multimodal sensors include cameras, microphones, and eye tracking devices, which are used to monitor the user's reactions in real time. The multimodal emotional feedback data include facial expression images, voice signals, and eye movement trajectory images, which are respectively denoted as X f , X v , and X g , get multimodal emotional feedback data X = {X f , X v , X g}.

[0076] S4. Based on the deep learning model, the collected multimodal emotional feedback data X is feature extracted and fused to transform it into a high-dimensional feature vector F final ;like Figure 3 As shown, it specifically includes the following steps:

[0077] S4.1. Input facial expression images, speech signals, and eye movement trajectory images into the deep learning model. Where N is the number of samples, H f , W f Represent the height and width of the facial expression image, C f Indicates the number of channels; voice signal Where T v represents the number of time steps, d v Dimensions for representing speech; eye-tracking images Among them, H g , W g Respectively represent the height and width of the eye movement track image, C g Indicates the number of channels.

[0078] S4.2. For image data X f and X g, use convolutional neural network (VGG16 or VGG19) for feature extraction, and convert it into facial expression features and eye movement trajectory features, facial expression features where d′ f represents the dimension of the extracted feature vector, which is 512 in this embodiment; the eye movement trajectory feature where d′ g represents the dimension of the extracted feature vector, which is also 512 in this embodiment.

[0079] S4.3. For the speech signal X v , and convert it into speech features using an automatic speech recognition model (wav2vec 2.0) where d′ v represents the dimension of the extracted feature vector, which is 512 in this embodiment.

[0080] S4.4. Use the Multi-Head Attention mechanism to calculate and fuse different features to obtain a high-dimensional feature vector The multi-head attention mechanism helps capture the relationship between different features. It includes the following steps:

[0081] S4.4.1. Convert facial expression features F, voice features V, and eye movement track features G into query matrix, key matrix, and value matrix respectively:

[0082]

[0083] in, Represents the weight matrix that needs to be optimized during training, which is used to project the input features into the latent space of the attention mechanism.

[0084] For the query matrix Represents the extracted facial expression feature vector, which contains N samples, and the feature dimension of each sample is d′ f ; is the linear transformation weight of the query matrix, where d a is the spatial dimension of the attention mechanism, and its size is set to 128; by multiplication, we get Query representation for representing facial expression features in attention mechanism.

[0085] For the key matrix Represents the extracted speech feature vector, which contains N samples, and the feature dimension of each sample is d′ v ; is the linear transformation weight of the key matrix; by multiplication, we get Used to represent speech features in the attention mechanism.

[0086] For the value matrix Represents the extracted speech feature vector, which contains N samples, and the feature dimension of each sample is d′ v ; is the linear transformation weight of the value matrix; by multiplication, we get Used to represent the information contribution of speech features in the attention mechanism.

[0087] For the key matrix Represents the extracted eye movement trajectory feature matrix, which contains N samples, and the feature dimension of each sample is d′ g ; is the linear transformation weight of the key matrix; by multiplication, we get Used to represent eye movement trajectory features in the attention mechanism.

[0088] For the value matrix Represents the extracted eye movement trajectory feature matrix, which contains N samples, and the feature dimension of each sample is d′ g ; is the linear transformation weight of the value matrix; by multiplication, we get Used to represent the information contribution of speech features in the attention mechanism.

[0089] According to the query matrix, key matrix and value matrix converted in step S4.4.1, the attention of F to V and G is calculated respectively, and the formula is as follows:

[0090]

[0091] For two attention matrices A F，V and A F，G Combine them by weighted average:

[0092]

[0093] Among them, α is a weight parameter that can be learned during the training process.

[0094] S4.4.2. Similar to step S4.4.1, find the attention matrices corresponding to V and G respectively. and At the same time, the mean square combination method is used to obtain the attention matrix of an attention head:

[0095]

[0096] S4.4.3. Concatenate and project the outputs of multiple attention heads:

[0097] F 融合=Concat(head1, head2,..., head h )W O

[0098] Among them, W O Represents the final projection matrix.

[0099] S4.4.4, the final fused feature is expressed as:

[0100] F final =MLP(LayerNorm(F 融合 )).

[0101] Features fused by multi-head attention mechanism It has stronger expressive power and can be used as input for subsequent tasks (such as classification or regression).

[0102] S5. According to the high-dimensional feature vector F final and the advertising copy T, dynamically adjust the current video content to make it more consistent with the user's emotional state, while ensuring that the core promotional information of the advertisement remains unchanged, and generate a new advertising video V through the following formula t+1 :

[0103] V t+1 =AdjustVideo(V t , F final , T)

[0104] Among them, V t+1 represents the new advertising video generated after the t+1th round of emotional feedback, V t represents the advertising video generated at the tth round of emotional feedback, F final Represents the high-dimensional feature vector extracted and fused from the user's emotional feedback data; T represents the initial input advertising copy.

[0105] S6. The new advertising video V t+1 For users to watch, multimodal sensors collect new multimodal emotional feedback data.

[0106] S7, through multiple rounds of feedback and optimization, the generated advertising video content gradually tends to maximize the user's emotional engagement and ensure the accurate communication of advertising information; through the adaptive optimization process, after each round of feedback, the video generation strategy parameter θ is updated t :

[0107]

[0108] Among them, θ t +1 represents the generation strategy parameter updated after the t+1th round of feedback, which is used to determine the strategy for the next round of advertising video generation; θt represents the generation strategy parameters used in the tth round of feedback to generate the current advertising video V t ; η represents the learning rate, which is used to control the step size of each gradient update, which determines the speed and stability of the optimization process; Denotes the loss function L(F final , T) is the gradient of the policy parameter θ, which is used to reflect the current generation strategy θ t Emotional feedback to users final and the response effect of advertising copy T; L(F final , T) represents the loss function, which measures the fit of the current video content to the user's emotional feedback and the consistency of the advertising copy. It is the objective function of the optimization process, which is minimized by L(F final , T) improves the effect of generating advertising videos.

[0109] S8. Repeat steps S5 to S7 until the generated advertising video content achieves an ideal optimization effect, that is, the generated video content reaches a threshold of a certain model loss value, which is determined by the trainer based on experience during the training process, or reaches a preset number of cycles.

[0110] S9, the optimization process is terminated and the final advertising video V is output final .

[0111] S10, the optimized final advertising video V final It will be put on the market and further analyzed and improved based on subsequent user feedback.

[0112] The framework of deep learning model is as follows Figure 4 As shown, the specific steps include:

[0113] M1. Input video frame processing: The input of the model is a video sequence containing multiple frames, each frame is an image in the video, and these frames are input into the TimeSformer module; the TimeSformer module analyzes the time dimension of the video frame. It will check the time relationship of each frame and identify the changes and actions of each frame in the video. Through this processing, the model can understand the dynamic content of the video, such as changes in facial expressions, continuity of actions, etc.; the TimeSformer module outputs a feature map of the time dimension, which retains the dynamic information between video frames.

[0114] M2. Extraction and fusion of multimodal features: Extract facial expression features, voice features and eye movement trajectory features respectively, and fuse these features into a feature matrix for representing the multimodal information of the video; facial expression features can represent changes in facial muscles, smiles, frowns and other information; voice features include emotional information in the text, etc. This part of the features helps the model understand the emotions or intentions expressed in the video; by tracking the movement path of the eyes, the focus of the user's attention is captured, and the eye movement trajectory features help understand which part of the video attracts the user's attention.

[0115] M3. Processing of text features: Input the advertising copy into the Batch Normalization layer, normalize the features in the advertising copy to make the data smoother, prevent gradient problems during training, and output the normalized text features.

[0116] M4. Processing of video features: The visual features of the video frame obtained in step M1 are processed by the VisionTransformer module. The VisionTransformer module is good at capturing global features and details in the image; it is based on the Transformer architecture, divides the image into patches of fixed size, and processes these patches as input to generate the final representation of the image features.

[0117] Step M4 specifically includes the following sub-steps:

[0118] M4.1. Input data: Input video dimensions (T, H, W, C), where T represents the number of frames, i.e., the time dimension; H represents the height of the frame, W represents the width of the frame, and C represents the number of channels of the frame, which is usually 3, representing RGB channels; for example, if the input video size is 16 frames, each frame size is 224x224, and is in RGB format, then the input video dimension is (16, 224, 224, 3).

[0119] M4.2, Processing steps:

[0120] Image patch embedding: The input frame is split into fixed-size image patches, each of which is of size P×P. For example, a 224x224 frame can be split into 16×16 patches, resulting in a total of (224 / 16)*(224 / 16)=14*14=196 patches. Each patch is flattened to generate a fixed-size image patch of length P. 2 The vector of C, for example, a 16×16 patch in an RGB image has a dimension of 16*16*3=768;

[0121] Linear transformation: The flattened patches are passed through a linear transformation layer and mapped to a fixed dimension D, such as 768 or 1024, which makes all patches represented as vectors with the same dimension;

[0122] Position encoding: In order to preserve the spatial information of the frame, a position encoding is added to each patch. These position encodings represent the position of the patch in the original image. The position encoding usually has the same size as the dimension after linear transformation, i.e. D, and is added to the representation of each patch.

[0123] M4.3, Transformer encoder:

[0124] Multi-Head Self-Attention: These patch embeddings are processed through a multi-head self-attention mechanism, and the relationship between them is calculated to capture global features;

[0125] Feed-Forward Network (FFN): further processes the output of self-attention;

[0126] Residual Connections: Each attention layer and feed-forward neural network layer contains residual connections to alleviate the gradient vanishing problem of deep networks;

[0127] Layer Normalization: Applied to the output of residual connections to stabilize the training process.

[0128] M4.4. Output data: After passing through the Transformer encoder, the embeddings of all patches are recombined and passed through a fully connected layer to generate the final representation of the image features, which can be fused with the outputs of other modules; the output dimension is (T, D), where T represents the time dimension, that is, the number of frames, and D represents the feature dimension, for example 1024.

[0129] M5. Multimodal feature fusion: The feature matrix used to represent the multimodal information of the video, the normalized advertising copy, and the video features flattened by the fully connected network are fused, so as to integrate multimodal information such as vision, language, eye movement trajectory, etc. with the information of the advertising copy to form a comprehensive feature and obtain a fused feature matrix.

[0130] M6, Advertising video content generation: The fused feature matrix passes through the fully connected layer, which converts the high-dimensional fused features into a format suitable for the generation task, preparing for the subsequent advertising video content generation; specifically, for the generation task, the fully connected layer adjusts the fused feature matrix to The format is, where N is the number of samples, To is the number of time steps of the output sequence, corresponding to the number of frames or time dimension of the generated video, d o The feature dimension of each time step represents the feature representation required by the generative model at each time point; and the residual network is used in the fully connected layer to enhance the network's ability to obtain and operate information;

[0131] Next, the model converts these features back to the format of video frames through a deconvolution layer, which means that the model maps the features to the pixel level through multiple deconvolution operations to generate new video frames.

[0132] M7, output generated video: After deconvolution processing, the final output is a new video sequence whose content meets the requirements of the input multimodal information and advertising copy; the model generates a customized video output by processing and converting the input video and multimodal information.

[0133] This embodiment dynamically adjusts the advertising content by collecting emotional data such as user facial expressions, voice features, and eye movement trajectories in real time to improve personalization, interactivity, and conversion rate; combines the text-generated video model with multimodal emotion recognition technology, and uses feature fusion and feedback processing to make the advertising video more in line with user emotions; multiple rounds of feedback and adaptive optimization ensure that the advertisement delivers core information while flexibly responding to the user's emotional state, greatly improving its appeal and engagement. The innovative feature fusion algorithm further improves the personalization and real-time response capabilities of the generated video, providing a new solution for the digital advertising field and promoting the development of personalization and emotional interaction.

[0134] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A text-driven advertising video generation method based on emotional feedback, characterized in that: The following steps are included S1. The advertiser or content creator provides a piece of advertising copy T to clarify the core promotional information; S2, convert the advertising copy T into the initial advertising video V0 through the text generation model; S3, collecting multimodal emotional feedback data X fed back by users when watching the initial advertising video in real time through a multimodal sensor; S4. Based on the deep learning model, the collected multimodal emotional feedback data X is feature extracted and fused to transform it into a high-dimensional feature vector F final ; S5. According to the high-dimensional feature vector F final and the advertising copy T, dynamically adjust the current video content and generate a new advertising video V t+1 ; S6. The new advertising video V t+1 For users to watch, the multimodal sensor collects new multimodal emotional feedback data; S7. Update the video generation strategy parameters θ through the adaptive optimization process t ; S8, repeating steps S5 to S7 until the generated advertising video content maximizes the user's emotional engagement or reaches a preset number of cycles; S9, the optimization process is terminated and the final advertising video V is output final ; S10, the optimized final advertising video V final It will be put on the market and further analyzed and improved based on subsequent user feedback.

2. The method for generating a text-driven advertising video based on emotional feedback according to claim 1, characterized in that: In step S2, the text generation model is set to Sora or CogVideoX-5b.

3. The method for generating a text-driven advertising video based on emotional feedback according to claim 1, characterized in that: In step S3, the multimodal sensor includes a camera, a microphone and an eye tracking device, and the multimodal emotion feedback data includes a facial expression image, a voice signal and an eye movement trajectory image, which are respectively denoted as X f , X v , and X g , get multimodal emotional feedback data X = {X f , X v , X g }.

4. The method for generating a text-driven advertising video based on emotional feedback according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S4.

1. Input facial expression images, speech signals, and eye movement trajectory images into the deep learning model. Where N is the number of samples, H f , W f Represent the height and width of the facial expression image, C f Indicates the number of channels; voice signal Where T v represents the number of time steps, d v Dimensions for representing speech; eye-tracking images Among them, H g , W g Respectively represent the height and width of the eye movement track image, C g Indicates the number of channels; S4.

2. For image data X f and X g , use convolutional neural network to extract features and convert them into facial expression features and eye movement trajectory features, facial expression features where d′ f Represents the dimension of the extracted feature vector; eye movement trajectory features where d′ g Represents the dimension of the extracted feature vector; S4.

3. For the speech signal X v , which is converted into speech features using an automatic speech recognition model where d′ v Represents the dimension of the extracted feature vector; S4.

4. Use the multi-head attention mechanism to calculate and fuse different features to obtain a high-dimensional feature vector d a Represents the dimension of each sample.

5. The method for generating text-driven advertising videos based on emotional feedback according to claim 4, characterized in that: In step S4.2, the convolutional neural network is set to VGG16 or VGG19; in step S4.3, the automatic speech recognition model is set to wav2vec 2.

0.

6. The method for generating text-driven advertising videos based on emotional feedback according to claim 4, characterized in that: The step S4.4 specifically includes the following sub-steps: S4.4.

1. Convert facial expression features F, speech features V, and eye movement track features G into query matrix, key matrix, and value matrix respectively: in, Represents the weight matrix that needs to be optimized during training, which is used to project the input features into the latent space of the attention mechanism; for the query matrix The feature dimension of each sample is d′ f , represents the linear transformation weight of the query matrix, where d a represents the dimension of the attention mechanism's factor space; for the key matrix The feature dimension of each sample is d′ v , represents the linear transformation weights of the key matrix; for the value matrix Represents the linear transformation weights of the value matrix; for the key matrix The feature dimension of each sample is d′ g , represents the linear transformation weights of the key matrix; for the value matrix Represents the linear transformation weights of the value matrix; According to the above matrix, we can calculate F's attention to V and G respectively. The formula is as follows: For two attention matrices A F，V and A F，G Combine them by weighted average: Among them, α is a weight parameter; S4.4.

2. Find the attention matrices corresponding to V and G respectively and At the same time, the mean square combination method is used to obtain the attention matrix of an attention head: S4.4.

3. Concatenate and project the outputs of multiple attention heads: F 融合 =Concat(head1,head2,…,head h )W O Among them, W O Represents the final projection matrix; S4.4.4, the final fused feature is expressed as: F final =MLP(LayerNorm(F 融合 ))。 7. The method for generating text-driven advertising videos based on emotional feedback according to claim 1, characterized in that: In step S5, a new advertisement video is generated by the following formula: V t+1 =AdjustVideo(V t ,F final ,T) Among them, V t+1 represents the new advertising video generated after the t+1th round of emotional feedback, V t represents the advertising video generated at the tth round of emotional feedback, F final Represents a high-dimensional feature vector; T represents the initial input advertising copy.

8. The method for generating text-driven advertising videos based on emotional feedback according to claim 1, characterized in that: In step S7, after each round of feedback, the video generation strategy is updated: Among them, θ t +1 represents the generation strategy parameter updated after the t+1th round of feedback, which is used to determine the strategy for the next round of advertising video generation; θ t represents the generation strategy parameters used in the tth round of feedback to generate the current advertising video V t ; η represents the learning rate, which is used to control the step size of each gradient update; Denotes the loss function L(F final , T) is the gradient of the policy parameter θ, which is used to reflect the current generation strategy θ t Emotional feedback to users final and the response effect of advertising copy T; L(F final , T) represents the loss function, which is the objective function of the optimization process. final , T) improves the effect of generating advertising videos.

9. The method for generating text-driven advertising videos based on emotional feedback according to claim 1, characterized in that: In step S4, the deep learning model includes the following working steps: M1. Input a video sequence containing multiple frames, each frame is an image in the video, input these frames into the TimeSformer module, and output a feature map of the time dimension; M2, extract facial expression features, voice features and eye movement trajectory features respectively, and fuse these features into a feature matrix for representing multimodal information of the video; M3, input the advertisement copy into the Batch Normalization layer, normalize the features in the advertisement copy, and output the normalized text features; M4, processes the visual features of the video frame obtained in step M1 through the VisionTransformer module, divides the image into patches of fixed size, and processes these patches as input to generate the final representation of the image features; M5, fusing the outputs of step M2, step M3 and step M4 to obtain a fused feature matrix; M6, the fused feature matrix passes through the fully connected layer to convert the high-dimensional fused features into a format suitable for the generation task, which is used for video content generation, and the residual network is used in the fully connected layer; then it passes through the deconvolution layer to convert these features back to the format of the video frame, and through multiple deconvolution operations, the features are mapped to the pixel level to generate a new video frame; M7, finally output a new video sequence.

10. The method for generating text-driven advertising videos based on emotional feedback according to claim 9, characterized in that: The step M4 specifically includes the following sub-steps: M4.

1. Input video dimensions (T, H, W, C), where T represents the number of frames, i.e., the time dimension; H represents the height of the frame, W represents the width of the frame, and C represents the number of channels of the frame; M4.

2. Split the input frame into fixed-size image patches, each of which is P×P in size, and flatten each patch to generate a fixed-size image patch of length P. 2 C; the flattened patch is mapped to a fixed dimension D through a linear transformation layer; a position code is added to each patch, which indicates the position of the patch in the original image. The position code has the same size as the dimension after the linear transformation, i.e. D, and is added to the representation of each patch; M4.

3. Process patch embeddings through a multi-head self-attention mechanism and calculate the relationship between them; the output of the multi-head self-attention mechanism passes through a feedforward neural network, residual connection, and layer normalization in turn; M4.

4. After passing through the Transformer encoder, the embeddings of all patches are recombined and passed through a fully connected layer to generate the final representation of the image features. The output dimension is (T, D), where T represents the time dimension, that is, the number of frames, and D represents the feature dimension.

Images