Method for generating video emotion description model based on visual emotion driving and application thereof

Abstract

The application discloses a kind of generation method and application of video emotional description model based on visual emotion driving, and the steps of the method include:1 the construction of word table;2 build video emotional description model, including video feature extraction module, video emotional feature acquisition module, context aggregator and decoder based on LSTM network;3 train the video emotional description model on the dataset of video emotional description.The application can adaptively obtain reliable video emotional features, and drive the generation process of video description, so as to generate emotional, accurate video description.

Description

Technical Field

[0001] This invention belongs to the multimodal field and involves technologies such as computer vision, natural language processing, and artificial intelligence. Specifically, it is a method for generating a video emotion description model based on visual emotion-driven approaches. Background Technology

[0002] Effectively summarizing and describing video content in a more human-like way using natural language is one of the key issues in the field of multimodal content understanding. Although significant progress has been made in recent years, existing work often neglects the emotional factors in video descriptions, resulting in monotonous and dull sentences. Emotion plays a crucial role in human communication. With the popularity of images and short videos on social networks, research on visual sentiment analysis has received increasing attention. Video sentiment description is an emerging task, still in its early stages. Currently, there are two main types of work using emotional cues to enhance video descriptions: one is stylized description, aiming to generate descriptions with specific linguistic styles, such as romance, pride, and shame; the other is video sentiment description covering a more comprehensive range of emotional categories. This invention belongs to the latter.

[0003] In the field of visual sentiment analysis, recent representative works mainly include: 1) simultaneously separating emotional cues from image semantics, aesthetics, and low-level features to predict the dominant emotion for each image; 2) identifying emotions from specific visual elements (such as colors, objects, and faces); and 3) combining facial expressions with the global scene within the entire image for emotion recognition. However, these visual sentiment analysis methods focus on predicting a single dominant emotion, neglecting the complexity and subjectivity of emotions. Furthermore, the aforementioned works all focus on visual sentiment recognition on images; video-based visual sentiment analysis has not yet been fully explored. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a method and application for generating a video emotion description model based on visual emotion-driven approaches. The aim is to adaptively acquire reliable video emotion features and drive the video description generation process, thereby generating emotional and accurate video descriptions.

[0005] The present invention adopts the following technical solution to solve the technical problem:

[0006] The method for generating a video emotion description model based on visual emotion-driven approach, as described in this invention, is characterized by the following steps:

[0007] Step 1: Building the vocabulary list:

[0008] Obtain the dataset Set of video sentiment descriptions and the sentiment dictionary Voc EIn this dataset, any video in the dataset Set is denoted as Q, and the sentiment description sentence corresponding to video Q in the dataset Set is C; using all words of all sentiment description sentences in the dataset Set and the sentiment dictionary Voc E The vocabulary list Voc is constructed from all the words in the dictionary.

[0009] Step 2: Construct a video sentiment description model, including: a video feature extraction module, a video sentiment feature acquisition module, a context aggregator, and an LSTM-based decoder;

[0010] Step 2.1: The video feature extraction module obtains the frame-level visual features of the video Q. Among them, v n This represents the nth frame-level visual feature, where N represents the number of frames in the video Q.

[0011] Step 2.2: The video emotion feature acquisition module acquires the video emotion feature e. C :

[0012] Step 2.2.1: Use equation (1) to aggregate the frame-level visual features V of the video Q into global video features v. A ;

[0013]

[0014] Step 2.2.2: Obtain the sentiment dictionary (Voc) using a text feature extractor. E The embedding vector e of the i-th sentiment word i And according to equation (2), the probability value q of the video Q on the i-th sentiment word is obtained. i ;

[0015]

[0016] In equation (2), u q U q H q b q All of these are learning parameters in the video emotion feature acquisition module;

[0017] Step 2.2.3: Analyze the emotional dictionary (Voc). E Step 2.2 is performed on all sentiment words in the video Q to obtain the sentiment dictionary Voc. E The probability distribution on q = (q1, ..., g) i , ..., q I ), where I is the emotional dictionary (Voc) E The number of words contained;

[0018] Step 2.2.4: Sort the elements in probability distribution q in descending order to obtain the sorted probability distribution q′; set parameter K, where 1≤K≤I, and select the first K probability values ​​of the sorted probability distribution q′ to construct a candidate probability set {q′1, ..., q′}. k , …, q′ K}, obtain the embedding vector of the sentiment word corresponding to each probability value in the candidate probability set to form the candidate sentiment word set {e′1, ..., e′}. k ,…,e′ K}, where e′ k and q′ k Let e ​​represent the embedding vector of the k-th candidate sentiment word and the probability of the k-th candidate, respectively; thus, the sentiment vector e is obtained according to equation (3). C ;

[0019]

[0020] Step 2.3, Context Aggregator Processing:

[0021] Step 2.3.1: Define the current time as t and initialize t = 0;

[0022] Step 2.3.2: Use equation (4) to obtain the nth frame-level visual feature v of the video Q. n The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weight α nt And the video context vector at time t is obtained using equation (5). ;

[0023]

[0024]

[0025] In equation (4), u α U α H α b α All of these are learning parameters for the video context vector obtained from the context aggregator; tanh represents the hyperbolic tangent function; softmax represents the Softmax function; Indicates the learning parameter u α transpose;

[0026] Step 2.3.3: Use the text feature extractor to obtain the words generated at time t. Embedding matrix Among them, y j w represents the j-th word that has been generated. j It is the j-th generated word yj The embedding vector;

[0027] Using equation (6), we obtain the j-th generated word y. j Embedded vector w j The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weight β jt And use equation (7) to obtain the text context vector c at time t. t ;

[0028]

[0029]

[0030] In equation (6), u β U β H β b β These are all learned parameters in the context aggregator for obtaining the text context vector;

[0031] Step 2.3.4: Use equation (8) to obtain the nth frame-level visual feature v of the video Q. n With the j-th already generated word y j Embedded vector w j semantic relevance Thus, the correlation matrix is ​​obtained.

[0032]

[0033] In equation (8), u r U r H r b r All of these are learning parameters obtained from the context aggregator based on semantic relevance;

[0034] Step 2.3.5: Obtain the visual alignment embedding matrix using equation (9). Among them, w′ nt Let be the embedding vector of the nth visual alignment at time t;

[0035] W′ t =softmax(R) t W t (9)

[0036] Step 2.3.6: Use equation (10) to obtain the embedding vector w′ of the nth visual alignment at time t. nt The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weights θnt And the joint context vector c′ at time t is obtained using equation (11). t ;

[0037]

[0038]

[0039] In equation (10), u θ U θ H θ b θ These are all learned parameters in the context aggregator for obtaining the joint context vector;

[0040] Step 2.4, Processing of the LSTM-based decoder:

[0041] Step 2.4.1: When t = 0, use equation (12) to obtain the hidden state h of the LSTM-based decoder at time t. t When 1≤t≤T, the hidden state h of the LSTM-based decoder at time t is obtained using equation (13). t Where T is the total length of the generated sentences;

[0042] h t =FC(e c (12)

[0043]

[0044] In equation (12), FC(·) represents the operation of a fully connected layer, and LSTM(·) represents the operation of an LSTM network;

[0045] Step 2.4.2: Use equation (14) to obtain the prediction probability vector p of the video sentiment description model on the vocabulary Voc at time t. t ;

[0046] p t =softmax(W o h t (14)

[0047] In equation (14), W o Here is the weight matrix to be learned;

[0048] Step 2.4.3: Obtain the word y generated at time t according to equation (15). t ;

[0049] y t =argmax(p t (15)

[0050] In equation (15), argmax(·) represents obtaining the predicted probability vector p. t The word corresponding to the maximum value in the list;

[0051] Step 2.4.4: Assign t+1 to t, and return to step 3 to execute sequentially until t = T, thus obtaining the sentiment description sentence {y1, y2, ..., y} for video Q. T};

[0052] Step 3: Train the video sentiment description model on the video sentiment description dataset Set:

[0053] Step 3.1: Calculate the cross-entropy loss value of the video sentiment description model using equation (16).

[0054]

[0055] In equation (16), Let be the t-th word of the sentiment description sentence C corresponding to video Q in dataset Set. for Index vectors on the vocabulary list Voc;

[0056] Step 3.2: Based on the emotional dictionary (Voc) E Obtain the sentiment words contained in the sentiment description sentence C. Among them, e g Let G be the g-th sentiment word contained in the sentiment description sentence C, and let G be the total number of sentiment words contained in the sentiment description sentence C.

[0057] The sentiment indication loss of the video sentiment description model is calculated using equation (17).

[0058]

[0059] In equation (17), δ(e g ) for e g In the emotional dictionary Voc E Index vectors on;

[0060] Step 3.3: Randomly select another video Q' from the dataset Set, and follow the steps from 2.1 to 2.3.4 to obtain another correlation matrix. in, For the nth frame-level visual feature of video Q' and the jth generated word y j Embedded vector w j semantic relevance;

[0061] The fact-contrast loss value of the video sentiment description model is calculated using equation (18).

[0062]

[0063] In equation (18), σ(·) represents the Sigmoid function;

[0064] Step 3.4: Calculate the total loss value of the video sentiment description model using equation (19).

[0065]

[0066] In equation (19), λ ce , λ cls and λ ctr The hyperparameters are set;

[0067] Step 3.5: Calculate the total loss of the video sentiment description model on the video sentiment description dataset Set using stochastic gradient descent. Perform optimization to solve the problem, so that The goal is to minimize this minimization, thereby obtaining the optimal model on the video sentiment description dataset.

[0068] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the generation method, and the processor is configured to execute the program stored in the memory.

[0069] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the generation method.

[0070] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0071] 1. This invention fully utilizes publicly available psychological sentiment dictionaries to explicitly model the sentiment distribution of videos, thereby obtaining reliable video sentiment features. Driven by sentiment cues, the video sentiment description model can effectively predict correct sentiment words, thus improving the diversity of generated video sentiment descriptions.

[0072] 2. This invention proposes a context aggregator that aggregates visual context, textual context, and joint context at each time step, providing rich contextual information for the video description generation process, thereby enhancing the model's contextual reasoning ability and enabling the video sentiment description model to generate high-quality video sentiment descriptions.

[0073] 3. This invention proposes a joint training method for sentiment-fact coordination, involving two loss functions: sentiment indication loss function and fact contrast loss function. These functions penalize errors in sentiment prediction and video-text fact relevance, respectively, thereby improving the accuracy of video sentiment description. Attached Figure Description

[0074] Figure 1 This is a flowchart of the method for generating a video emotion description model based on visual emotion driving according to the present invention. Detailed Implementation

[0075] In this embodiment, as Figure 1 As shown, a method for generating a video sentiment description model based on visual emotion-driven approaches includes: 1. constructing a vocabulary; 2. constructing a video sentiment description model, including a video feature extraction module, a video sentiment feature acquisition module, a context aggregator, and an LSTM-based decoder; 3. training the video sentiment description model on a video sentiment description dataset; specifically, it is performed according to the following steps:

[0076] Step 1: Building the vocabulary list:

[0077] Obtain the dataset Set of video sentiment descriptions and the sentiment dictionary Voc E In this dataset, any video in the dataset Set is denoted as Q, and the sentiment description sentence corresponding to video Q in the dataset Set is denoted as C. In this embodiment, the EmVidCap and EmVidCap-S video sentiment description datasets are used. All words in all sentiment description sentences in the dataset Set and the sentiment dictionary Voc are utilized. E The vocabulary list Voc is constructed from all the words in the dictionary.

[0078] Step 2: Construct a video sentiment description model, including: a video feature extraction module, a video sentiment feature acquisition module, a context aggregator, and an LSTM-based decoder;

[0079] Step 2.1: The video feature extraction module obtains the frame-level visual features of video Q. Among them, v n This represents the nth frame-level visual feature, where N represents the number of frames in video Q; in this embodiment, N = 30.

[0080] Step 2.2: The video emotion feature acquisition module acquires the video emotion feature e. C :

[0081] Step 2.2.1: Use equation (1) to aggregate the frame-level visual features V of video Q into global video features v. A ;

[0082]

[0083] Step 2.2.2: Obtain the sentiment dictionary (Voc) using a text feature extractor. E The embedding vector e of the i-th sentiment word i And according to equation (2), the probability value q of video Q on the i-th sentiment word is obtained. i ;

[0084]

[0085] In equation (2), u q U q H q b q These are all learning parameters from the video sentiment feature acquisition module;

[0086] Step 2.2.3: Analyze the emotional dictionary (Voc). E Step 2.2 is performed on all sentiment words to obtain the video Q in the sentiment dictionary Voc. E The probability distribution q = (q1, ..., q) on the given surface. i , ..., q I ), where I is the emotional dictionary (Voc) E The number of words contained;

[0087] Step 2.2.4: Sort the elements in probability distribution q in descending order to obtain the sorted probability distribution q′; set parameter K, where 1≤K≤I, and select the first K probability values ​​of the sorted probability distribution q′ to construct a candidate probability set {q′1, ..., q′}. k , …, q′ K}, obtain the embedding vector of the sentiment word corresponding to each probability value in the candidate probability set to form the candidate sentiment word set {e′1, ..., e′}. k ,…,e′ K}, where e′ k and q′ k Let e ​​represent the embedding vector of the k-th candidate sentiment word and the probability of the k-th candidate, respectively; thus, the sentiment vector e is obtained according to equation (3). C ;

[0088]

[0089] Step 2.3, Context Aggregator Processing:

[0090] Step 2.3.1: Define the current time as t and initialize t = 0;

[0091] Step 2.3.2: Use equation (4) to obtain the nth frame-level visual feature v of video Q. n The hidden state h of the LSTM-based decoder at time t-1 t-1Attention weight α nt And the video context vector at time t is obtained using equation (5). ;

[0092]

[0093]

[0094] In equation (4), u α U α H α b α These are all learned parameters in the context aggregator for obtaining video context vectors; tanh represents the hyperbolic tangent function; softmax represents the softmax function; Indicates the learning parameter u α transpose;

[0095] Step 2.3.3: Use a text feature extractor to obtain the words generated at time t. Embedding matrix Among them, y j w represents the j-th word that has been generated. j It is the j-th generated word y j The embedding vector;

[0096] Using equation (6), we obtain the j-th generated word y. j Embedded vector w j The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weight β jt And use equation (7) to obtain the text context vector c at time t. t ;

[0097]

[0098]

[0099] In equation (6), u β U β H β b β These are all learned parameters in the context aggregator for obtaining the text context vector;

[0100] Step 2.3.4: Use equation (8) to obtain the nth frame-level visual feature v of video Q. n y with the j-th already generated word j Embedded vector w j semantic relevance Thus, the correlation matrix is ​​obtained.

[0101]

[0102] In equation (8), u r U r H r b r These are all learned parameters obtained from the context aggregator based on semantic relevance;

[0103] Step 2.3.5: Obtain the visual alignment embedding matrix using equation (9). Among them, w′ nt Let be the embedding vector of the nth visual alignment at time t;

[0104] W′ t =softmax(R) t W t (9)

[0105] Step 2.3.6: Use equation (10) to obtain the embedding vector w′ of the nth visual alignment at time t. nt The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weights θ nt And the joint context vector c′ at time t is obtained using equation (11). t ;

[0106]

[0107]

[0108] In equation (10), u θ U θ H θ b θ These are all learned parameters in the context aggregator for obtaining the joint context vector;

[0109] Step 2.4, Processing of the LSTM-based decoder:

[0110] Step 2.4.1: When t = 0, use equation (12) to obtain the hidden state h of the LSTM-based decoder at time t. t When 1≤t≤T, the hidden state h of the LSTM-based decoder at time t is obtained using equation (13). t Where T is the total length of the generated sentences;

[0111] h t =FC(e c (12)

[0112]

[0113] In equation (12), FC(·) represents the operation of a fully connected layer, and LSTM(·) represents the operation of an LSTM network;

[0114] Step 2.4.2: Use equation (14) to obtain the prediction probability vector p of the video sentiment description model on the vocabulary Voc at time t. t ;

[0115] p t =softmax(W o h t (14)

[0116] In equation (14), W o Here is the weight matrix to be learned;

[0117] Step 2.4.3: Obtain the word y generated at time t according to equation (15). t ;

[0118] y t =argmax(p t (15)

[0119] In equation (15), argmax(·) represents the process of obtaining the predicted probability vector p. t The word corresponding to the maximum value in the list;

[0120] Step 2.4.4: Assign t+1 to t, and return to step 3 to execute sequentially until t = T, thus obtaining the sentiment description sentence {y1, y2, ..., y} for video Q. T};

[0121] Step 3: Train the video sentiment description model on the video sentiment description dataset Set:

[0122] Step 3.1: Calculate the cross-entropy loss value of the video sentiment description model using equation (16).

[0123]

[0124] In equation (16), Let be the t-th word of the sentiment description sentence C corresponding to video Q in dataset Set. for Index vectors on the vocabulary list Voc;

[0125] Step 3.2: Based on the emotional dictionary (Voc) E Extract the sentiment words contained in the sentiment description sentence C. Among them, e gLet G be the g-th sentiment word contained in the sentiment description sentence C, and let G be the total number of sentiment words contained in the sentiment description sentence C.

[0126] The sentiment indication loss of the video sentiment description model is calculated using equation (17).

[0127]

[0128] In equation (17), δ(e g ) for e g In the emotional dictionary Voc E Index vectors on;

[0129] Step 3.3: Randomly select another video Q' from the dataset Set, and follow the steps from 2.1 to 2.3.4 to obtain another correlation matrix. in, For the nth frame-level visual feature of video Q' and the jth generated word y j Embedded vector w j semantic relevance;

[0130] The fact-contrast loss value of the video sentiment description model is calculated using equation (18).

[0131]

[0132] In equation (18), σ(·) represents the Sigmoid function;

[0133] Step 3.4: Calculate the total loss value of the video sentiment description model using equation (19). :

[0134]

[0135] In equation (19), λ ce , λ cls and λ ctr The hyperparameter is set; in this embodiment, λ ce =1,λ cls =0.2, λ ctr =1;

[0136] Step 3.5: Calculate the total loss of the video sentiment description model on the video sentiment description dataset Set using stochastic gradient descent. Perform optimization to solve the problem, so that The goal is to minimize this minimization, thereby obtaining the optimal model on the video sentiment description dataset.

[0137] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described generation method, and the processor is configured to execute the program stored in the memory.

[0138] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above-described generation method.

Claims

1. A method for generating a video emotion description model based on visual emotion-driven approaches, characterized by: Follow these steps: Step 1: Building the vocabulary list: Obtain the dataset Set of video sentiment descriptions and the sentiment dictionary Voc E In this dataset, any video in the dataset set is denoted as Q, and the sentiment description sentence corresponding to video Q in the dataset set is C; using all words of all sentiment description sentences in the dataset set and the sentiment dictionary Voc E The vocabulary list Voc is constructed from all the words in the dictionary. Step 2: Construct a video sentiment description model, including: a video feature extraction module, a video sentiment feature acquisition module, a context aggregator, and an LSTM-based decoder; Step 2.1: The video feature extraction module obtains the frame-level visual features of the video Q. Among them, v n This represents the nth frame-level visual feature, where N represents the number of frames in the video Q. Step 2.2: The video emotion feature acquisition module acquires the video emotion feature e. C : Step 2.2.1: Use equation (1) to aggregate the frame-level visual features V of the video Q into global video features v. A ; Step 2.2.2: Obtain the sentiment dictionary (Voc) using a text feature extractor. E The embedding vector e of the i-th sentiment word i And according to equation (2), the probability value q of the video Q on the i-th sentiment word is obtained. i ; In equation (2), u q u q H q b q All of these are learning parameters in the video emotion feature acquisition module; Step 2.2.3: Analyze the emotional dictionary (Voc). E Step 2.2 is performed on all sentiment words in the video Q to obtain the sentiment dictionary Voc. E The probability distribution q = (q1, ..., q) on the given surface is given. i ,…,q I ), where I is the emotional dictionary (Voc) E The number of words contained; Step 2.2.4: Sort the elements in probability distribution q in descending order to obtain the sorted probability distribution q′; set parameter K, where 1≤K≤I, and select the first K probability values ​​of the sorted probability distribution q′ to construct a candidate probability set {q′1,…,q′}. k ,…,q′ K }, obtain the embedding vector of the sentiment word corresponding to each probability value in the candidate probability set to form the candidate sentiment word set {e′1,…,e′}. k ,…,e′ K }, where e′ k and q′ k Let e ​​represent the embedding vector of the k-th candidate sentiment word and the probability of the k-th candidate, respectively; thus, the sentiment vector e is obtained according to equation (3). C ; Step 2.3, Context Aggregator Processing: Step 2.3.1: Define the current time as t and initialize t = 0; Step 2.3.2: Use equation (4) to obtain the nth frame-level visual feature v of the video Q. n The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weight α nt And the video context vector at time t is obtained using equation (5). In equation (4), u α U α H α b α All of these are learning parameters for the video context vector obtained from the context aggregator; tanh represents the hyperbolic tangent function; softmax represents the Softmax function; Indicates the learning parameter u α transpose; Step 2.3.3: Use the text feature extractor to obtain the words generated at time t. Embedding matrix Among them, y j w represents the j-th word that has been generated. j It is the j-th generated word y j The embedding vector; Using equation (6), we obtain the j-th generated word y. j Embedded vector w j The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weight β jt And use equation (7) to obtain the text context vector c at time t. t ; In equation (6), u β U β H β b β These are all learned parameters in the context aggregator for obtaining the text context vector; Step 2.3.4: Use equation (8) to obtain the nth frame-level visual feature v of the video Q. n With the j-th already generated word y j Embedded vector w j semantic relevance Thus, the correlation matrix is ​​obtained. In equation (8), u r U r H r b r All of these are learning parameters obtained from the context aggregator based on semantic relevance; Step 2.3.5: Obtain the visual alignment embedding matrix using equation (9). Among them, w′ nt Let be the embedding vector of the nth visual alignment at time t; W′ t =softmax(R t )W t (9) Step 2.3.6: Use equation (10) to obtain the embedding vector w′ of the nth visual alignment at time t. nt The hidden state h of the LSTM-based decoder at time t-1 t-1 Attention weights θ nt And the joint context vector c′ at time t is obtained using equation (11). t ; In equation (10), u θ U θ H θ b θ These are all learned parameters in the context aggregator for obtaining the joint context vector; Step 2.4, Processing of the LSTM-based decoder: Step 2.4.1: When t = 0, use equation (12) to obtain the hidden state h of the LSTM-based decoder at time t. t When 1≤t≤T, the hidden state h of the LSTM-based decoder at time t is obtained using equation (13). t Where T is the total length of the generated sentences; h t =FC(e c ) (12) In equation (12), FC(·) represents the operation of a fully connected layer, and LSTM(·) represents the operation of an LSTM network; Step 2.4.2: Use equation (14) to obtain the prediction probability vector p of the video sentiment description model on the vocabulary Voc at time t. t ; p t =softmax(W o h t ) (14) In equation (14), W o Here is the weight matrix to be learned; Step 2.4.3: Obtain the word y generated at time t according to equation (15). t ; y t =argmax(p t ) (15) In equation (15), argmax(·) represents obtaining the predicted probability vector p. t The word corresponding to the maximum value in the list; Step 2.4.4: Assign t+1 to t, and return to step 3 to execute sequentially until t = T, thus obtaining the sentiment description sentence {y1, y2, ..., y} for video Q. T }; Step 3: Train the video sentiment description model on the video sentiment description dataset Set: Step 3.1: Calculate the cross-entropy loss value of the video sentiment description model using equation (16). In equation (16), Let be the t-th word of the sentiment description sentence C corresponding to video Q in dataset Set. for Index vectors on the vocabulary list Voc; Step 3.2: Based on the emotional dictionary (Voc) E Obtain the sentiment words contained in the sentiment description sentence C. Among them, e g Let G be the g-th sentiment word contained in the sentiment description sentence C, and let G be the total number of sentiment words contained in the sentiment description sentence C. The sentiment indication loss of the video sentiment description model is calculated using equation (17). In equation (17), δ(e g ) for e g In the emotional dictionary Voc E Index vectors on; Step 3.3: Randomly select another video Q' from the dataset Set, and follow the steps from 2.1 to 2.3.4 to obtain another correlation matrix. in, For the nth frame-level visual feature of video Q' and the jth generated word y j Embedded vector w j semantic relevance; The fact-contrast loss value of the video sentiment description model is calculated using equation (18). In equation (18), σ(·) represents the Sigmoid function; Step 3.4: Calculate the total loss value of the video sentiment description model using equation (19). In equation (19), λ ce , λ cls and λ ctr The hyperparameters are set; Step 3.5: Calculate the total loss of the video sentiment description model on the video sentiment description dataset Set using stochastic gradient descent. Perform optimization to solve the problem, so that The goal is to minimize this minimization, thereby obtaining the optimal model on the video sentiment description dataset.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the generation method of claim 1, wherein the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the generation method of claim 1.

Images