A cross-modal video emotion analysis method based on emotion guidance

Abstract

The application relates to a cross-modal video emotion analysis method based on emotion guidance, and relates to natural language processing. Around how to deal with the problem of emotion information fading in cross-modal original data, an emotion guidance mechanism based on enhanced text embedding and an emotion guidance mechanism based on prompt learning Prompt are proposed. According to a predefined emotion dictionary, emotion content in source data is matched. The emotion guidance mechanism based on enhanced text embedding intervenes in the attention weight of the text modal emotion content. The emotion guidance mechanism based on prompt learning introduces a prompt template with an emotion slot, and through an auxiliary task of emotion prediction, the learning ability and attention of the model to emotion information are increased. The emotion guidance mechanism can reach or even exceed the current mainstream model level, and has the ability to generalize to the mainstream model. The ablation experiment results of the several emotion guidance mechanisms prove the effectiveness of each.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and in particular relates to a cross-modal video sentiment analysis method based on sentiment guidance, which improves the sentiment fading problem in video sentiment analysis tasks by introducing a sentiment guidance mechanism. Background Technology

[0002] Sentiment analysis technology originally referred to text sentiment analysis. This technology involves fields such as text mining, computational linguistics, and natural language processing, aiming to extract information from text and make inductive inferences to determine the subjective emotional polarity within the text. Among the related tasks of sentiment analysis, the most common type is determining a person's emotional inclination, ranging from simple emotional inclinations such as positive, negative, and neutral emotions, to more complex inclinations such as specific emotions like "joy," "anger," "sorrow," and "happiness."

[0003] With the development of the internet and social media, more and more users prefer to express their personal emotions by combining images, videos, and other non-textual content. Therefore, research on cross-modal sentiment analysis based on text sentiment analysis has attracted increasing attention from scholars. Especially in recent years, short videos have become widely popular across all age groups, and analyzing the sentiment orientation of videos has become a popular trend. Therefore, cross-modal video sentiment analysis technology expands the applications of traditional text sentiment analysis to a larger platform.

[0004] Video data includes not only text content but also non-textual content such as speech and images. These three types of content are respectively referred to as the text modality, audio modality, and visual modality. The research object of this invention is video data, which is a cross-modal data. Traditional sentiment analysis is based on the single-modality text data. Sentiment analysis of the text modality is relatively mature, and there are also mature works on the single-modality sentiment analysis of audio and visual data. However, the related technologies of the single modality are insufficient to support the processing and analysis of cross-modal data such as video. Just as human expression contains comprehensive information of language content, sound, and action, requiring comprehensive understanding, cross-modal data analysis also needs to consider the text, audio, and visual modalities comprehensively. Therefore, how to fuse modalities, how to perform cross-modal modeling, and how to weigh the contribution ratio of different modalities on different tasks are all key issues worthy of in-depth consideration in cross-modal sentiment analysis of video.

[0005] In the field of cross-modal sentiment analysis for videos, existing methods suffer from a problem: sentiment dilution. As the number of network layers in deep learning models gradually increases, the weight given to the strong emotional information inherent in the cross-modal data tends to become more even. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-modal video sentiment analysis method based on sentiment guidance. To address the problem of sentiment information fading in cross-modal raw data, two sentiment guidance mechanisms are proposed. In solving the problem of sentiment information diffusion, to further enhance the sentiment association between textual sentiment content and audio / visual modalities, two cross-modal sentiment guidance mechanisms are proposed: one based on enhanced text embedding, and the other based on prompt learning. The former can fully utilize the inherent sentiment information of the textual data, reducing the dissipation of textual sentiment content and strengthening the guidance of sentiment information in the textual modality on the other two modalities. The latter can enhance the model's attention to and learning ability of cross-modal sentiment associations through auxiliary sentiment prediction tasks.

[0007] This invention includes the following steps:

[0008] a. Introduce a predefined dictionary of sentiment words;

[0009] b. Extract shallow features of the modality;

[0010] c. Input the shallow modal features into a cross-modal multi-stream Transformer model with sentiment guidance mechanism or a cross-modal single-stream Transformer model to obtain sentiment analysis results.

[0011] In step a, the sentiment word dictionary is used to match the location of the sentiment, enhancing the influence of the features corresponding to the location of the sentiment information in sentiment analysis; for each video segment given text input, the sentiment word dictionary is first used to match the sentiment words in the text sequence, and then a set of Boolean values ​​with the same length as the text is used to record whether each word in the text sequence represents sentiment content, and the Boolean value of the sentiment part is marked as 1.

[0012] In step b, the specific steps for extracting shallow modal features are to extract word vectors of the text modality using a pre-trained language model, extract features such as pitch, speech polarity, glottal closure moment, and spectral envelope using an acoustic analysis framework, and extract facial features using a facial expression analysis toolkit.

[0013] In step c, the sentiment guidance mechanism for enhancing text embedding employs a cross-modal multi-stream Transformer model, which can intervene in sentiment information weights through step a. The sentiment guidance mechanism based on prompt learning employs a cross-modal single-stream Transformer model. This model introduces a prompt template with sentiment slots and dynamically learns sentiment weights through a sentiment prediction auxiliary task, increasing the model's ability to learn and focus on sentiment information. Inputting shallow modal features into either of these sentiment guidance models yields sentiment analysis results.

[0014] This invention addresses the problem of sentiment information fading in cross-modal raw data by proposing two sentiment guidance mechanisms: one based on enhanced text embedding and the other based on prompt learning. The first mechanism matches sentiment content from the source data using a predefined sentiment lexicon. The enhanced text embedding-based mechanism intervenes by adjusting the attention weights of sentiment content across text modalities. The prompt learning-based mechanism introduces a prompt template with sentiment slots, enhancing the model's ability to learn and focus on sentiment information through a sentiment prediction auxiliary task. Experiments show that the sentiment guidance mechanisms of this invention can reach or even surpass the performance of current mainstream models and have the ability to generalize to mainstream models. Ablation experiments of several sentiment guidance mechanisms demonstrate their respective effectiveness. Attached Figure Description

[0015] Figure 1 This is the overall structure diagram of the cross-modal multi-stream model for enhanced text embedding proposed in this invention.

[0016] Figure 2 This is a diagram showing the overall structure of the cross-modal single-stream Transformer emotion guidance method proposed in this invention.

[0017] Figure 3 This is an ablation experiment based on the sentiment guidance mechanism of text prompts.

[0018] Figure 4 This is an ablation experiment based on the emotion guidance mechanism of facial expression prompts. Detailed Implementation

[0019] The following embodiments will further illustrate the present invention with reference to the accompanying drawings. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. The following embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

[0020] 1. Emotional guidance mechanism

[0021] On the one hand, to enhance the strong emotional content contained in the text modality and strengthen the guiding role of the text primary modality on the audio and visual secondary modalities, this invention further proposes an emotion guidance mechanism based on enhanced text embedding. On the other hand, to strengthen the model's attention to and learning ability of cross-modal related emotions, an emotion prediction auxiliary task based on cue learning is designed, and a cross-modal emotion guidance mechanism based on cue learning Prompt is proposed by utilizing the intersection of emotional information between different modalities. Since the second emotion guidance method not only needs to capture the single temporal features of each modality, but also needs to capture the temporal correlation of the three modalities of text, audio, and vision simultaneously, this embodiment designs relevant experiments for two classic cross-modal models, multi-stream and single-stream, respectively, to verify the above two emotion guidance mechanisms.

[0022] 1.1. Sentiment Guidance Mechanism Based on Enhanced Text Embedding

[0023] The original cross-modal video data contains a lot of emotional content, such as sentiment words in the text modality, strong tones in the audio modality, and facial expressions in the visual modality. In the sentiment guidance mechanism based on enhanced text embedding, this invention mainly focuses on the sentiment content in the text modality. A predefined sentiment word dictionary is introduced to match the location of sentiment and enhance the influence of features corresponding to the location of sentiment information in sentiment analysis. For each given text input video segment, the sentiment word dictionary is first used to match the sentiment words in the text sequence. Then, a set of Boolean values ​​of the same length as the text is used to record whether each word in the text sequence represents sentiment content, and the Boolean values ​​of the sentiment part are marked as 1, as shown in the following formula:

[0024]

[0025] After labeling sentiment words using the above formula, the cross-modal multi-stream Transformer model has three Transformers. The sentiment guidance mechanism based on enhanced text embedding requires weight intervention on the text-only Transformer. Specifically, in the first layer of the text-only Transformer, attention weights are intervened using a Boolean variable, SenPos, based on the labeled sentiment position. The attention mechanism is mainly related to the Softmax operation. Therefore, to strengthen the model's attention to the sentiment component in the text modality, a certain intervention (ITV) strategy is adopted to recalculate QK. T The value of M = QK. T ∈R N*N This refers to the original QK. T If the value is obtained, then the new weight value after interference can be obtained.

[0026]

[0027] use Participating in new attention calculations:

[0028]

[0029] The core intervention (ITV) strategy, in this invention, involves using the original QK... T The value is multiplied by the amplification factor α, which directly increases the weight, as shown in the following formula:

[0030] ITV(x) = αx.

[0031] First layer The emotion-enhanced text embedding is further passed to subsequent Transformer network layers in both the "audio-text" and "visual-text" cross-modal modes. In the cross-modal Transformer, higher-level features... and The algorithm also considers the emotional information of the text modality after intervention. The process of the sentiment guidance method based on enhanced text embedding is shown in Algorithm 1.

[0032]

[0033]

[0034] 1.2. A cross-modal emotion guidance mechanism based on prompt-based learning (Prompt).

[0035] The core idea of ​​this mechanism is to introduce emotion-related auxiliary tasks and design emotion-related Prompt templates in cross-modal learning to enhance the model's ability to learn emotional content, enabling the model to learn emotional knowledge more effectively, thereby maximizing and mining the emotional information of the original data itself.

[0036] 1.2.1. Text-based sentiment guidance based on auxiliary tasks

[0037] In the text sentiment guidance mechanism based on auxiliary tasks, the sentiment content in the text modality of the input data is also required. Therefore, it is also necessary to match the sentiment part of the text sequence according to the predefined sentiment dictionary, construct a boolean variable SenPos to represent the sentiment position, mark the position of sentiment words as 1, and mark the position of non-sentiment words as 0.

[0038] After the sentiment words in the text modality are processed, an additional sentiment prediction auxiliary task is designed using the Mask Language Model (MLM). Specifically, given the input data for each video segment, for the text modality sequence (i.e., the subjective evaluation content of people in the video), referring to the MLM ratio of BERT, the marked sentiment words are replaced with the special symbol [MASK] with an 80% probability, replaced with other words in the vocabulary with a 10% probability, and left unchanged with a 10% probability.

[0039] After masking the text input sequence of the video clip according to the above proportions, the subsequent calculation process is consistent with that described in the cross-modal single-stream Transformer. Finally, after calculation by the L-layer Transformer, in addition to the cross-modal sentiment analysis task, there is an additional word-level sentiment prediction task. That is, while learning to predict sentiment scores, the model also needs to learn to predict the masked words.

[0040] Specifically, suppose the initial sentence is T = [T1, ..., T n The masked token is located at position m = [m1, ..., m]. k The sentence after the above masking process is T. masked Therefore, the process of predicting the mask word can be formalized as follows:

[0041]

[0042] The MLM loss function for this part is shown in the following equation:

[0043]

[0044] The multi-class cross-entropy loss function for cross-modal sentiment analysis is shown in the following equation:

[0045] L = L ce (y′,y)

[0046] Where y′ represents the output of the cross-modal single-stream Transformer, as shown in the following equation:

[0047]

[0048] Where f represents the final cross-modal sentiment classifier function.

[0049] The loss function in the above formula is added together with the loss function of sentiment analysis to form the final loss function, as shown in the following formula. This allows the model to learn not only sentiment scores during training, but also, to the maximum extent, the sentiment content of the data itself.

[0050] L = L ce (y′,y)+L MLM (T,θ G )

[0051] The process of the text sentiment guidance mechanism based on auxiliary tasks is shown in Algorithm 2, where [MASK] is a special identifier, random(words) means randomly sampling a word, and P1 and P2 represent fixed probability constants.

[0052]

[0053] 1.2.2. Facial Expression-Based Prompt Emotion Guidance

[0054] First, a pre-trained model is used to extract facial expression features from the video frames. This model, based on ResNet, can extract seven basic human facial expressions, including [angry, disgust, fear, happy, sad, surprise, neutral]. Assume the visual shallow feature input for each video segment is V = [V1, ..., V...]. N The model for extracting facial expressions is denoted as RMN, and the extracted facial expressions are V_express = [V_express1, ..., V_express]. N The process is shown in the equation.

[0055] V_express=RMN(V)

[0056] After extracting facial expressions, this invention takes the most frequently occurring expression in each video segment as the overall facial emotion of the video segment, denoted as max_facial_exp, as shown in the following formula:

[0057] max_facial_exp=Max(V_express)

[0058] Design a cue template with facial expression slots, and fill the slots with `max_facial_exp` as an additional text sequence containing visual modality information. This additional text sequence is then merged into the original text sequence of each video segment, forming a combined text and visual modality input sequence. An example of the template is shown below:

[0059] template="the person looks"<max_facial_exp> "

[0060] The reconstructed text input sequence, along with shallow features from the audio and visual modalities, is then fed into a cross-modal single-stream Transformer model.

[0061] The emotion guidance mechanism based on facial expression prompts is shown in Algorithm 3, where RMN represents the method for extracting facial expressions, [MASK] is a special identifier, random(words) represents randomly sampling a word, and P1 and P2 represent fixed probability constants.

[0062]

[0063] This invention is applicable to video data. The text modality can be obtained through speech transcription, the speech modality can be directly extracted from the original video audio, and the visual modality is selected from video clip frames.

[0064] The effectiveness of the two sentiment mechanisms proposed in this invention is evaluated on two publicly available benchmark datasets, CMU-MOSI and CMU-MOSEI. Both datasets are from YouTube and contain videos with strong subjective biases, including movie reviews, current affairs analysis, and product reviews. Each video is pre-segmented into several short video segments, and each segment is labeled with a sentiment score by professionals, ranging from -3 to 3 ({strong negative: -3, negative: -2, weak negative: -1, neutral: 0, weak positive: +1, positive: +2, strong positive: +3). The CMU-MOSI dataset contains 2199 short video segments, while CMU-MOSEI is a larger dataset containing 23,454 short video segments. Furthermore, in this embodiment, binary and seven-class sentiment analysis tasks are designed respectively. In the binary sentiment analysis, all scores are considered negative sentiment (negative class), and the remaining scores are considered positive. Finally, in terms of evaluating the experimental results, this embodiment selects accuracy (Acc) and F1 score (F1) from commonly used classification metrics as evaluation metrics.

[0065] The specific experimental steps are as follows:

[0066] 1) Predefined sentiment dictionary. Used to match the location of sentiment and enhance the influence of features corresponding to the location of sentiment information in sentiment analysis.

[0067] 2) Modal shallow feature extraction.

[0068] (1) For the CMU-MOSI dataset, word vectors for the text modality were extracted using a pre-trained GloVe (840B, 300D) model. For the audio modality, the acoustic analysis framework COVAREP was used to extract features in the audio domain, such as pitch, speech polarity, glottal closure moment, and spectral envelope. For the visual modality, the facial expression analysis toolkit FACET was used to extract facial features, including facial landmarks, action units, gaze tracking, and head pose.

[0069] (2) For the CMU-MOSEI dataset, the input for the text modality is the same as that for the CMU-MOSEI dataset. For audio, the Mel-plot spectrum is used to obtain low-order features of the audio. For the visual modality, a pre-trained video classification convolutional neural network is used to extract low-dimensional "temporal-spatial" features for each frame of video.

[0070] 3) The shallow features are input into the cross-modal multi-stream Transformer or cross-modal single-stream Transformer with emotion guidance mechanism proposed in this invention. In the cross-modal multi-stream Transformer model, there are three Transformers, each with the following structure: 6 layers, 4 multi-heads, and a hidden state size of 512. In the cross-modal single-stream Transformer model, the model size used in this embodiment is consistent with the BERT series models used, with 12 layers and 12 multi-heads, and a hidden state size of 768.

[0071] 2. Examples of Emotional Guidance Methods

[0072] To verify the effectiveness of the emotion guidance mechanism proposed in this invention, the method is compared with two advanced models, TBJE and MulT, both based on Transformer. The TBJE model won the ACL20: Second Multimodal Language Challenge. The MulT model, proposed by the Cross-Modal Lab at Carnegie Mellon University, is the first Transformer-based cross-modal sentiment analysis model and a classic model in this field.

[0073] 2.1. Comparison of Emotional Guidance Methods and Popular Methods

[0074] Experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets to compare the results of the sentiment guidance mechanism based on augmented text embedding (AS) with those of the TBJE and MulT models. The comparison results are shown in the table below. In the table, Acc and F1 represent accuracy and F1 score, respectively. The subscripts indicate the number of classes in the classification task. AS represents the sentiment guidance mechanism based on augmented text embedding, RB represents royalty-based, PSentiGM represents the sentiment guidance mechanism based on text cue learning, PFSentiGM represents the sentiment guidance mechanism based on facial expression cue learning, and w indicates inclusion (with).

[0075]

[0076] The following conclusions can be drawn from the table above:

[0077] 1) The multi-stream Transformer model (MFT) in this embodiment, after introducing a sentiment guidance mechanism based on enhanced text embedding, outperforms TBJE and MulT on both binary and seven-class classification datasets. Specifically, the MFT sentiment guidance mechanism based on enhanced text embedding achieves an accuracy of 40.17% on the seven-class classification task on the CMU-MOSI dataset, which is 3.93% and 6.11% higher than TBJE and MulT, respectively. This result demonstrates that the proposed sentiment guidance method based on enhanced text embedding is indeed effective in cross-modal sentiment analysis tasks and can achieve performance comparable to state-of-the-art models on these tasks.

[0078] 2) Compared with TBJE and MulT models, the sentiment guidance mechanism based on enhanced text embedding proposed in this invention shows a significant advantage in the seven-class classification task compared to the two-class classification task. This result indicates that paying more attention to sentiment information in the text modality and guiding the audio and visual modalities helps the model learn more discriminative features, thereby completing cross-modal sentiment analysis at a finer granularity.

[0079] 3) After introducing a sentiment prediction task based on text prompts, the model significantly outperformed state-of-the-art (SOTA) models (TBJE and MulT) on both the MOSEI and MOSI datasets. Furthermore, compared to sentiment guidance mechanisms based on enhanced text embeddings, the text prompt-based mechanism performed better because it increased the learning difficulty for the model, thus leading to greater improvements in sentiment classification. In addition, the results of this experiment also clearly demonstrate that artificially fixing the sentiment attention weights is less effective than increasing the difficulty of the sentiment-related learning tasks and automatically adjusting the attention weights.

[0080] 2.2. Analysis of the Emotional Guidance Results of Enhanced Text Embedding

[0081] For the first proposed sentiment guidance mechanism, namely the sentiment guidance mechanism based on enhanced text embedding, experiments were designed to evaluate its effectiveness. The model used in this guidance mechanism is a cross-modal multi-stream Transformer model, such as... Figure 1 As shown.

[0082] Several different cross-modal multistream models were trained to evaluate the effectiveness of the sentiment guidance mechanism based on enhanced text embedding, and the ablation experiment results of this mechanism are reported in the table below. This table shows the ablation experiment results before and after introducing the sentiment guidance mechanism based on three multistream Transformer models. MFT is the multistream Transformer model in this embodiment, AS represents the sentiment guidance mechanism based on enhanced text embedding, Acc and F1 represent the accuracy and F1 score of the classification task, respectively, their subscripts 2 and 7 represent binary and seven-class sentiment analysis tasks, respectively, and w indicates inclusion (with).

[0083]

[0084] Based on the table above, we can draw the following conclusions:

[0085] 1) Compared with the multi-stream Transformer baseline model (MFT) proposed in this invention, the accuracy of sentiment analysis is improved after introducing a sentiment guidance mechanism based on enhanced text embedding into the MFT. For example, on the MOSEI dataset, the F1 score for 7 categories is improved by 1.98%.

[0086] 2) Introducing the sentiment guidance mechanism based on enhanced text embedding into both the TBJE and MulT models resulted in similar increases. Particularly on the MOSI dataset, the Acc7 of the TBJE model increased by 3.5%. These results demonstrate that the sentiment guidance mechanism based on enhanced text embedding proposed in this invention can be applied to other multi-stream Transformer models and exhibits a certain degree of generalization ability in addressing the problem of sentiment information fading in cross-modal sentiment analysis tasks.

[0087] 2.3. Cross-modal affective guidance in prompt learning.

[0088] Experiments were designed to verify the effectiveness of the prompt-based emotion guidance mechanism on the CMU-MOSEI and CMU-MOSEI public video datasets. These experiments included emotion guidance ablation experiments based on text prompts and cross-modal emotion guidance ablation experiments based on facial expression prompts. These experiments were conducted using a cross-modal single-stream Transformer model, such as... Figure 2 As shown.

[0089] 2.3.1. Text Prompt Ablation Experiment

[0090] Figure 3These are the results of ablation experiments on the text-based Prompt sentiment guidance mechanism, based on a single-stream Transformer model. The experiments were also conducted on the two publicly available video datasets, CMU-MOSEI and CMU-MOSI, and the evaluation metrics remained the same as those for multi-class classification tasks: accuracy and F1 score. Here, RB represents robotic-base, and PSediGM represents the cross-modal sentiment guidance mechanism based on text-based Prompt.

[0091] from Figure 3 As can be seen, after introducing a text-based sentiment guidance mechanism on both datasets, the model's binary and seven-class classification prediction performance is significantly improved, with the most significant improvement on the CMU-MOSI dataset. These experimental results demonstrate that text-based sentiment word prediction assistance can help the model improve its ability to learn sentiment information and acquire cross-modal sentiment knowledge beneficial for video sentiment analysis.

[0092] 2.3.2. Facial Expression Prompt Ablation Experiment

[0093] The experiment verifies the effect of emotion guidance based on facial expression cue learning. It is worth noting that since the Prompt template with facial expressions designed in this embodiment also requires an additional emotion prediction auxiliary task, this part of the experiment is carried out on the basis of text cue learning.

[0094] Figure 4 The results of the ablation experiment are based on the emotion guidance mechanism of facial expression prompts. The top four sets of data in the figure are results from the CMU-MOSEI dataset, and the bottom four sets are results from the CMU-MOSEI dataset. PFSentiGM represents the cross-modal emotion guidance mechanism based on facial expression prompts. This experiment was also conducted on a single-stream Transformer model, using accuracy and F1 score as evaluation metrics.

[0095] from Figure 4 As can be seen, compared to the case without prompting, the model showed a significant improvement in performance on both datasets after adding the task of learning prompts based on facial expressions. This indicates that the emotion guidance method based on facial expression prompts can also enable the model to have a stronger learning ability and focus on emotional knowledge.

[0096] 2.4. Summary

[0097] This invention addresses the issue of sentiment information fading in cross-modal raw data by proposing two sentiment guidance mechanisms: one based on enhanced text embedding and the other based on prompt learning. Both mechanisms require a predefined sentiment lexicon to match the sentiment content in the source data. The enhanced text embedding-based mechanism intervenes in the attention weights of sentiment content in the text modality by amplifying them within a threshold (to prevent excessive values ​​from negatively impacting the probability distribution), thereby enhancing the model's focus on sentiment information in the text modality. The prompt learning-based mechanism is further subdivided into two types: text prompts and facial expression prompts from the visual modality. The key to both methods is the introduction of a prompt template with sentiment slots. Through a sentiment prediction auxiliary task, the model dynamically learns sentiment weights, increasing its ability to learn and focus on sentiment information.

[0098] For sentiment guidance mechanisms based on enhanced text embedding and those based on prompt learning, this embodiment designs experiments to evaluate their effectiveness. First, after introducing both sentiment guidance mechanisms on top of the baseline, this embodiment conducts comparative experiments with mainstream models. The results show that the sentiment guidance mechanism of this invention can reach or even surpass the level of current mainstream models and has the ability to generalize to mainstream models. Second, the ablation experiments of several sentiment guidance mechanisms demonstrate their respective effectiveness.

Claims

1. A cross-modal video sentiment analysis method based on sentiment guidance, characterized in that... Includes the following steps: a. Introduce a predefined dictionary of sentiment words; b. Extract shallow features of the modality; c. Input the shallow modal features into a cross-modal multi-stream Transformer model with sentiment guidance mechanism or a cross-modal single-stream Transformer model to obtain sentiment analysis results; The sentiment guidance mechanism for enhancing text embedding employs a cross-modal multi-stream Transformer model. This model intervenes in the sentiment information weights through step a, as follows: In the sentiment guidance mechanism based on enhanced text embedding, a predefined sentiment word dictionary is introduced to match the location of sentiment. Using the sentiment word dictionary, sentiment words in the text sequence are matched. Then, a set of Boolean values ​​with the same length as the text is used to record whether each word in the text sequence represents sentiment content, and the Boolean value of the sentiment part is marked as 1, as shown in the following formula: To enhance the model's focus on the sentiment component of text modalities, intervention was implemented. Strategy recalculation in the Transformer module The value, if used Indicates the original If the value is obtained, then the new weight value after interference is obtained. : use Participating in new attention calculations: Intervention The strategy is to use the original The value is multiplied by the amplification factor α, which directly increases the weight, as shown in the following formula: Will This is further passed to subsequent Transformer network layers in both audio-text and visual-text cross-modal applications; in the cross-modal Transformer, higher-level features... and The text modality will also be considered in relation to the emotional information after intervention; The emotion guidance mechanism based on prompt learning uses a cross-modal single-stream Transformer model. This model introduces a prompt template with emotion slots and allows the model to dynamically learn emotion weights through an auxiliary task of emotion prediction, thereby increasing the model's ability to learn and pay attention to emotion information. By inputting shallow modal features into these two sentiment guidance models, sentiment analysis results can be obtained.

2. The cross-modal video sentiment analysis method based on sentiment guidance as described in claim 1, characterized in that... In step a, the sentiment word dictionary is used to match the location of the sentiment, enhancing the influence of the features corresponding to the location of the sentiment information in sentiment analysis; for each video segment given text input, the sentiment word dictionary is first used to match the sentiment words in the text sequence, and then a set of Boolean values ​​with the same length as the text is used to record whether each word in the text sequence represents sentiment content, and the Boolean value of the sentiment part is marked as 1.

3. The cross-modal video sentiment analysis method based on sentiment guidance as described in claim 1, characterized in that... In step b, the specific steps for extracting shallow modal features are to extract word vectors of the text modality using a pre-trained language model, extract pitch, speech polarity, glottal closure moment and spectral envelope features using an acoustic analysis framework, and extract facial features using a facial expression analysis toolkit.

4. The cross-modal video sentiment analysis method based on sentiment guidance as described in claim 1, characterized in that... In step c, the sentiment guidance mechanism based on prompt learning uses a cross-modal single-stream Transformer model. This model introduces a prompt template with sentiment slots, and through the auxiliary task of sentiment prediction, the model dynamically learns sentiment weights, increasing the model's ability to learn and focus on sentiment information, as detailed below: 1) Text-based sentiment guidance based on auxiliary tasks In text sentiment guidance mechanisms based on auxiliary tasks, the sentiment content of the text modal in the input data is also required. Therefore, it is necessary to match the sentiment part of the text sequence according to a predefined sentiment dictionary and construct a Boolean variable representing the sentiment position. The positions of sentiment words are marked as 1, and the positions of non-sentiment words are marked as 0; After the sentiment words in the text modality are processed, an additional sentiment prediction auxiliary task is designed using a masked language model (MLM). Given the input data for each video segment, for the text modality sequence (i.e., the subjective evaluation content of people in the video), referring to the MLM ratio of BERT, the marked sentiment words are replaced with special markers with an 80% probability. There is a 10% probability that it will be replaced with another word in the dictionary, and a 10% probability that it will remain unchanged. After masking the text input sequence of the video clip according to the above proportions, the subsequent calculation process is consistent with the calculation process described in the cross-modal single-stream Transformer; finally, after... The computation of the Transformer layer, in addition to the cross-modal sentiment analysis task, also includes a word-level sentiment prediction task. That is, while learning to predict sentiment scores, the model also needs to learn to predict masked words. The masked language modeling loss function and the sentiment analysis loss function are added together as the final loss function, as shown in the following formula: in To model the loss function for masked language, For cross-modal sentiment analysis, the multi-class cross-entropy loss function is used. 2) Facial expression-based prompt emotion guidance First, a pre-trained model is used to extract facial expression features from the video frames. This model uses... As a foundation, facial expressions are extracted; assuming the visual shallow feature input for each video segment is... The model for extracting facial expressions is denoted as The extracted facial expressions are The process is shown in the equation. After extracting facial expressions, the most frequently occurring expression in each video clip is selected as the overall facial emotion of that video clip, denoted as... As shown in the following formula: Design a prompt template with facial expression slots. and will The additional text sequence, containing visual modality information, is filled into the slots and then merged into the original text sequence of each video segment to form a combined input sequence of text and visual modality. (Template) An example is shown in the following formula: " " The reconstructed text input sequence, along with shallow features from the audio and visual modalities, is then fed into a cross-modal single-stream Transformer model.

Images