A video emotion analysis method based on cross-modal semantic association

Abstract

A video emotion analysis method based on cross-modal semantic association belongs to the field of natural language processing. In view of the cross-modal semantic gap problem existing in the subjective video data in emotion analysis, a semantic guidance mechanism is proposed. The mechanism uses contrastive learning and metric learning to narrow the distance between the emotion auxiliary modal and the emotion main modal, so that the auxiliary modal is close to the main modal, thereby strengthening the semantic association between the emotion main modal and the auxiliary modal, and alleviating the cross-modal semantic gap problem in video data. Based on the two kinds of cross-modal models of multi-flow and single-flow, the semantic guidance mechanism is introduced respectively, and the contrast experiment with the mainstream model is designed. The results show that the proposed semantic guidance mechanism can reach or even surpass the current mainstream model level. The ablation experiment of the semantic guidance mechanism is also carried out. From the experimental results, the semantic guidance mechanism can play an effectiveness in multiple models.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a video sentiment analysis method based on cross-modal semantic association. BACKGROUND

[0002] Early sentiment analysis technology refers to sentiment analysis on text content, aiming to extract and summarize information from the text so as to quickly obtain sentiment tendency such as network public opinion or product analysis report. With the development of the Internet and social platforms, more and more users like to use videos, pictures and texts to express their views or emotions on something in the network. Therefore, in recent years, more and more scholars have carried out cross-modal sentiment analysis research on videos, pictures and texts on the basis of text sentiment analysis. Simple sentiment tendency can be divided into positive, negative and neutral, and complex sentiment also includes emotional analysis such as 'anger', 'happiness' and'sadness'.

[0003] In the current cross-modal related research, whether it is video cross-modal (text, audio and visual modal) data, or picture-text task (such as picture description, picture-text retrieval, picture-text matching, etc.) data and multi-feature (such as RGB, infrared, channel, etc.) data, there is a huge difference in semantics among different modalities, that is, there is a semantic gap problem in cross-modal, and the essential reason for this problem is that there is a very obvious gap in the data distribution and shallow features of different modalities.

[0004] For the semantic gap problem existing in the video cross-modal sentiment analysis technology, specifically, on the one hand, the text modality adopts Glove or Bert related model to extract initial word level features, the audio modality needs to use a speech toolkit or a speech recognition pre-training model to extract audio pitch, spectrum and other features, and the visual modality adopts a face feature model in computer vision to extract features. Since text, audio and visual modalities use different feature extractors to obtain shallow initial features, the shallow features of each modality will have great differences in semantics, that is, the semantic gap problem. On the other hand, the video with opinions has strong subjectivity, and the text modality in it contains the most important subjective evaluation information. In the cross-modal sentiment analysis task, the text modality is regarded as the main emotion modality, and the audio and visual modalities are regarded as the auxiliary emotion modalities; not only is there a semantic gap problem between the main modality and the auxiliary modality, but also there is a lack of semantic association and guidance. If the shallow features of different modalities are simply combined together, it will bring noise to the main modality instead of assisting the main modality.

[0005] In summary, in the field of video cross-modal sentiment analysis, the existing methods have the problem of cross-modal semantic gap. Due to the lack of semantic guidance from the text sentiment main modality, the audio and visual sentiment secondary modalities are more susceptible to noise in the learning process. SUMMARY

[0006] The purpose of the present application is to address the problem of cross-modal semantic gap, provide a semantic guidance mechanism based on metric learning, use the text sentiment main modality to guide the semantics of the audio and visual sentiment secondary modalities, strengthen the semantic correlation between different modalities, thereby reducing the noise effect of the secondary modalities on the semantics of the main modality, and allowing the model to further learn the complementary feature representation of cross-modal data.

[0007] The present application comprises the following steps:

[0008] a. Extracting modal shallow features;

[0009] b. Encoding the modal shallow features through the L layers of the Transformer model to obtain the embeddings of the text modality, audio modality and visual modality;

[0010] c. For the multi-flow Transformer model, the text modality in the three modalities is taken as the sentiment main modality, and the audio and visual modalities are taken as the secondary modalities;

[0011] d. For the single-flow Transformer model, the features of the three modalities are combined as the input of the final sentiment classifier;

[0012] e. The model obtains the final cross-modal sentiment analysis result.

[0013] In step a, the modal shallow features include pitch, speech polarity, glottal closure instant and spectral envelope, etc. The specific steps of extracting the modal shallow features can be: using a pre-trained language model to extract the word vector of the text modality, using an acoustic analysis framework to extract the modal shallow features, and using a facial expression analysis toolkit to extract the facial features.

[0014] In step b, the encoding of the modal shallow features through the L layers of the Transformer model obtains the embeddings of the text modality, audio modality and visual modality. For cross-modal semantic guidance of contrastive learning, the text modality needs to be sampled at a certain proportion, and the audio modality or visual modality segments from the same video segment are taken as positive samples, and the audio modality or visual modality segments from different video segments are taken as negative samples.

[0015] In steps c, d, for the multi-stream Transformer model or for the single-stream Transformer model, the embeddings of the three modalities take the cosine distance, and the distances between the audio modality and the text modality and between the visual modality and the text modality are minimized, together with the final cross-modal sentiment analysis task as the optimization target.

[0016] In step e, the model obtains the final cross-modal sentiment analysis result, and the final classifier input of the multi-stream Transformer is the addition of the text-audio, text-visual and text-text cross-modal embeddings; the final classifier input of the single-stream Transformer is the combination of the embeddings of the three modalities.

[0017] The present application proposes a semantic guidance mechanism around the cross-modal semantic gap problem existing in subjective video data in sentiment analysis. The mechanism uses contrastive learning and metric learning to narrow the distance between the emotional secondary modality and the emotional primary modality, so that the secondary modality approaches the primary modality, thereby strengthening the semantic association between the emotional primary modality and the secondary modality, and alleviating the cross-modal semantic gap problem in video data. Based on the multi-stream and single-stream cross-modal models, the semantic guidance mechanism is introduced respectively, and comparative experiments with mainstream models are designed. The results show that the proposed semantic guidance mechanism can reach or even surpass the current mainstream model level. Ablation experiments of the semantic guidance mechanism are also conducted. From the experimental results, it can be seen that the semantic guidance mechanism can play an effectiveness in multiple models. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the cross-modal semantic guidance mechanism based on metric learning proposed by the present application.

[0019] Figure 2 is the semantic guidance mechanism of the cross-modal multi-stream Transformer model proposed by the present application.

[0020] Figure 3 is the semantic guidance mechanism structure diagram based on the cross-modal single-stream Transformer model proposed by the present application.

[0021] Figure 4 is the ablation experiment result of the semantic guidance mechanism based on the cross-modal single-stream model.

[0022] Figure 5 is the ablation experiment result of the semantic guidance mechanism based on facial expression prompt learning.

[0023] Figure 6 is the ablation experiment result of the semantic guidance mechanism based on contrastive learning.

[0024] Figure 7The method is the method provided by the application, and the specific case results of the method and other methods in the data set CMU-MOSEI test set. DETAILED DESCRIPTION

[0025] The application will be further described below with reference to the drawings.

[0026] 1. Semantic guidance mechanism

[0027] The text modality is regarded as the main modality of emotion, and the audio and visual modalities are regarded as the secondary modalities of emotion. The text main modality is used to guide the model to learn the audio and visual secondary modalities, so as to narrow the semantic gap between the secondary modalities and the main modality.

[0028] In the semantic guidance mechanism, the text emotion main modality is used to guide the late high-level semantics of the audio and visual two emotion secondary modalities, so as to reduce the noise influence of the audio and visual secondary modalities on the text main modality, and play their positive auxiliary role. On the basis of the classic Transformer model, the semantic guidance mechanism is integrated into the multi-stream Transformer and single-stream Transformer cross-modal sentiment analysis models respectively.

[0029] 1.1. Cross-modal semantic guidance based on metric learning

[0030] Whether it is a multi-stream or single-stream Transformer model, after L layers of encoding, the embeddings of the text, audio and visual three modalities can be obtained, that is, Since the text modality is taken as the main modality of emotion in the application, in the process of training the model to learn, the audio and visual two secondary modalities need to be guided to approach the text main modality. In order to pull the secondary modalities to the main modality, the method of metric learning is adopted, and the distances between the audio and text modalities and the visual and text modalities are minimized respectively. The distance loss function and the loss function of the sentiment analysis task are taken as the final optimization target.

[0031] Specifically, the cosine distance is used to measure the similarity between different modalities, and the cosine distance between the modalities that need to be pulled close is added to the final loss function. Taking the text modality and the audio modality as an example, the corresponding loss function of the text guiding the audio modality can be expressed as

[0032]

[0033] The cross-modal semantic guidance mechanism based on metric learning is shown in Algorithm 1, and the flowchart is shown in Figure 1 .

[0034]

[0035] 1.2 Cross-modal semantic guidance based on contrastive learning

[0036] The semantic guidance mechanism is based on metric learning. Cross-modal positive and negative samples are obtained by random sampling method, and then contrastive learning is used to minimize the distance between positive samples and maximize the distance between positive and negative samples.

[0037] Specifically, the cross-modal semantic guidance mechanism also takes the text modality as the main emotion modality. Before inputting data into the model, the text modality is sampled at a certain preset ratio. Audio or visual modality segments from the same video segment are considered as positive samples, and audio or visual modality segments from different video segments are considered as negative samples. Finally, the embeddings of the text, audio and visual modalities obtained by L-layer Transformer encoding are The cosine distance is still used to minimize the distance between the audio and text modalities and the distance between the visual and text modalities, which is used as an optimization target together with the cross-modal sentiment analysis task.

[0038] The cross-modal semantic guidance mechanism based on contrastive learning is shown in Algorithm 2, where S represents the size of the entire dataset, P represents the preset probability constant, random represents the random sampling operation, the subscript A, T, and V of X represent the audio, text, and visual modalities, respectively, and the subscript of A, T, and V represents the sequence number in the total number of samples, i.e. i = 1, …, S. The sequence numbers k1 and k2 obtained by random sampling are not equal to the value of i in each loop. If the current sampling is all positive samples, this method degenerates into the semantic guidance method based on cosine distance metric learning in the previous section. Therefore, this method is an optimization form of metric learning.

[0039]

[0040]

[0041] 1.3 Semantic guidance mechanism based on cross-modal multi-stream Transformer

[0042] For original video input data with three modalities (i.e. audio modality, text modality, and visual modality), first, use a shallow feature extractor to extract corresponding input features (X A , X T , X V ). Then, use three different GRUs (i.e. GRU A , GRU T , GRU V ) to obtain context single-modality information related to time for each modality, thereby obtaining enhanced features embeddings (X'A ,X' T ,X' V ). The embedding of the text modality is taken as the main modality embedding, and the attention-weighted features between the "text-text" modality, the "audio-text" modality, and the "vision-text" modality are calculated by using the self-attention mechanism within the modalities and the cross-attention mechanism across the modalities The weighted features of the text, audio, and vision modalities are fused together as the input of the classifier for the final sentiment analysis, and the classifier is trained to learn the sentiment score of the entire video clip.

[0043] In order to enable the cross-modal sentiment analysis model to learn more distinctive and complementary features from the text, audio, and vision modalities, the present application proposes a semantic guidance mechanism based on a multi-flow Transformer cross-modal sentiment analysis model, thereby reducing the noise negative impact of the audio and vision sentiment sub-modalities on the text sentiment main modality due to semantic gap.

[0044] As Figure 2 , after 3 layers of transformers, 3 embeddings can be obtained from the last layer, namely respectively represent the cross-modal embedding of the "text-audio" modality, the embedding of the text modality itself, and the cross-modal embedding of the "text-vision". Finally, they are added together as the comprehensive feature for final classification, and the comprehensive feature is mapped to the corresponding sentiment category using a fully connected layer, as shown in the following formula,

[0045]

[0046] where f represents the final classifier function.

[0047] In order to train the cross-modal multi-flow Transformer model to learn the task of sentiment analysis, a common cross-entropy loss function is used as the main loss function for sentiment classification. The semantic guidance mechanism needs to add two cross-modal distance measurement terms based on the loss function, which serves as an additional regularization for the sentiment analysis task. Specifically, the final loss function of the semantic guidance method based on the cross-modal multi-flow Transformer sentiment analysis is as follows:

[0048]

[0049] where L ce is the cross-entropy loss function. λ is a hyperparameter that controls the influence of L cms .

[0050] 1.4. Semantic guidance mechanism based on cross-modal single-stream Transformer

[0051] As shown in Figure 3 After L-layer Transformer encoding, the embedding is obtained The length of the embedding is equal to the sum of the lengths of the text, audio, and visual modalities. Since the maximum length of the three modalities is set, the embeddings corresponding to the three modalities in the position can be divided, i.e. Unlike the additive fusion method in multi-stream Transformer, the features of the three modalities are combined in the single-stream model as the input of the classifier for the cross-modal sentiment classification task, as shown in the following formula:

[0052]

[0053] where f represents the final classifier function.

[0054] To train the single-stream Transformer model for the sentiment classification task, the cross-entropy loss function is also used as the main loss function. In addition, the model needs to be trained to learn the sentiment prediction task in the single-stream Transformer. Therefore, similar to the semantic guidance mechanism based on multi-stream Transformer, the semantic guidance mechanism for single-stream Transformer adds two cross-modal terms as additional regularization on the basis of the cross-entropy loss function and the sentiment prediction task. Finally, the final loss function of the semantic guidance mechanism based on cross-modal single-stream Transformer is shown in the formula:

[0055]

[0056] where L ce is the cross-entropy loss function. λ is a hyperparameter that controls the influence of I cms .

[0057] The present application is applicable to video data, the text modality can be obtained by speech transcription, the audio modality is directly extracted from the original sound of the video, and the visual modality selects the frames of the video segment.

[0058] ​The effects of the two emotional mechanisms proposed by the present application are evaluated below on two benchmark disclosed data sets CMU-MOSI and CMU-MOSEI. Both of the two data sets are from YouTube website, and the contents are all videos with strong subjective bias, including film reviews, current affairs analysis, product reviews, etc. Each video is previously divided into several short video clips, and each video clip is annotated with an emotional score by a professional, with the score ranging from -3 to 3 ({strong negative: -3, negative: -2, weak negative: -1, neutral: 0, weak positive: +1, positive: +2, strong positive: +3). Among them, the CMU-MOSI data set has a total of 2199 short video clips, while the CMU-MOSEI is a larger data set, containing 23,454 short video clips. In addition, in this embodiment, binary classification and seven-class classification emotion analysis tasks are designed respectively. In the binary classification emotion analysis, scores with negative values are all regarded as negative emotions, i.e. negative class, and the rest of the scores are all positive class. Finally, in the evaluation of experimental results, the accuracy (Acc) and F1 value (F1) among the commonly used classification indicators are selected as the evaluation indicators.

[0059] Specific steps:

[0060] 1) Predefined emotional dictionary. Used to match the position of emotion and enhance the influence of emotional information corresponding position feature in emotion analysis

[0061] 2) Modal shallow feature extraction.

[0062] (1) For the CMU-MOSI data set, the pre-trained Glove (840B, 300D) model is used to extract the word vector of the text modality. For the audio modality, the acoustic analysis framework COVAREP is used to extract features such as pitch, speech polarity, open-close instant and spectral envelope in the audio field. For the visual modality, the facial expression analysis toolkit FACET is used to extract facial features, including facial landmarks, action units, gaze tracking, head pose, etc.

[0063] (2) For the CMU-MOSEI data set, the input of the text modality is the same as that of the CMU-MOSI data set. For audio, the Mel spectrogram is used to obtain the low-order features of the audio. For the visual modality, the pre-trained video classification convolutional neural network is used to extract low-dimensional "time-space" features of each frame of video.

[0064] (3) There are differences in the cross-modal single-flow Transformer model. Since the cross-modal single-flow model is based on roberta-base, the text modality adopts the feature extraction method in roberta-base, and the shallow feature extraction of the audio and visual modalities remains the same as described above.

[0065] 3) The shallow features are input into the cross-modal multi-stream Transformer or the cross-modal single-stream Transformer with the semantic guidance mechanism proposed in the application. In the cross-modal multi-stream Transformer model, there are three Transformers in total, and the size of each Transformer is set according to the following structure: 6 layers, 4 multi-heads, and a hidden size of 512. In the cross-modal single-stream Transformer model, the model size and the BERT series model used in this embodiment are consistent, and the number of layers and the number of heads are both 12, and the hidden size is 768.

[0066] 2. Analysis of the embodiment of the semantic guidance method

[0067] This embodiment introduces the experimental results of the semantic guidance mechanism based on the cross-modal multi-stream Transformer and single-stream Transformer models. First, the details of the data set and feature extraction are introduced, then the semantic guidance mechanism is compared with popular methods, and finally the semantic guidance ablation experiments based on multi-stream and single-stream models are introduced, as well as the embodiment analysis.

[0068] In order to verify the effectiveness of the semantic guidance mechanism proposed in the application, this embodiment compares the method proposed in the application with two advanced models, TBJE and MulT, which are also based on Transformer. The TBJE model won the championship in the ACL20: Second Multimodal Language Challenge. The MulT model is proposed by the Multimodal Laboratory of Carnegie Mellon University, and it is the first Transformer-based cross-modal sentiment analysis model and a classic model in this field.

[0069] In terms of data sets, in this embodiment, the two public benchmark data sets for cross-modal sentiment analysis mentioned above, CMU-MOSI and CMU-MOSEI, are used. The following experimental results are obtained on these two data sets. In this embodiment, two sentiment analysis tasks, binary classification and seven classification, are designed. The sentiment score range that the model needs to predict is -3 to 3 (strong negative: -3, negative: -2, weak negative: -1, neutral: 0, weak positive: +1, positive: +2, strong positive: +3). In the binary sentiment analysis, scores with negative values are all considered as negative sentiment, i.e. negative class, and the rest of the scores are positive class. Finally, in terms of evaluation indicators, the commonly used classification indicators, accuracy (Acc) and F1 value (F1), are adopted.

[0070] 2.1 Comparison Results of Semantic Guidance Methods and Popular Methods

[0071] The comparison results are shown in Table 1. In the table, Acc and F1 represent accuracy and F1 score, respectively. The subscripts indicate the number of categories in the classification task. MFT represents the cross-modal multi-stream Transformer model of the present invention. w indicates with. semanGM represents the semantic guidance mechanism based on cosine metric learning.

[0072] Table 1

[0073]

[0074] The following conclusions can be drawn from Table 1:

[0075] 1) The cross-modal multi-stream Transformer model (MFT) in this embodiment outperforms TBJE and MulT on both binary and heptasyl classification datasets. Specifically, the MFT-based semantic guidance method achieves an accuracy of 42.39% on the CMU-MOSEI heptasyl classification task, which is 1.01% and 1.74% higher than TBJE and MulT, respectively. This result demonstrates the effectiveness of the multi-stream Transformer-based semantic guidance method and optimizes its performance in the task of cross-modal sentiment analysis of videos.

[0076] 2) The semantic guidance mechanism (RB) based on the cross-modal single-stream Transformer model outperforms MFT, TBJE, and MulT in both binary and seven-class classification on two datasets. Specifically, RB with semi-GM achieves an accuracy of 44.53% on the seven-class classification task, which is 3.15%, 3.88%, and 2.14% higher than TBJE, MulT, and MFT with semi-GM, respectively. These experimental results further validate the effectiveness of the semantic guidance mechanism based on the cross-modal single-stream Transformer model.

[0077] 3) Compared to TBJE and MulT, the semantic guidance mechanism shows a significant advantage in seven-class classification tasks over two-class classification tasks. This indicates that the semantic guidance mechanism has better learning capabilities for complex tasks and can alleviate the semantic gap between audio, visual submodalities, and textual primary modalities, thereby enabling more granular cross-modal sentiment analysis.

[0078] 2.2. Results of an Implementation Based on a Cross-Modal Multi-Stream Transformer Model

[0079] To evaluate the semantic guidance method based on cross-modal multi-stream Transformer introduced in this invention, and the effectiveness of the semantic guidance mechanism based on the cross-modal multi-stream Transformer model, this embodiment trains several different cross-modal multi-stream Transformer models and compares the semantic guidance ablation experiment results based on these models in the table below. In the table, the subscripts Acc and F1 represent binary and seven-class video sentiment analysis tasks, MFT represents the cross-modal multi-stream Transformer model proposed in this invention, semanGM represents the semantic guidance mechanism, AS represents another sentiment guidance mechanism based on augmented text embedding (which will not be elaborated here), and both refers to the simultaneous use of semantic guidance and sentiment guidance methods based on augmented text embedding.

[0080] Table 2

[0081]

[0082] Based on the table above, the following conclusions can be drawn:

[0083] 1) Compared to the baseline models of the three multi-stream Transformers, the introduction of semantic guidance mechanisms or sentiment guidance mechanisms based on enhanced text embedding improved the sentiment analysis prediction performance of the models. For example, the MFT model increased the seven-class F1 score by 1.82% and 1.98% respectively on the MOSEI dataset.

[0084] 2) Incorporating both semantic guidance and sentiment guidance based on enhanced text embedding into these three baseline models simultaneously reveals a further improvement in accuracy. For example, the MFT model achieves a 41.68% F1 score for seven classifications on the MOSEI dataset. These results demonstrate that semantic guidance and sentiment guidance based on enhanced text embedding have a mutually reinforcing effect.

[0085] 3) After introducing the semantic guidance mechanism into the TBJE and MulT models, a similar increase in performance as the MFT model can be observed. Particularly on the MOSI dataset, the Acc7 of the TBJE model increased by approximately 5%. These results demonstrate that the semantic guidance mechanism proposed in this invention has a certain generalization ability in addressing the cross-modal semantic gap problem in sentiment analysis tasks.

[0086] 2.3. Results of an Example of Semantic Guidance Based on a Cross-Modal Single-Stream Model

[0087] This embodiment further explores the effectiveness of the semantic guidance mechanism, demonstrating the results of the semantic guidance mechanism based on the cross-modal single-stream Transformer model. There are two main cross-modal models: the cross-modal multi-stream Transformer model and the cross-modal single-stream Transformer model.

[0088] 2.3.1. Ablation Experiment of Semantic Guidance Mechanism for Cross-Modal Single-Stream Model

[0089] Figure 4 In the cross-modal single-stream Transformer model ( Figure 3 Based on this, we present the ablation experiment results before and after introducing a semantic guidance mechanism.

[0090] exist Figure 4 In this embodiment, the single-stream Transformer used is Roberta-base, and the model is initialized with pre-trained weights. Figure 4 The top four sets of data represent experimental results on the CMU-MOSEI dataset, while the bottom four sets represent experimental results on the CMU-MOSEI dataset. RB stands for robotic-base, SemanticGM represents a semantic guidance mechanism based on metric learning, and w and w / o represent "present" and "absent," respectively. The evaluation metrics are the accuracy and F1 score used in the classification task.

[0091] As can be seen from the figure, after introducing the semantic guidance mechanism, the model's sentiment analysis performance on both the CMU-MOSEI and CMU-MOSI datasets has been improved. On the MOSEI dataset, both binary and seven-class classification experiments showed positive results after introducing the semantic guidance mechanism. However, on the MOSI dataset, the binary classification effect was not very obvious. This may be because the MOSI dataset is relatively small (2199 video segments), leading to overfitting.

[0092] 2.3.2. Ablation Experiment of Semantic Guidance Mechanism Based on Facial Expression Prompt Learning

[0093] Figure 5 These are ablation experiments conducted on a semantic guidance mechanism based on the introduction of facial expression prompts into a single-stream Transformer model. The experiments were performed on two benchmark datasets: the top four sets are results from the CMU-MOSEI dataset, and the bottom four sets are results from the CMU-MOSEI dataset. The evaluation metrics remain the same as for classification tasks: accuracy and F1 score. Here, RBF represents the robotic-based model with facial expression prompts, and SemanticGM represents the semantic guidance mechanism based on metric learning.

[0094] The experiment, based on a single-stream Transformer architecture, introduces a facial expression-based prompting mechanism for sentiment guidance. This is then combined with a metric-learning-based semantic guidance mechanism to further explore the role of semantic guidance in cross-modal sentiment analysis. Figure 5 As can be seen, the improvement in the model's performance on both datasets after introducing the semantic guidance mechanism is not significant. This invention believes that this result is because after introducing the emotion guidance mechanism of facial expression prompt, the model has already been able to learn relatively important cross-modal interaction information related to emotion between the two different modalities of text and vision. Therefore, after further adding the method of using the text modality to guide the visual modality, the effect of the semantic guidance mechanism is relatively weak.

[0095] 2.3.3. Ablation Experiment of Semantic Guidance Mechanism Based on Contrastive Learning

[0096] Ablation experiment results based on contrastive learning-based semantic guidance mechanism are as follows: Figure 6 As shown, RB represents robotic-base, SemanComGM represents a semantic guidance mechanism based on contrastive learning, and w and w / o represent "present" and "absent" respectively. That is, light and dark colors represent the experimental results of the semantic guidance method with contrastive learning and the method without the guidance method.

[0097] from Figure 6 As can be seen, after introducing a semantic guidance mechanism based on contrastive learning on the basis of the cross-modal single-stream Transformer model, the cross-modal sentiment analysis performance of the model is slightly improved. Compared with the cross-modal single-stream Roberta-base model, it is improved by an average of 1 percentage point, and it is slightly better than the cross-modal semantic guidance mechanism based on cosine distance metric.

[0098] 2.3.4. Data Case Analysis

[0099] Using the CMU-MOSEI test dataset as an example, this paper demonstrates sentiment prediction examples of the proposed method and popular methods on specific datasets, such as... Figure 7As shown in the figure, MFT w bothGM represents two guidance mechanisms based on the cross-modal multi-stream Transformer model, including a sentiment guidance mechanism based on enhanced text embedding and a semantic guidance mechanism based on metric learning. RB w PSentiGM represents the cue-learning text sentiment guidance mechanism based on the cross-modal single-stream Transformer, and RB wSemanGM represents the metric learning semantic guidance mechanism based on the cross-modal single-stream Transformer. The predicted categories are the results of transforming the sentiment scores predicted by the neural network. After transformation, each category is an integer in the range [-3, 3], representing: (strong negative: -3, negative: -2, weak negative: -1, neutral: 0, weak positive: +1, positive: +2, strong positive: +3). Scores with underlines in the table indicate correct predictions.

[0100] Figure 7 The video clip in Figure (a) is an example of negative emotion. It is easy to see from the figure that the text modality of the video clip has strong negative emotional content, the audio modality spectrum also has relatively dramatic fluctuations, and the face in the visual modality has obvious negative facial expressions. Figure 7 The video clip in Figure (b) is a case of weak positive emotion. As can be seen from the figure, the text modality has weak positive emotional content, the audio modality's spectrum does not fluctuate much overall, and the face in the visual modality does not contain obvious facial expression features. This video clip has an objectively positive emotion.

[0101] However, for both types of data, the sentiment prediction results of the TBJE and MulT models do not match the actual situation. In contrast, the semantic guidance method based on multimodal multistream proposed in this invention, which combines sentiment guidance method (MFT w bothGM), sentiment guidance method of single-stream Transformer (RB w PSentiGM), and semantic guidance method (RB w SemanGM), can accurately predict the sentiment category.

[0102] In the CMU-MOSEI test set, samples with the same condition accounted for as high as 20.54% of the erroneous samples. These samples were corrected after the introduction of emotion guidance and semantic guidance methods. This shows that the guidance mechanism proposed in this invention can make full use of the strong cross-modal emotional information of the video data itself, and can also use the text emotion main modality to guide the audio and visual submodalities, allowing the audio and visual modalities to play an auxiliary role, alleviating the cross-modal semantic gap problem, and thus obtaining more granular classification results.

[0103] 2.4 Summary

[0104] To address the cross-modal semantic gap in sentiment analysis of subjective video data, a semantic guidance mechanism is proposed. This mechanism utilizes contrastive learning and metric learning to bring the submodal of sentiment closer to the dominant sentiment modality, thereby strengthening the semantic association between the dominant and submodal of sentiment and alleviating the cross-modal semantic gap in video data.

[0105] For the aforementioned semantic guidance mechanism, based on both multi-stream and single-stream cross-modal models, a semantic guidance mechanism was introduced, and comparative experiments were designed with mainstream models. The results show that the semantic guidance mechanism proposed in this invention can reach or even surpass the level of current mainstream models. Furthermore, ablation experiments of the semantic guidance mechanism were conducted, and the experimental results clearly demonstrate its effectiveness in multiple models.

Claims

1. A method for video emotion analysis of cross-modal semantic association, characterized in that The method comprises the following steps: a. Extracting modal shallow features; the modal shallow features include pitch, speech polarity, glottal closure instant and spectral envelope; b. The modal shallow features are encoded by a L-layer Transformer model to obtain the embeddings of the text, audio and visual modalities; for cross-modal semantic guidance of contrastive learning, the text modality is sampled at a certain proportion, and the audio or visual modality segments from the same video segment are regarded as positive samples, and the audio or visual modality segments from different video segments are regarded as negative samples; c. For a multi-stream Transformer model, the text modality among the three modalities is taken as the main emotion modality, and the audio and visual modalities are taken as the auxiliary modalities; d. For a single-stream Transformer model, the features of the three modalities are combined as the input of the final emotion classifier; For the multi-stream Transformer model or for the single-stream Transformer model, the embeddings of the three modalities adopt cosine distance, and the distances between the audio modality and the text modality and between the visual modality and the text modality are minimized, which are taken as the optimization target together with the final cross-modal emotion analysis task; e. The model obtains the final cross-modal emotion analysis result, and the final classifier input of the multi-stream Transformer is the addition of the cross-modal embeddings of text-audio, text-visual and text-text; the final classifier input of the single-stream Transformer is the combination of the embeddings of the three modalities.

2. The video sentiment analysis method for cross-modal semantic association as described in claim 1, characterized in that... In step a, the specific steps of extracting the modal shallow features are as follows: a pre-trained language model is used to extract the word vector of the text modality, an acoustic analysis framework is used to extract the modal shallow features, and a facial expression analysis toolkit is used to extract the facial features.

Images