A cross-modal video emotion information analysis method

Abstract

The present invention discloses a method for parsing cross-modal video emotion information, which belongs to the field of multimedia video understanding technology. The method comprises the following steps: (1) a single-modal feature extraction module, which captures the semantic information unique to each modality through temporal modeling; (2) an uncertainty estimation module, which performs uncertainty modeling on different modalities to obtain an uncertainty distribution representation of each modality; (3) an uncertainty fusion module, which integrates multimodal information according to an uncertain probability fusion method to reduce the uncertainty of cross-modal fusion; and (4) a sentiment classification module, which outputs the final sentiment classification result through a fully connected network layer.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia video understanding, and in particular relates to a cross-modal video emotion information parsing method. Background Art

[0002] With the development of modern multimedia technology, people are increasingly engaging in personal activities on social platforms, expressing their opinions and emotions through images with text or short videos. This type of multimodal data presents both opportunities and challenges for sentiment analysis. On the one hand, multimodal data provides more information, complementing unimodal information and helping us analyze sentiment more accurately. For example, if a speaker expresses a sarcastic opinion in text, the text alone might interpret it as positive, but if the facial expressions and tone of voice are combined, it can be accurately identified as negative. On the other hand, these modal data are often misaligned and heterogeneous, which poses challenges in integrating information from different modalities.

[0003] Current methods for cross-modal sentiment fusion mainly focus on intra-modal interaction and inter-modal interaction: (1) Intra-modal interaction, which is manifested in the temporal dynamics within a single sequence. For example, a speaker may repeat or pause too much in a speech, which results in semantic inconsistencies. For intra-modal interaction, a recurrent neural network-based temporal dynamic modeling is used to obtain intra-modal interaction. (2) Inter-modal interaction, which is manifested in the semantic association between different modalities. These temporal information interacts at the same or different time moments. For inter-modal interaction, there are two main approaches. One is to divide multimodal data into sequences of the same time length and perform multi-modal information fusion at each time moment. The other is to extract features for each modal sequence separately, and finally perform fusion of different modal information at the feature level. A search of existing patents and related technologies found that the existing methods related to cross-modal video sentiment analysis are:

[0004] (1) Chen Qiaohong, Sun Jiajin, Lou Yangbo. Multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion [P]. Zhejiang Province: CN114694076A, 2022-07-01. A multimodal sentiment analysis method based on multi-task learning and stacked cross-modal fusion is proposed. The method first uses a single-modal feature extractor to learn hidden layer features, then feeds the learned hidden layer features into a stacked cross-modal fusion module to further extract high-level features, and finally concatenates the high-level features with the single-modal features for sentiment classification.

[0005] (2) Li Hengyu, Xue Hui. A non-aligned temporal multimodal sentiment analysis method based on deep kernel mapping network [P]. Jiangsu Province: CN114626456A, 2022-06-14. A non-aligned temporal multimodal sentiment analysis method based on deep kernel mapping network. This method first uses a deep kernel mapping network to extract temporal features of multimodal data, then uses the DK-Transformer model to perform fine-grained semantic alignment of multimodal data, and fuses multimodal features through a bidirectional gated recurrent unit. Finally, the multimodal features are used for sentiment classification.

[0006] (3) Cen Jinglun, Li Zhipeng, Qingchunmei, Luo Wanxiang. A method and system for multimodal emotion classification of audio and video [P]. Guangdong Province: CN113408385B, 2022-06-14. A method and system for multimodal emotion classification of audio and video is proposed. The system consists of three network branches: an image feature extraction network consisting of a Vit network, a path integral network, and an LSTM network; an audio feature extraction network consisting of a Con-LSTM network and a Sig module; and a text feature extraction network consisting of BERT. The system extracts features from the three modalities and fuses them through a tensor fusion network. Finally, the emotion classification is obtained through a fully connected layer.

[0007] However, there are two problems with the above methods: (1) In the natural state, there may be noise in the data collection process. One is that the data itself has noise. For example, if the speaker is in a noisy open environment, the audio will be mixed with the surrounding sounds, which will hinder the sentiment analysis; the other is artificial noise. For example, when adding text to a video or picture, there may be spelling errors. In addition, there is also varying degrees of information loss during the data transmission process. (2) The amount of information contained in different modalities is different, and these modal information cannot be regarded as equally important. For example, in sentiment analysis tasks, text is usually the main carrier of emotion, while images and audio serve as auxiliary modalities.

[0008] Therefore, to address these key issues, this paper proposes a cross-modal video emotion information parsing method. This method innovatively designs an uncertainty estimation module to independently estimate the mean and variance for each sample's different modalities, thereby modeling uncertainty. Finally, the method automatically weighs the information from different modalities based on uncertainty probabilities, thereby overcoming the noise interference introduced by these modalities and making the model more versatile and robust. Summary of the Invention

[0009] The purpose of the present invention is to provide a cross-modal video emotion information analysis method, which can effectively solve the technical problems of cross-modal feature fusion, reduce the noise brought by different modalities, and make emotion analysis more accurate and stable.

[0010] The present invention is achieved through the following technical solutions:

[0011] 1. A cross-modal video emotion information analysis method, comprising the following steps:

[0012] Step 1: Process the original video data and extract the feature sequence for model input:

[0013] (1) Use the OpenFace toolkit to process the video image and extract the facial feature sequence x v , the feature dimension of each video image is 20;

[0014] (2) Use the pre-trained BERT model to obtain the text sequence x for the text data in the video t , the feature dimension of each word is 768;

[0015] (3) Use the COVAREP toolkit to extract the feature sequence x from the audio signal in the video a , the feature dimension of each audio information is 5;

[0016] (4) The three extracted feature sequences are used to construct a dataset X: Among them, m=v,t,a means that the modality m has three values: v, t, a, where v, t, a represent the three modalities of video, text, and audio respectively. m and D m Respectively represent the sequence length and feature dimension from modality m;

[0017] (5) Divide the constructed dataset X into 8:1:1 to obtain the training set, test set, and validation set;

[0018] Step 2: Build a cross-modal sentiment prediction network model:

[0019] (1) Construct a single-modal feature extraction module: Take the constructed dataset X as input, use the bidirectional long short-term memory network biLSTM to model the three modal features, learn the temporal changes within each modality, and use the final state output as the feature representation. The formula is as follows:

[0020] h m =biLSTM(x m ,θ m )

[0021] Among them, x m is the sequence feature of the input modality m, θ m is the corresponding neural network parameter, h m is a modal feature;

[0022] (2) Construct uncertainty estimation module:

[0023] The above three modal features h m As input, in order to achieve feature alignment, a multi-layer perceptron (MLP) is used for feature transformation to map modal features of different dimensions into a unified feature space:

[0024] f m =MLP(h m , w m )

[0025] Among them, w m are the network parameters of the multilayer perceptron, f m is the mapped feature vector, and the number of hidden layer neural units of the multilayer perceptron is 128;

[0026] The data of the modal sequence features are regarded as being sampled from different Gaussian distributions, that is, in, Indicates that the mean value is u m , with variance σ m 2 Gaussian distribution. Each mode has its own distribution, um represents the mean, which is regarded as the sample mode itself, and σ m 2 Represents the variance, which is regarded as modal noise. In order to model the uncertainty distribution, two fully connected layers FC1 and FC2 are used to estimate the mean u of the mode m respectively. m and variance σ m :

[0027] u m =FC1(f m , w1)

[0028] σ m 2 =FC2(f m , w2)

[0029] Among them, w1 and w2 are the learnable parameters of the fully connected layers FC1 and FC2 respectively;

[0030] Although the fully connected layer can estimate the variance σ m 2 , but in the process of network model training, due to the small variance σ m 2 This will cause the model to degenerate into a deterministic representation; to solve this problem, a regularization term is added to constrain the parameter distribution to be approximately normal through KL divergence:

[0031]

[0032] in, is the loss calculated by KL divergence, u m，i represents the mean u m The i-th dimension of , D represents the vector dimension, is set to 128;

[0033] In order to estimate the variance σ m 2 The distribution effectively captures the information of each mode and constrains it through data reconstruction, thereby minimizing the error between input and output. Since the sampling process is not differentiable, parameterized resampling is introduced for back propagation. The specific process is as follows:

[0034] z m =u m +∈·σ m 2 ,∈∈(0,I)

[0035]

[0036] Among them, z m represents the resampled features, ∈ represents the sampling noise of the normal distribution, 0 and 1 are vectors of all 0 and all 1 respectively, and the reconstruction loss The absolute mean error was used for calculation;

[0037] (3) Constructing the uncertain fusion module:

[0038] The mean u of the distribution estimated based on uncertainty m and variance σ m 2 , perform uncertain fusion, first calculate the image mean u v and image variance σ v 2 And the text mean u t and text variance σ t 2 The fusion of the two modal distributions is then combined with the audio mean u a and audio variance σ a 2 Distributed fusion; the specific process is as follows:

[0039]

[0040]

[0041]

[0042]

[0043] Among them, u vt and σ vt2 is the distribution after the fusion of image and text modalities, u vta and σ vta 2 is the final distribution obtained by fusing the three modalities;

[0044] (4) Constructing the sentiment prediction module:

[0045] During the training phase, the obtained cross-modal fusion results are input into the fully connected layer FC reg Perform sentiment classification, output the regressed sentiment score, and use the mean absolute error as the loss function; the specific process is as follows:

[0046] O m =FC reg (u vta , w class )

[0047]

[0048] Among them, w class is the network parameter of the fully connected layer, O m is the predicted sentiment score, g m is the true sentiment score of the sample, is the calculated error loss;

[0049] Step 3: Train and optimize the network model on the training set:

[0050] The overall loss function that needs to be optimized is:

[0051]

[0052] Step 4: Verify and test the network model on the validation set and test set:

[0053] During the validation and testing phases, the predicted sentiment scores were segmented and limited to [-3, 3]. The sentiment scores were rounded to the nearest integer, resulting in the following categories: very negative, negative, slightly negative, neutral, slightly positive, positive, and very positive.

[0054] O class =round(O m )

[0055] Among them, the function round() is a rounding operation, class It is the rounded result, which represents the category finally predicted by the network model.

[0056] Compared with the prior art, the advantages and effects of the present invention are:

[0057] (1) The present invention designs an uncertainty estimation module that dynamically estimates the mean and variance for different modalities, thereby modeling the uncertainty of the distribution. (2) The present invention proposes an uncertainty fusion module that automatically weighs the information of different modalities based on uncertainty probability fusion. (3) The present invention has flexible scalability and can be adapted to other cross-modal fusion fields, especially in the case of low-quality data and semantic ambiguity, with good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is an overall flow chart of the method of the present invention,

[0059] Figure 2 This is a structural diagram of the network model proposed in this invention. DETAILED DESCRIPTION

[0060] The present invention will be described in further detail below with reference to the accompanying drawings.

[0061] like Figure 1 As shown, the present invention proposes a cross-modal video emotion information analysis method, which consists of the following steps:

[0062] 1. A cross-modal video emotion information analysis method, comprising the following steps:

[0063] Step 1: Process the original video data and extract the feature sequence for model input:

[0064] (1) Use the OpenFace toolkit to process the video image and extract the facial feature sequence x v , the feature dimension of each video image is 20;

[0065] (2) Use the pre-trained BERT model to obtain the text sequence x for the text data in the video t , the feature dimension of each word is 768;

[0066] (3) Use the COVAREP toolkit to extract the feature sequence x from the audio signal in the video a , the feature dimension of each audio information is 5;

[0067] (4) The three extracted feature sequences are used to construct a dataset X: Among them, m=v,t,a means that the modality m has three values: v, t, a, where v, t, a represent the three modalities of video, text, and audio respectively. m and D m Respectively represent the sequence length and feature dimension from modality m;

[0068] (5) Divide the constructed dataset X into 8:1:1 to obtain the training set, test set, and validation set;

[0069] Step 2: Build a cross-modal sentiment prediction network model:

[0070] (1) Construct a single-modal feature extraction module: Take the constructed dataset X as input, use the bidirectional long short-term memory network biLSTM to model the three modal features, learn the temporal changes within each modality, and use the final state output as the feature representation. The formula is as follows:

[0071] h m =biLSTM(x m ,θ m )

[0072] Among them, x m is the sequence feature of the input modality m, θ m is the corresponding neural network parameter, h m is a modal feature;

[0073] (2) Construct uncertainty estimation module:

[0074] The above three modal features h m As input, in order to achieve feature alignment, a multi-layer perceptron (MLP) is used for feature transformation to map modal features of different dimensions into a unified feature space:

[0075] f m =MLP(h m , w m )

[0076] Among them, w m are the network parameters of the multilayer perceptron, f m is the mapped feature vector, and the number of hidden layer neural units of the multilayer perceptron is 128;

[0077] The data of the modal sequence features are regarded as being sampled from different Gaussian distributions, that is, in, Indicates that the mean value is u m , with variance σ m 2 Gaussian distribution. Each mode has its own distribution, u m represents the mean, which is regarded as the sample mode itself, and σ m 2 Represents the variance, which is regarded as modal noise. In order to model the uncertainty distribution, two fully connected layers FC1 and FC2 are used to estimate the mean u of the mode m respectively. m and variance σ m :

[0078] u m =FC1(fm , w1)

[0079] σ m 2 =FC2(f m , w2)

[0080] Among them, w1 and w2 are the learnable parameters of the fully connected layers FC1 and FC2 respectively;

[0081] Although the fully connected layer can estimate the variance σ m 2 , but in the process of network model training, due to the small variance σ m 2 This will cause the model to degenerate into a deterministic representation; to solve this problem, a regularization term is added to constrain the parameter distribution to be approximately normal through KL divergence:

[0082]

[0083] in, is the loss calculated by KL divergence, u m，i represents the mean u m The i-th dimension of , D represents the vector dimension, is set to 128;

[0084] In order to estimate the variance σ m 2 The distribution effectively captures the information of each mode and constrains it through data reconstruction, thereby minimizing the error between input and output. Since the sampling process is not differentiable, parameterized resampling is introduced for back propagation. The specific process is as follows:

[0085] z m =u m +∈·σ m 2 ,∈∈(0,I)

[0086]

[0087] Among them, z m Represents the resampled features, ∈ represents the sampling noise of the normal distribution, 0 and I are vectors of all 0 and all 1 respectively, and the reconstruction loss The absolute mean error was used for calculation;

[0088] (3) Constructing the uncertain fusion module:

[0089] The mean u of the distribution estimated based on uncertainty m and variance σ m 2 , perform uncertain fusion, first calculate the image mean u vand image variance σ v 2 And the text mean u t and text variance σ t 2 The fusion of the two modal distributions is then combined with the audio mean u a and audio variance σ a 2 Distributed fusion; the specific process is as follows:

[0090]

[0091]

[0092]

[0093]

[0094] Among them, u vt and σ vt 2 is the distribution after the fusion of image and text modalities, u vta and σ vta 2 is the final distribution obtained by fusing the three modalities;

[0095] (4) Constructing the sentiment prediction module:

[0096] During the training phase, the obtained cross-modal fusion results are input into the fully connected layer FC reg Perform sentiment classification, output the regressed sentiment score, and use the mean absolute error as the loss function; the specific process is as follows:

[0097] O m =FC reg (u vta , w class )

[0098]

[0099] Among them, w class is the network parameter of the fully connected layer, O m is the predicted sentiment score, g m is the true sentiment score of the sample, is the calculated error loss;

[0100] Step 3: Train and optimize the network model on the training set:

[0101] The overall loss function that needs to be optimized is:

[0102]

[0103] Step 4: Verify and test the network model on the validation set and test set:

[0104] During the validation and testing phases, the predicted sentiment scores were segmented and limited to [-3, 3]. The sentiment scores were rounded to the nearest integer, resulting in the following categories: very negative, negative, slightly negative, neutral, slightly positive, positive, and very positive.

[0105] O class =round(O m )

[0106] Among them, the function round() is a rounding operation, class It is the rounded result, which represents the category finally predicted by the network model.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. The scope of the present invention is defined by the appended claims, not the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be encompassed therein.

Claims

1. A cross-modal video emotion information analysis method, comprising the following steps: Step 1: Process the original video data and extract the feature sequence for model input: (1) Use the OpenFace toolkit to process the video image and extract the facial feature sequence x v , the feature dimension of each video image is 20; (2) Use the pre-trained BERT model to obtain the text sequence x for the text data in the video t , the feature dimension of each word is 768; (3) Use the COVAREP toolkit to extract the feature sequence x from the audio signal in the video a , the feature dimension of each audio information is 5; (4) The three extracted feature sequences are used to construct a dataset X: Among them, m=v,t,a means that the modality m has three values: v,t,a, v,t,a represent the three modes of video, text and audio respectively, T m and D m Respectively represent the sequence length and feature dimension from modality m; (5) Divide the constructed dataset X into 8:1:1 to obtain the training set, test set, and validation set; Step 2: Build a cross-modal sentiment prediction network model: (1) Construct a single-modal feature extraction module: Take the constructed dataset X as input, use the bidirectional long short-term memory network biLSTM to model the three modal features, learn the temporal changes within each modality, and use the final state output as the feature representation. The formula is as follows: h m =biLSTM(x m ,θ m ) Among them, x m is the sequence feature of the input modality m, θ m is the corresponding neural network parameter, h m is a modal feature; (2) Construct uncertainty estimation module: The above three modal features h m As input, in order to achieve feature alignment, a multi-layer perceptron (MLP) is used for feature transformation to map modal features of different dimensions into a unified feature space: f m =MLP(h m ,w m ) Among them, w m are the network parameters of the multilayer perceptron, f m is the mapped feature vector, and the number of hidden layer neural units of the multilayer perceptron is 128; The data of the modal sequence features are regarded as being sampled from different Gaussian distributions, that is, in, Indicates that the mean value is u m , with variance σ m 2 Gaussian distribution; each mode has its own distribution, u m represents the mean, which is regarded as the sample mode itself, and σ m 2 Represents the variance, which is regarded as modal noise. In order to model the uncertainty distribution, two fully connected layers FC1 and FC2 are used to estimate the mean u of the mode m respectively. m and variance σ m : u m =FC1(f m ,w1) s m 2 =FC2(f m ,w2) Among them, w1 and w2 are the learnable parameters of the fully connected layers FC1 and FC2 respectively; Although the fully connected layer can estimate the variance σ m 2 , but in the process of network model training, due to the small variance σ m 2 This will cause the model to degenerate into a deterministic representation; to solve this problem, a regularization term is added to constrain the parameter distribution to be approximately normal through KL divergence: in, is the loss calculated by KL divergence, u m，i represents the mean u m The i-th dimension of , D represents the vector dimension, is set to 128; In order to estimate the variance σ m 2 The distribution effectively captures the information of each mode and constrains it through data reconstruction, thereby minimizing the error between input and output. Since the sampling process is not differentiable, parameterized resampling is introduced for back propagation. The specific process is as follows: With m =u m +∈·σ m 2 ,∈∈(0,I) Among them, z m Represents the resampled features, ∈ represents the sampling noise of the normal distribution, 0 and I are vectors of all 0 and all 1 respectively, and the reconstruction loss The absolute mean error was used for calculation; (3) Constructing the uncertain fusion module: The mean u of the distribution estimated based on uncertainty m and variance σ m 2 , perform uncertain fusion, first calculate the image mean u v and image variance σ v 2 And the text mean u t and text variance σ t 2 The fusion of the two modal distributions is then combined with the audio mean u a and audio variance σ a 2 Distributed fusion; the specific process is as follows: Among them, u vt and σ vt 2 is the distribution after the fusion of image and text modalities, u vta and σ vta 2 is the final distribution obtained by fusing the three modalities; (4) Constructing the sentiment prediction module: During the training phase, the obtained cross-modal fusion results are input into the fully connected layer FC reg Perform sentiment classification, output the regressed sentiment score, and use the mean absolute error as the loss function; the specific process is as follows: About m =FC reg (at vta ,w class ) Among them, w class is the network parameter of the fully connected layer, O m is the predicted sentiment score, g m is the true sentiment score of the sample, is the calculated error loss; Step 3: Train and optimize the network model on the training set: The overall loss function that needs to be optimized is: Step 4: Verify and test the network model on the validation set and test set: During the validation and testing phases, the predicted sentiment scores were segmented and limited to [-3, 3]. The sentiment scores were rounded to the nearest integer, resulting in the following categories: very negative, negative, slightly negative, neutral, slightly positive, positive, and very positive. O class =round(O m ) Among them, the function round() is a rounding operation, class It is the rounded result, which represents the category finally predicted by the network model.

Images