Multimodal sentiment analysis method, device and equipment based on uncertainty estimation
Patent Information
- Application Number
- CN202210560653.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-05-20
Smart Images

Figure CN114999006B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multimodal sentiment analysis method, device and equipment based on uncertainty estimation, and belongs to the field of data analysis. Background Art
[0002] Multimodal sentiment analysis aims to analyze the emotional information contained in the images, audio and text information in the video, such as happiness, anger and sadness. Existing multimodal sentiment analysis methods mainly focus on the feature representation of multimodal data and multimodal fusion.
[0003] In terms of feature representation, a pre-trained feature extraction network is usually used to extract features from different modal data, and then the extracted features are further characterized and learned. For the video modality, the face in the image is usually recognized first, and then the facial expression features are extracted; for the speech modality, the acoustic feature analysis framework COVAREP is usually used to extract audio features; for the text modality, the LSTM (Long Short Term Memory) model is usually used, and the semantic word vector of the text is input to obtain the contextual features of the text.
[0004] In terms of multimodal fusion, it is mainly necessary to take into account the consistency features and difference features between different modalities. For example, TFN (Zadeh A, Chen M, Poria S, et al. Tensor fusion network for multimodal sentiment analysis [J]. arXiv preprint arXiv: 1707.07250, 2017) proposed by Zadeh earlier, performs Cartesian product on the features of different modalities to obtain the consistency features between modalities. However, this method is prone to excessive dimension and information redundancy when there are a large number of modalities. It has high requirements on computing resources and takes a long time to analyze multimodal data, so it is not very practical. In addition, there is MISA (Hazarika D, Zimmermann R, Poria S. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis [C]. The 28th ACM International Conference on Multimedia. 2020: 1122-1131.) proposed by Hazarika et al., which projects multimodality into modality-invariant and modality-specific spaces respectively, capturing the common and unique features of the modalities respectively, and using these overall features to fuse and predict the results. However, this method is difficult to directly represent the complementary information in the modalities based on the difference in spatial distance.
[0005] At the same time, existing multimodal sentiment analysis methods ignore the measurement of the reliability of prediction results. Due to the uneven quality of data between different modalities, the reliability of prediction results is difficult to measure and has poor interpretability.
[0006] The above issues are those that should be considered and resolved in the process of multimodal sentiment analysis. Summary of the invention
[0007] The purpose of the present invention is to provide a multimodal sentiment analysis method based on uncertainty estimation to solve the problems existing in the prior art that the reliability of prediction results is difficult to measure, the interpretability is poor, the analysis is time-consuming, and the complementary information in different modes is difficult to effectively retain.
[0008] The technical solution of the present invention is:
[0009] A multimodal sentiment analysis method based on uncertainty estimation comprises the following steps:
[0010] S1. Obtain a multimodal sentiment analysis dataset. The video samples in the multimodal sentiment analysis dataset include audio data, image data, text data and sample labels. Feature extraction is performed on the audio data, image data and text data in the video samples to obtain corresponding audio features, video features and text features.
[0011] S2, performing context representation on the audio features and video features obtained in step S1 to obtain represented audio features and video features;
[0012] S3, inputting the characterized audio features, video features and text features obtained in step S2 into their respective uncertainty estimation models based on Dirichlet distribution, and obtaining parameters of the Dirichlet distribution of the audio mode, parameters of the Dirichlet distribution of the video mode and parameters of the Dirichlet distribution of the text mode and uncertainty estimation respectively;
[0013] S4, using DS evidence theory to solve the confidence of the parameters of the Dirichlet distribution of the audio mode, the parameters of the Dirichlet distribution of the video mode, and the parameters of the Dirichlet distribution of the text mode obtained in step S3, and then integrating the confidence and uncertainty of each mode using the Dempster combination rule to obtain the sentiment classification result and the overall uncertainty estimation;
[0014] S5, taking the sentiment classification result obtained in step S4 and the sample labels in the multimodal sentiment analysis data set obtained in step S1 as input, and using the cross entropy function as the loss function to train the uncertainty estimation model based on the Dirichlet distribution in step S3, to obtain a trained uncertainty estimation model based on the Dirichlet distribution;
[0015] S6. Taking the video to be subjected to sentiment analysis as input, the uncertainty estimation model based on Dirichlet distribution obtained in step S5 is used to output the obtained prediction result.
[0016] Furthermore, in step S1, feature extraction is performed on the audio data, image data and text data in the video sample, specifically,
[0017] S11, segmenting the audio data in the data set frame by frame, and inputting the pre-trained audio feature extraction network to obtain audio features;
[0018] S12, segmenting the video data in the data set frame by frame, inputting the pre-trained image feature extraction network to obtain video features;
[0019] S13. Input the text data in the dataset into the text pre-training model BERT to obtain the represented text features.
[0020] Furthermore, in step S2, the audio features and video features obtained in step S1 are contextually characterized to obtain characterized audio features and video features, specifically,
[0021] S21, inputting the audio features obtained in step S1 into a bidirectional long short-term memory network to extract the contextual relationship in the audio features, then inputting the output of the bidirectional long short-term memory network into a random dropout layer to avoid overfitting during the training process, and finally inputting the output of the random dropout layer into a fully connected layer to obtain the characterized audio features;
[0022] S22. The video features obtained in step S1 are input into a bidirectional long short-term memory network to extract the contextual relationship in the video features. The output of the bidirectional long short-term memory network is then input into a random dropout layer to avoid overfitting during training. Finally, the output of the random dropout layer is input into a fully connected layer to obtain the represented video features.
[0023] Furthermore, in step S3, the characterized audio features, video features, and text features obtained in step S2 are respectively input into respective uncertainty estimation models based on Dirichlet distribution to obtain Dirichlet distribution and uncertainty estimation corresponding to the audio, video, and text modalities, specifically,
[0024] S31, respectively input the audio features, video features and text features into their respective uncertainty estimation models based on Dirichlet distribution, the model consists of a fully connected layer and a linear rectifier activation function layer, wherein the fully connected layer maps the features after the corresponding modal context representation obtained in step S2 into an evidence form, and outputs it to the linear rectifier activation function layer to ensure that the output result is a non-negative value, and its output dimension is equal to the number of emotion categories. The model output is the evidence representation of the corresponding modality based on Dirichlet distribution:
[0025]
[0026] Among them, a, v, t correspond to audio modality, video modality and text modality respectively, and K is the number of emotion categories;
[0027] S32, according to the evidence representation of the corresponding mode based on the Dirichlet distribution obtained in step S31, solve the Dirichlet distribution Using the parameters of the Dirichlet distribution, we get the corresponding uncertainty estimate u a 、u v 、u t .
[0028] Further, in step S32, the Dirichlet distribution is solved Using the parameters of the Dirichlet distribution, we get the corresponding uncertainty estimate u a 、u v 、u t , the calculation formula is as follows:
[0029]
[0030] Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity, and K is the number of sentiment categories.
[0031] Furthermore, in step S4, the DS evidence theory is used to solve the confidence of the parameters of the Dirichlet distribution of the audio mode, the parameters of the Dirichlet distribution of the video mode, and the parameters of the Dirichlet distribution of the text mode obtained in step S3, and then the confidence and uncertainty of each mode are fused using the Dempster combination rule to obtain the sentiment classification result and the overall uncertainty estimation, specifically,
[0032] S41. For the Dirichlet distribution of audio, video and text modalities, solve the confidence of each modality based on DS evidence theory. The calculation formula is as follows:
[0033]
[0034] Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity;
[0035] S42, the confidence and uncertainty corresponding to the audio mode and the video mode are preliminarily integrated using the Dempster combination rule to obtain the confidence after preliminary integration β′=[β′1,...,β′ K ] and uncertainty u′, and then β′=[β′1,...,β′ K ] and u′ and the confidence of text modality and uncertainty u t Fusion is performed to obtain the final confidence β″=[β″1, ..., β″ K ] and uncertainty u″;
[0036] S43, using the confidence level β″, the final predicted probability distribution result p=[p1, ..., p K ].
[0037] Furthermore, in step S42, the confidence and uncertainty corresponding to the audio modality and the video modality are preliminarily fused using the Dempster combination rule to obtain the confidence after preliminary fusion β′=[β′1, ..., β′ K] and uncertainty u′, calculated as follows:
[0038]
[0039] in, is the confidence of the i-th emotion category corresponding to the audio modality, is the confidence of the jth emotion category corresponding to the audio modality, are the confidence scores of the kth emotion category corresponding to the audio and video modalities, u a 、u v 、u t They are the uncertainty estimates corresponding to the audio modality, video modality, and text modality respectively;
[0040] Then β′=[β′1,...,β′ K ] and u′ and the confidence of text modality and uncertainty u t Fusion is performed to obtain the final confidence β″=[β″1…,β″ K ] and uncertainty u″, the calculation formula is as follows:
[0041]
[0042] in, β′ i is the confidence of the i-th emotion category after preliminary fusion, is the confidence of the jth sentiment category of the text modality, β′ k is the confidence of the kth emotion category after initial fusion, is the confidence of the kth sentiment category of the text modality, u t is the uncertainty estimate corresponding to the text modality, and u′ is the uncertainty after preliminary fusion.
[0043] Furthermore, in step S43, the final predicted probability distribution result p=[p1, ..., p K ], the calculation formula is as follows:
[0044] p k =exp[ψ(β″ k ×S+1)-ψ(S)], k∈[1,K]
[0045] Among them, ψ(·) is the Digamma function, β″ k is the final confidence of the kth emotion category obtained in step S42, Wherein, u″ is the final uncertainty obtained in step S42, and K is the number of emotion categories.
[0046] A device for implementing any of the above-mentioned multimodal sentiment analysis methods based on uncertainty estimation, comprising a feature extraction module, a sentiment classification module and a prediction analysis module,
[0047] Feature extraction module: obtains a multimodal sentiment analysis data set, extracts features from the audio data, image data, and text data in the video sample, and obtains corresponding audio features, video features, and text features; contextually represents the obtained audio features and video features, and obtains the represented audio features and video features;
[0048] Sentiment classification module: The characterized audio features, video features and text features obtained by the feature extraction module are respectively input into their respective uncertainty estimation models based on Dirichlet distribution to obtain the Dirichlet distribution and uncertainty estimation of the corresponding audio, video and text modalities; the confidence of the Dirichlet distribution of the audio modality, the Dirichlet distribution of the video modality and the Dirichlet distribution of the text modality is solved using the DS evidence theory, and then the confidence and uncertainty of each modality are integrated to obtain the sentiment classification result and the overall uncertainty estimation;
[0049] Prediction and analysis module: The sentiment classification results obtained by the sentiment classification module and the sample labels in the multimodal sentiment analysis dataset are used as input, and the cross entropy function is used as the loss function for training to obtain the trained uncertainty estimation model based on Dirichlet distribution; the video that needs to be sentimentally analyzed is used as input, and the trained uncertainty estimation model based on Dirichlet distribution is used to output the prediction results.
[0050] A computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of any of the above-mentioned multimodal sentiment analysis methods based on uncertainty estimation are implemented.
[0051] The beneficial effects of the present invention are:
[0052] 1. Compared with the prior art, this multimodal sentiment analysis method, device and equipment based on uncertainty estimation has higher recognition accuracy, stronger robustness and better interpretability, and has low requirements for computing resources, and takes less time to analyze multimodal data. This multimodal sentiment analysis method based on uncertainty estimation estimates the uncertainty of multimodal data and models its probability distribution for audio and video sentiment analysis. It can improve classification accuracy while accurately evaluating the quality of each modal data in the multimodal data sample, solving the problem of low reliability and lack of interpretability of multimodal data sentiment prediction results due to noise and uncertainty in single modal data.
[0053] Second, the uncertainty estimation of the method of the present invention based on Dirichlet distribution can obtain the uncertainty estimation of the overall prediction result while performing sentiment analysis on multimodal data. The uncertainty estimation index can be used to measure whether the prediction result is reliable. The smaller the value, the higher the reliability. At the same time, the uncertainty of single-modal data can be dynamically obtained through single-modal sentiment analysis, which is used to measure the quality of each single-modal data, solving the problem of lack of interpretability in obtaining prediction results in existing methods.
[0054] 3. This multimodal sentiment analysis method, device and equipment based on uncertainty estimation performs sentiment analysis on each modality. The predictions of each modality are independent of each other and do not interfere with each other, and can effectively retain the complementary information between different modalities.
[0055] Fourth, this multimodal sentiment analysis method, device and equipment based on uncertainty estimation adopts Dirichlet distribution and Dempster combination rule in the process of fusing the prediction results of different modes, which has higher recognition accuracy than the existing softmax-based classifier. The method of the present invention fuses the prediction results of different modes, and the fusion calculation process does not involve a large number of learning parameters. It converges quickly during the training process, has low requirements for computing resources, and takes less time to analyze multimodal data, which is more in line with the needs of actual scenarios.
[0056] 5. This multimodal sentiment analysis method, device and equipment based on uncertainty estimation uses DS evidence theory to represent and fuse multimodal features. When there is noise and uncertainty in the data, the classification accuracy attenuation is lower and it has stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 It is a flowchart of a multimodal sentiment analysis method based on uncertainty estimation according to an embodiment of the present invention.
[0058] Figure 2 It is a schematic diagram illustrating how to obtain sentiment classification results according to the embodiment.
[0059] Figure 3 It is a schematic diagram illustrating a multimodal sentiment analysis device based on uncertainty estimation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0061] Example
[0062] A multimodal sentiment analysis method based on uncertainty estimation, such as Figure 1 , including the following steps,
[0063] S1. Obtain a multimodal sentiment analysis dataset. The video samples in the multimodal sentiment analysis dataset include audio data, image data, text data and sample labels. Feature extraction is performed on the audio data, image data and text data in the video samples to obtain corresponding audio features, video features and text features.
[0064] S11, segmenting the audio data in the data set frame by frame, and inputting the pre-trained audio feature extraction network to obtain audio features;
[0065] S12, segmenting the video data in the data set frame by frame, inputting the pre-trained image feature extraction network to obtain video features;
[0066] S13. Input the text data in the dataset into the text pre-training model BERT to obtain the represented text features.
[0067] S2, performing context representation on the audio features and video features obtained in step S1 to obtain represented audio features and video features;
[0068] S21, input the audio features obtained in step S1 into a bidirectional long short-term memory network sLSTM to extract the contextual relationship in the audio features, then input the output of the bidirectional long short-term memory network into a random dropout layer, i.e., a dropout layer, to avoid overfitting during training, and finally input the output of the random dropout layer into a fully connected layer to obtain the characterized audio features;
[0069] S22. The video features obtained in step S1 are input into a bidirectional long short-term memory network sLSTM to extract the contextual relationship in the video features. The output of the bidirectional long short-term memory network is then input into a random dropout layer to avoid overfitting during training. Finally, the output of the random dropout layer is input into a fully connected layer to obtain the represented video features.
[0070] S3, inputting the characterized audio features, video features and text features obtained in step S2 into their respective uncertainty estimation models based on Dirichlet distribution, and obtaining parameters of the Dirichlet distribution of the audio mode, parameters of the Dirichlet distribution of the video mode and parameters of the Dirichlet distribution of the text mode and uncertainty estimation respectively;
[0071] S31, respectively input the audio features, video features and text features into their respective uncertainty estimation models based on Dirichlet distribution, the model consists of a fully connected layer and a linear rectifier activation function layer, i.e., a ReLU activation function layer, wherein the fully connected layer maps the features represented by the corresponding modal context obtained in step S2 into an evidence form, and outputs it to the linear rectifier activation function layer to prevent the gradient vanishing problem generated during the training process, and ensure that the output result is a non-negative value, and its output dimension is equal to the number of emotion categories. The model output is the evidence representation of the corresponding modality based on Dirichlet distribution:
[0072]
[0073] Among them, a, v, t correspond to audio modality, video modality and text modality respectively, and K is the number of emotion categories;
[0074] S32, according to the evidence representation of the corresponding mode based on the Dirichlet distribution obtained in step S31, solve the Dirichlet distribution Using the parameters of the Dirichlet distribution, we get the corresponding uncertainty estimate u a 、u v 、u t , the calculation formula is:
[0075]
[0076] Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity, and K is the number of sentiment categories.
[0077] S4. Use the DS evidence theory, i.e., the Dempster-Shafer evidence theory, to solve the confidence of the parameters of the Dirichlet distribution of the audio modality, the parameters of the Dirichlet distribution of the video modality, and the parameters of the Dirichlet distribution of the text modality obtained in step S3, and then use Dempster's combinational rule to fuse the confidence and uncertainty of each modality to obtain the sentiment classification result and the overall uncertainty estimation, such as Figure 2 , specifically:
[0078] S41. For the Dirichlet distribution of audio, video and text modalities, solve the confidence of each modality based on DS evidence theory. The calculation formula is as follows:
[0079]
[0080] Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity;
[0081] S42, the confidence and uncertainty corresponding to the audio mode and the video mode are preliminarily fused using the Dempster fusion rule to obtain the confidence after preliminary fusion β′=[β′1, ..., β′ K ] and uncertainty u′, and then β′=[β′1,...,β′ K ] and u′ and the confidence of text modality and uncertainty u t Fusion is performed to obtain the final confidence β″=[β″1,…,β″ K ] and uncertainty u″, the calculation formula is as follows:
[0082]
[0083]
[0084] in, is the confidence of the i-th emotion category corresponding to the audio modality, is the confidence of the jth emotion category corresponding to the audio modality, are the confidence scores of the kth emotion category corresponding to the audio and video modalities, u a 、u v 、u t They are the uncertainty estimates corresponding to the audio modality, video modality, and text modality respectively; β′ i is the confidence of the i-th emotion category after preliminary fusion, is the confidence of the jth sentiment category of the text modality, β′ k is the confidence of the kth emotion category after initial fusion, is the confidence of the kth sentiment category of the text modality, u t is the uncertainty estimate corresponding to the text modality, and u′ is the uncertainty after preliminary fusion.
[0085] S43, using the confidence level β″, the final predicted probability distribution result p=[p1, ..., p K ], the calculation formula is as follows:
[0086] p k =exp[ψ(β″ k ×S+1)-ψ(S)], k∈[1,K]
[0087] Among them, ψ(·) is the Digamma function, β″ k is the final confidence of the kth emotion category obtained in step S42, Wherein, u″ is the final uncertainty obtained in step S42, and K is the number of emotion categories.
[0088] S5, taking the sentiment classification result obtained in step S4 and the sample labels in the multimodal sentiment analysis data set obtained in step S1 as input, and using the cross entropy function as the loss function to train the uncertainty estimation model based on the Dirichlet distribution in step S3, to obtain a trained uncertainty estimation model based on the Dirichlet distribution;
[0089] S6. Taking the video to be subjected to sentiment analysis as input, the uncertainty estimation model based on Dirichlet distribution obtained in step S5 is used to output the obtained prediction result.
[0090] A device for implementing any of the above-mentioned multimodal sentiment analysis methods based on uncertainty estimation, comprising a feature extraction module, a sentiment classification module and a prediction analysis module, such as Figure 3 .
[0091] Feature extraction module: obtains a multimodal sentiment analysis data set, extracts features from the audio data, image data, and text data in the video sample, and obtains corresponding audio features, video features, and text features; contextually represents the obtained audio features and video features, and obtains the represented audio features and video features;
[0092] Sentiment classification module: The characterized audio features, video features and text features obtained by the feature extraction module are respectively input into their respective uncertainty estimation models based on Dirichlet distribution to obtain the Dirichlet distribution and uncertainty estimation of the corresponding audio, video and text modalities; the confidence of the Dirichlet distribution of the audio modality, the Dirichlet distribution of the video modality and the Dirichlet distribution of the text modality is solved using the DS evidence theory, and then the confidence and uncertainty of each modality are integrated to obtain the sentiment classification result and the overall uncertainty estimation;
[0093] Prediction and analysis module: The sentiment classification results obtained by the sentiment classification module and the sample labels in the multimodal sentiment analysis dataset are used as input, and the cross entropy function is used as the loss function for training to obtain the trained uncertainty estimation model based on Dirichlet distribution; the video that needs to be sentimentally analyzed is used as input, and the trained uncertainty estimation model based on Dirichlet distribution is used to output the prediction results.
[0094] A computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of any of the above-mentioned multimodal sentiment analysis methods based on uncertainty estimation are implemented.
[0095] Compared with the prior art, this multimodal sentiment analysis method, device and equipment based on uncertainty estimation has higher recognition accuracy, stronger robustness and better interpretability, and has low requirements for computing resources, and takes less time to analyze multimodal data. This multimodal sentiment analysis method based on uncertainty estimation estimates the uncertainty of multimodal modal data and models its probability distribution for audio and video sentiment analysis. It can improve classification accuracy while accurately evaluating the quality of each modal data in the multimodal data sample, solving the problem of low reliability and lack of interpretability of multimodal data sentiment prediction results due to noise and uncertainty in single modal data.
[0096] The method of the present invention, based on the uncertainty estimation of Dirichlet distribution, can obtain the uncertainty estimation of the overall prediction result while performing sentiment analysis on multimodal data. The uncertainty estimation index can be used to measure whether the prediction result is reliable. The smaller the value, the higher the reliability. At the same time, the uncertainty of single-modal data can be dynamically obtained through sentiment analysis of single-modal data, which is used to measure the quality of each single-modal data, solving the problem of lack of interpretability in obtaining prediction results in existing methods.
[0097] This multimodal sentiment analysis method, device and equipment based on uncertainty estimation performs sentiment analysis on each modality. The predictions of each modality are independent of each other and do not interfere with each other, and can effectively retain the complementary information between different modalities.
[0098] This multimodal sentiment analysis method, device and equipment based on uncertainty estimation adopts Dirichlet distribution and Dempster combination rule in the process of fusing prediction results of different modes, and has higher recognition accuracy than the existing softmax-based classifier. The method of the present invention fuses the prediction results of different modes, and the fusion calculation process does not involve a large number of learning parameters. It converges quickly during the training process, has low requirements for computing resources, and takes less time to analyze multimodal data, which is more in line with the needs of actual scenarios.
[0099] This multimodal sentiment analysis method, device and equipment based on uncertainty estimation adopts DS evidence theory to represent and fuse multimodal features. When there is noise and uncertainty in the data, the classification accuracy attenuation is lower and it has stronger robustness.
[0100] This multimodal sentiment analysis method, device and equipment based on uncertainty estimation can dynamically evaluate the quality of single-modal data to obtain reliable uncertainty estimation under the premise of ensuring classification accuracy, which is of great significance for the interpretability of prediction results. The present invention can obtain the uncertainty of each modal prediction to evaluate the modality quality while improving the accuracy of multimodal sentiment analysis results, solving the problem of low reliability and poor interpretability of prediction results of existing methods.
[0101] The multimodal sentiment analysis method based on uncertainty estimation in the embodiment is experimentally verified as follows:
[0102] The implementation example uses the Pytorch library in the Python environment on a GeForce GTX 2080 Ti processor. In the experiment, the Adam optimizer is used. In the training phase, the number of iterations is set to 100 and the feature representation network learning rate is set to 10. -3 , the learning rate of the pre-trained BERT model is set to 50 -5 .
[0103] Experiments were conducted using the multimodal dataset MOSI, and five existing models with better results were compared: TFN (Tensor Fusion Network), MFN (Zadeh, Amir, et al. "Memory fusion network for multi-view sequential learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.), MFM (Tsai YH, Liang PP, Zadeh A, et al. Learning Factorized Multimodal Representations [C]. International Conference on Learning Representations, 2018.), ICCN (Sun, Zhongkai, et al. "Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020.), and MISA (Modality Invariant and Specific Representations). The experimental results are shown in Table 1.In Table 1, TFN represents the Tensor Fusion Network proposed by Zadeh, MFN represents the Memory fusion network proposed by Zadeh, MFM represents the Multimodal Factorization Model proposed by Tsai, ICCN represents the Interaction Canonical Correlation Network proposed by Sun, MISA represents the Modality Invariant and Specific Representations network model proposed by Hazarika, Acc-7 represents the recognition accuracy of the seven-classification task, Acc-2 represents the recognition accuracy of the two-classification task, and F1 represents the F1 score, which is an indicator used to measure the accuracy of the two-classification model.
[0104] Table 1 Comparison of the results of the embodiment method with the existing five models
[0105]
[0106] The results in Table 1 show that compared with the existing methods, the embodiment method has a more significant improvement effect on recognition accuracy.
[0107] The above is a detailed description of the implementation mode of the present invention in combination with the accompanying drawings, but the present invention is not limited to the above-mentioned implementation mode. Within the scope of knowledge possessed by ordinary technicians in the field, associative changes or substitutions should be included in the scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A multimodal sentiment analysis method based on uncertainty estimation, characterized by: The following steps are included: S1. Obtain a multimodal sentiment analysis dataset. The video samples in the multimodal sentiment analysis dataset include audio data, image data, text data and sample labels. Feature extraction is performed on the audio data, image data and text data in the video samples to obtain corresponding audio features, video features and text features. S2, performing context representation on the audio features and video features obtained in step S1 to obtain represented audio features and video features; S3, inputting the characterized audio features, video features and text features obtained in step S2 into their respective uncertainty estimation models based on Dirichlet distribution, and obtaining parameters of Dirichlet distribution of audio mode, parameters of Dirichlet distribution of video mode and parameters of Dirichlet distribution of text mode and uncertainty estimation respectively; specifically, S31, respectively input the audio features, video features and text features into their respective uncertainty estimation models based on Dirichlet distribution, the model consists of a fully connected layer and a linear rectifier activation function layer, wherein the fully connected layer maps the features after the corresponding modal context representation obtained in step S2 into an evidence form, and outputs it to the linear rectifier activation function layer to ensure that the output result is a non-negative value, and its output dimension is equal to the number of emotion categories. The model output is the evidence representation of the corresponding modality based on Dirichlet distribution: Among them, a, v, t correspond to audio modality, video modality and text modality respectively, and K is the number of emotion categories; S32, according to the evidence representation of the corresponding mode based on the Dirichlet distribution obtained in step S31, solve the Dirichlet distribution Using the parameters of the Dirichlet distribution, we get the corresponding uncertainty estimate u a 、u v 、u t ; S4, using DS evidence theory to solve the confidence of the parameters of the Dirichlet distribution of the audio mode, the parameters of the Dirichlet distribution of the video mode, and the parameters of the Dirichlet distribution of the text mode obtained in step S3, and then integrating the confidence and uncertainty of each mode using the Dempster combination rule to obtain the sentiment classification result and the overall uncertainty estimation; S5, taking the sentiment classification result obtained in step S4 and the sample labels in the multimodal sentiment analysis data set obtained in step S1 as input, and using the cross entropy function as the loss function to train the uncertainty estimation model based on the Dirichlet distribution in step S3, to obtain a trained uncertainty estimation model based on the Dirichlet distribution; S6. Taking the video to be subjected to sentiment analysis as input, the uncertainty estimation model based on Dirichlet distribution obtained after training in step S5 is used to output the obtained prediction result.
2. The multimodal sentiment analysis method based on uncertainty estimation according to claim 1, characterized in that: In step S1, feature extraction is performed on the audio data, image data and text data in the video sample respectively, specifically, S11, segmenting the audio data in the data set frame by frame, and inputting the pre-trained audio feature extraction network to obtain audio features; S12, segmenting the video data in the data set frame by frame, inputting the pre-trained image feature extraction network to obtain video features; S13. Input the text data in the dataset into the text pre-training model BERT to obtain the represented text features.
3. The multimodal sentiment analysis method based on uncertainty estimation according to claim 1, characterized in that: In step S2, the audio features and video features obtained in step S1 are contextually represented to obtain the represented audio features and video features, specifically, S21, inputting the audio features obtained in step S1 into a bidirectional long short-term memory network to extract the contextual relationship in the audio features, then inputting the output of the bidirectional long short-term memory network into a random dropout layer to avoid overfitting during the training process, and finally inputting the output of the random dropout layer into a fully connected layer to obtain the characterized audio features; S22. The video features obtained in step S1 are input into a bidirectional long short-term memory network to extract the contextual relationship in the video features. The output of the bidirectional long short-term memory network is then input into a random dropout layer to avoid overfitting during training. Finally, the output of the random dropout layer is input into a fully connected layer to obtain the represented video features.
4. The multimodal sentiment analysis method based on uncertainty estimation according to any one of claims 1 to 3, characterized in that: In step S32, the Dirichlet distribution is solved Using the parameters of the Dirichlet distribution, we get the corresponding uncertainty estimate u a 、u v 、u t , the calculation formula is as follows: to m =and m +1, Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity, and K is the number of sentiment categories.
5. The multimodal sentiment analysis method based on uncertainty estimation according to any one of claims 1 to 3, characterized in that: In step S4, the DS evidence theory is used to solve the confidence of the parameters of the Dirichlet distribution of the audio mode, the parameters of the Dirichlet distribution of the video mode, and the parameters of the Dirichlet distribution of the text mode obtained in step S3, and then the confidence and uncertainty of each mode are fused using the Dempster combination rule to obtain the sentiment classification result and the overall uncertainty estimation, specifically, S41. For the Dirichlet distribution of audio, video and text modalities, solve the confidence of each modality based on DS evidence theory. The calculation formula is as follows: Among them, m∈{a, v, t}, is the corresponding mode, represents the Dirichlet intensity; S42, the confidence and uncertainty corresponding to the audio mode and the video mode are preliminarily integrated using the Dempster combination rule to obtain the confidence after preliminary integration β′=[β′1,...,β′ K ] and uncertainty u′, and then β′=[β′1,...,β′ K ] and u′ and the confidence of text modality and uncertainty u t Fusion is performed to obtain the final confidence β″=[β″1, ..., β″ K ] and uncertainty u″; S43, using the confidence level β″, the final predicted probability distribution result p=[p1, ..., p K ].
6. The multimodal sentiment analysis method based on uncertainty estimation according to claim 5, characterized in that: In step S42, the confidence and uncertainty corresponding to the audio mode and the video mode are preliminarily fused using the Dempster combination rule to obtain the confidence after preliminary fusion β′=[β′1, ..., β′ K ] and uncertainty u′, calculated as follows: in, is the confidence of the i-th emotion category corresponding to the audio modality, is the confidence of the jth emotion category corresponding to the audio modality, are the confidence scores of the kth emotion category corresponding to the audio and video modalities, u a 、u v 、u t They are the uncertainty estimates corresponding to the audio modality, video modality, and text modality respectively; Then β′=[β′1,...,β′ K ] and u′ and the confidence of text modality and uncertainty u t Fusion is performed to obtain the final confidence β″=[β″1, ..., β″ K ] and uncertainty u″, the calculation formula is as follows: in, β' i is the confidence of the i-th emotion category after preliminary fusion, is the confidence of the jth sentiment category of the text modality, β′ k is the confidence of the kth emotion category after initial fusion, is the confidence of the kth sentiment category of the text modality, u t is the uncertainty estimate corresponding to the text modality, and u′ is the uncertainty after preliminary fusion.
7. The multimodal sentiment analysis method based on uncertainty estimation according to claim 5, characterized in that: In step S43, the final predicted probability distribution result p=[p1, ..., p K ], the calculation formula is as follows: p k =exp[ψ(β″ k ×S+1)-ψ(S)], k∈[1, K] Among them, ψ(·) is the Digamma function, β″ k is the final confidence of the kth emotion category obtained in step S42, Wherein, u″ is the final uncertainty obtained in step S42, and K is the number of emotion categories.
8. A device for implementing the multimodal sentiment analysis method based on uncertainty estimation according to any one of claims 1 to 7, characterized in that: Including feature extraction module, sentiment classification module and prediction analysis module, Feature extraction module: obtains a multimodal sentiment analysis data set, extracts features from the audio data, image data, and text data in the video sample, and obtains corresponding audio features, video features, and text features; contextually represents the obtained audio features and video features, and obtains the represented audio features and video features; Sentiment classification module: The characterized audio features, video features and text features obtained by the feature extraction module are respectively input into their respective uncertainty estimation models based on Dirichlet distribution to obtain the Dirichlet distribution and uncertainty estimation of the corresponding audio, video and text modalities; the confidence of the Dirichlet distribution of the audio modality, the Dirichlet distribution of the video modality and the Dirichlet distribution of the text modality is solved using the DS evidence theory, and then the confidence and uncertainty of each modality are integrated to obtain the sentiment classification result and the overall uncertainty estimation; Prediction and analysis module: The sentiment classification results obtained by the sentiment classification module and the sample labels in the multimodal sentiment analysis dataset are used as input, and the cross entropy function is used as the loss function for training to obtain the trained uncertainty estimation model based on Dirichlet distribution; the video that needs to be sentimentally analyzed is used as input, and the trained uncertainty estimation model based on Dirichlet distribution is used to output the prediction results.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the computer program, the steps of the multimodal sentiment analysis method based on uncertainty estimation as described in any one of claims 1 to 7 are implemented.
