Multimodal sentiment analysis method and system with audio modality as target modality

Abstract

The present application relates to the technical field of multimodal data processing, specifically to a multimodal sentiment analysis method and system with audio modality as the target modality; in order to solve the problem of low accuracy of multimodal sentiment analysis in the prior art, the present application formulates that when the audio modality exists, the image modality and the text modality are translated into the audio modality, and after fusion, neural network processing is performed to obtain a multimodal sentiment prediction result; when the audio modality does not exist, the mutual information score is used to determine which modality contains more sentiment information, so that other modalities can be translated to it to obtain missing joint features, and the standard complete joint features are used as the target, and the missing joint features are approximated to the complete joint features for neural network processing to obtain a multimodal sentiment prediction result; this method is applied to the field of multimodal sentiment analysis, the processing process is more flexible, the analysis process is more comprehensive and reasonable, and the sentiment prediction results are more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and specifically to a multimodal sentiment analysis method and system with audio modality as target modality. Background Art

[0002] Multimodal sentiment analysis is a study of mining user opinions and emotional states from multimodal data such as text, images, and audio. Currently, due to some uncontrollable environmental and human factors, such as when the camera device is blocked, the user's image cannot be captured, and when there is a sudden noise, the user's words cannot be obtained, etc., multimodal data is lost and incompletely collected, reducing the accuracy of sentiment analysis results.

[0003] In response to the above problems, the current sentiment analysis methods under missing modalities include generative learning methods, joint learning methods, and modal translation. Generative learning methods aim to deal with the lack of uncertain modalities by generating new data that matches the observed distribution of existing modalities. Although generative learning can enhance the data set by simulating data distribution, improve the generalization ability of the model, and generate data for the missing modalities, thereby restoring the complete multimodal information to a certain extent, the generated data may have significant deviations from the real modal data, affecting the accuracy of sentiment analysis results. Joint learning methods aim to learn the potential features of the missing modalities directly from the existing modalities. This method can integrate information from different modalities, but the missing modality easily leads to the loss of balance between the modalities during feature fusion, and cannot accurately reflect the information of the remaining modalities. Modal translation methods aim to convert one modality into other modalities to solve the problem of missing modalities in multimodal data. However, this method currently makes other modalities learn the main modality when the text modality is missing. This method increases the uncertainty of the modal analysis results, affects the quality of modal analysis, and ultimately reduces the accuracy of sentiment analysis. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal emotion analysis method and system with audio modality as target modality.

[0005] The technical solution of the present invention is as follows:

[0006] A multimodal sentiment analysis method with audio modality as target modality comprises the following steps:

[0007] S1, obtain the multimodal data to be analyzed, determine whether there is an audio mode, if so, execute S2, if not, execute S3;

[0008] S2, respectively extracting context features of audio modality data, image modality data, and text modality data to obtain audio features, image features, and text features; translating image features and text features into audio features to obtain image translation features and text translation features; fusing the audio features, image translation features, and text translation features to obtain complete joint features, and executing S4;

[0009] S3, respectively extracting context features of image modality data and text modality data to obtain image features and text features; based on the joint probability and marginal probability between text features, image features and emotion tags, respectively obtaining the mutual information scores between text features, image features and emotion tags, and after comparison, taking the modality feature corresponding to the highest value of the mutual information score between emotion tags as the dominant modality feature; translating other modality features into dominant modality features to obtain translated dominant modality features; fusing the dominant modality features and translated dominant modality features to obtain missing joint features, and executing S5;

[0010] S4. Processing the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal sentiment prediction result;

[0011] S5. Use the neural network tool, take the standard complete joint feature as the target, approximate the missing joint feature to the complete joint feature, until the loss function reaches the set value, and obtain the missing feature vector. After the missing feature vector is classified, the multimodal emotion prediction result is obtained.

[0012] In S3, the mutual information scores between text features, image features and emotional labels are obtained based on the joint probability and marginal probability between them. MI It is obtained by the following formula:

[0013] ,

[0014] MI ( f,l ) indicates characteristics f With emotional labels l The mutual information score between f is a text feature or an image feature, F is the total number of features, L is the total number of sentiment labels, p ( f,l ) is a feature f and emotional labels l The joint probability mass function of p ( f )and p ( l ) are the characteristicsf and emotional labels l The marginal probability mass function of .

[0015] In S3, the operation of obtaining the mutual information score includes: inputting the mutual information scores between the text features, the image features and the emotional tags into the random forest regression model for learning and training respectively, and when the emotional tags are missing, using the trained random forest regression model to predict the optimized mutual information scores of the text features or image features; the optimized mutual information scores are used to perform comparison operations; in the operation of obtaining the optimized mutual information score based on the random forest regression model: the trained random forest regression model processes a number of text features or image features to obtain a first mutual information score for each text feature or image feature; the first mutual information scores of all text features or image features are sorted in descending order, the first mutual information scores that account for the first percentage of the sorting are averaged, the first mutual information scores that account for the second percentage of the sorting are medianed, and the mean and median are weighted averaged to obtain the second mutual information score as the optimized mutual information score for each text feature or image feature.

[0016] The operations of extracting context features in S2 are specifically as follows: audio modality data or image modality data or text modality data are processed by multi-head attention and residual connection to obtain audio residual data or image residual data or text residual data; audio residual data or image residual data or text residual data and audio attention data or image attention data or text attention data are processed by layer normalization and nonlinearity to obtain audio features or image features or text features.

[0017] In S2, the image features are translated into audio features to obtain the following operations: based on the image features, the key matrix and the value matrix, an image key vector and an image value vector are obtained; based on the audio features and the query matrix, an audio query vector is obtained; after the audio query vector and the image key vector are subjected to dot product operation, scaling operation and linear processing, a dot product operation is performed with the image value vector to obtain an initial image translation feature; the initial image translation feature is subjected to residual connection processing to obtain an initial image translation residual feature; the initial image translation residual feature and the initial image translation feature are subjected to layer normalization and nonlinear processing to obtain an image translation feature.

[0018] In S2, the fusion operation is specifically as follows: based on the interactive relationship among the audio modal data, the image modal data and the text modal data, the audio-image interaction weight matrix, the audio-text interaction weight matrix and the image-text interaction weight matrix are obtained; the audio-image interaction weight matrix and the image-text interaction weight matrix are respectively concatenated with the image translation features after dot product operation to obtain the image initial fusion features; the audio-image interaction weight matrix and the audio-text interaction weight matrix are respectively concatenated with the audio features after dot product operation to obtain the audio initial fusion features; the image-text interaction weight matrix and the audio-text interaction weight matrix are respectively concatenated with the text translation features after dot product operation to obtain the text initial fusion features; the image initial fusion features, the audio initial fusion features and the text initial fusion features are processed by weighted summation to obtain the complete joint features.

[0019] A multimodal sentiment analysis system with audio modality as target modality, used to implement the above-mentioned multimodal sentiment analysis method with audio modality as target modality, comprising:

[0020] The audio modality existence judgment module is used to obtain the multimodal data to be analyzed and judge whether the audio modality exists. If so, the complete joint feature generation module is executed; if not, the missing joint feature generation module is executed;

[0021] A complete joint feature generation module is used to extract context features of audio modal data, image modal data and text modal data respectively to obtain audio features, image features and text features; translate the image features and text features into audio features respectively to obtain image translation features and text translation features; perform a fusion operation on the audio features, image translation features and text translation features to obtain a complete joint feature for executing the first multimodal emotion prediction result generation module;

[0022] The missing joint feature generation module extracts the context features of the image modal data and the text modal data respectively to obtain the image features and the text features; based on the joint probability and marginal probability between the text features, the image features and the emotion tags, the mutual information scores between the text features, the image features and the emotion tags are obtained respectively, and after comparison, the modal feature corresponding to the highest value of the mutual information score between the emotion tags is used as the dominant modal feature; other modal features are translated into dominant modal features to obtain translated dominant modal features; the dominant modal features and the translated dominant modal features are fused to obtain the missing joint features, which are used to execute the second multimodal emotion prediction result generation module;

[0023] The first multimodal emotion prediction result generating module is used to process the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal emotion prediction result;

[0024] The second multimodal emotion prediction result generation module is used to use a neural network tool, with the standard complete joint feature as the target, and the missing joint feature is approximated to the complete joint feature until the loss function reaches a set value, thereby obtaining a missing feature vector. The missing feature vector is classified to obtain a multimodal emotion prediction result.

[0025] A multimodal emotion analysis device with an audio modality as a target modality comprises a processor and a memory, wherein the processor implements the above-mentioned multimodal emotion analysis method with the audio modality as a target modality when executing a computer program stored in the memory.

[0026] A computer-readable storage medium is used to store a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned multimodal emotion analysis method with the audio modality as the target modality.

[0027] The beneficial effects of the present invention are:

[0028] The present invention provides a multimodal emotion analysis method with an audio mode as a target mode. It determines whether the audio mode exists according to the multimodal data to be analyzed. When the audio mode exists, the image mode and the text mode are translated into the audio mode, the emotion clues in the image features and the text features are added, and the features are integrated to construct a more comprehensive and representative emotion feature, and a complete joint feature is obtained. After being processed by a neural network, a multimodal emotion prediction result is obtained. When the audio mode does not exist, the mutual information score is used to determine which mode contains more emotion information, so that other modes can be translated to it, so that the modality with better effect in different situations can be better utilized, and the missing joint feature is obtained. Then, the standard complete joint feature is taken as the target, and the missing joint feature is approximated to the complete joint feature for neural network processing to obtain a multimodal emotion prediction result. The method is flexible, the prediction and analysis process is more comprehensive and reasonable, and the emotion prediction result obtained has higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] By reading the detailed description of the preferred embodiment below, the scheme and advantages of the present application will become clear to those skilled in the art. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0030] In the attached picture:

[0031] Figure 1 It is a schematic diagram of the flow chart of the prediction method in the embodiment;

[0032] Figure 2 In the embodiment, the 4-classification result diagram of this embodiment and the existing method on the data sets CMU-MOSI and IEMOCAP is shown, where: Figure 2(a) in the figure shows the four-category sentiment classification results on the CMU-MOSI dataset. Figure 2 (b) shows the four-category sentiment classification results on the IEMOCAP dataset;

[0033] Figure 3 In the embodiment, the 7-classification result diagram of this embodiment and the existing method on the data sets CMU-MOSI and IEMOCAP is shown, where: Figure 3 (a) in the figure is the result of seven-category sentiment classification on the CMU-MOSI dataset. Figure 3 (b) in the figure shows the seven-category sentiment classification results on the IEMOCAP dataset. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0035] This embodiment provides a multimodal sentiment analysis method with audio modality as the target modality, see Figure 1 , including the following steps:

[0036] S1, obtain the multimodal data to be analyzed, determine whether there is an audio mode, if so, execute S2, if not, execute S3;

[0037] S2, respectively extracting context features of audio modality data, image modality data, and text modality data to obtain audio features, image features, and text features; translating image features and text features into audio features to obtain image translation features and text translation features; fusing the audio features, image translation features, and text translation features to obtain complete joint features, and executing S4;

[0038] S3, respectively extracting context features of image modality data and text modality data to obtain image features and text features; based on the joint probability and marginal probability between text features, image features and emotion tags, respectively obtaining the mutual information scores between text features, image features and emotion tags, and after comparison, taking the modality feature corresponding to the highest value of the mutual information score between emotion tags as the dominant modality feature; translating other modality features into dominant modality features to obtain translated dominant modality features; fusing the dominant modality features and translated dominant modality features to obtain missing joint features, and executing S5;

[0039] S4. Processing the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal sentiment prediction result;

[0040] S5. Use the neural network tool, take the standard complete joint feature as the target, approximate the missing joint feature to the complete joint feature, until the loss function reaches the set value, and obtain the missing feature vector. After the missing feature vector is classified, the multimodal emotion prediction result is obtained.

[0041] S1. Obtain multimodal data to be analyzed and determine whether there is an audio mode. If so, execute S2; if not, execute S3.

[0042] It is determined whether there is an audio modality based on the multimodal data to be analyzed. When there is an audio modality, S2 is executed, and the image modality and text modality are translated into the audio modality with the audio modality data as the main modality (target modality), thereby improving the accuracy and comprehensiveness of the sentiment analysis. When there is no audio modality, S3 is directly executed, thereby avoiding invalid attempts on the audio modality data and making the processing process more efficient.

[0043] S2. Extract the context features of the audio modal data, image modal data and text modal data respectively to obtain audio features, image features and text features; translate the image features and text features into audio features respectively to obtain image translation features and text translation features; fuse the audio features, image translation features and text translation features to obtain complete joint features, and execute S4.

[0044] When the audio modality is missing, the image modality and text modality are translated into the audio modality, the emotional clues in the image features and text features are added, and they are fused to construct a more comprehensive and representative emotional feature and obtain a complete joint feature.

[0045] First, the context features of audio modality data, image modality data and text modality data are extracted respectively to reduce noise interference and obtain audio features, image features and text features.

[0046] The operations for extracting contextual features are specifically as follows: audio modality data or image modality data or text modality data are processed by multi-head attention to obtain audio modality attention data or image modality attention data or text modality attention data; audio modality attention data or image modality attention data or text modality attention data are processed by residual connection to obtain audio residual data or image residual data or text residual data; audio residual data or image residual data or text residual data and corresponding audio attention data or image attention data or text attention data are processed by layer normalization and nonlinearity to obtain audio features or image features or text features.

[0047] Before the residual connection processing operation, the audio modality attention data is also subjected to spectrum analysis and feature extraction, the image modality attention data is subjected to image cropping, scaling and normalization, and the text modality attention data is subjected to word embedding and encoding.

[0048] Then, the image features and text features are translated into audio features respectively, the image features are translated into the rhythm and timbre of the audio, and the text features are translated into the melody and lyrics of the audio, and the emotional clues in the image features and text features are added to facilitate the fusion of image features and text features at the audio level, so as to have a deeper understanding of the comprehensive expression of emotions and obtain image translation features and text translation features.

[0049] The operations of translating image features into audio features to obtain image translation features are as follows: based on image features, key matrix and value matrix, image key vector and image value vector are obtained; based on audio features and query matrix, an audio query vector is obtained; after dot product operation, scaling and linear processing of the audio query vector and the image key vector, a dot product operation is performed with the image value vector to obtain initial image translation features; the initial image translation features are processed by residual connection to obtain initial image translation residual features; the initial image translation residual features and the initial image translation features are subjected to layer normalization and nonlinear processing to obtain image translation features.

[0050] The operation of translating text features into audio features is the same as the translation process of image features, and will not be repeated here to save space.

[0051] Finally, the audio features, image translation features, and text translation features are fused to construct a more comprehensive and representative sentiment feature and obtain a complete joint feature.

[0052] The fusion operation is specifically as follows: based on the interactive relationship between audio modal data, image modal data, and text modal data, the audio-image interaction weight matrix, the audio-text interaction weight matrix, and the image-text interaction weight matrix are obtained; the audio-image interaction weight matrix and the image-text interaction weight matrix are respectively concatenated with the image translation features after dot product operations to obtain the image initial fusion features; the audio-image interaction weight matrix and the audio-text interaction weight matrix are respectively concatenated with the audio features after dot product operations to obtain the audio initial fusion features; the image-text interaction weight matrix and the audio-text interaction weight matrix are respectively concatenated with the text translation features after dot product operations to obtain the text initial fusion features; the image initial fusion features, the audio initial fusion features, and the text initial fusion features are processed by weighted summation to obtain the complete joint features.

[0053] The operation of obtaining the audio-image interaction weight matrix is ​​as follows: the audio modal data and the image modal data are processed by the long short-term memory network respectively to obtain the audio vector and the image vector; the audio vector and the image vector are processed by singular value inverse decomposition, and then multiplied with the corresponding matrices of the Mel-frequency cepstral coefficients of the audio modal data and the high-level semantic feature vectors of the image modal data (which can be obtained by processing the image modal data with a graph convolutional neural network) to obtain the audio-image interaction weight matrix.

[0054] The specific operation of obtaining the audio-text interaction weight matrix is ​​as follows: the audio modal data and the text modal data are processed by the long short-term memory network respectively to obtain the audio vector and the text vector; the audio vector and the text vector are processed by singular value inverse decomposition, and then multiplied with the corresponding matrix of the Mel-frequency cepstral coefficient of the audio modal data and the word embedding vector of the text modal data to obtain the audio-text interaction weight matrix.

[0055] The specific operations for obtaining the image-text interaction weight matrix are as follows: the image modality data and the text modality data are processed by the long short-term memory network respectively to obtain the image vector and the text vector; the image vector and the text vector are processed by singular value inverse decomposition, and then multiplied with the corresponding matrix of the high-level semantic feature vector of the image modality data and the corresponding matrix of the word embedding vector of the text modality data to obtain the image-text interaction weight matrix.

[0056] S3. Extract the context features of the image modal data and the text modal data respectively to obtain image features and text features; based on the joint probability and marginal probability between the text features, image features and the sentiment labels, obtain the mutual information scores between the text features, image features and the sentiment labels respectively, and after comparison, take the modal feature corresponding to the highest value of the mutual information score between the sentiment labels as the dominant modal feature; translate other modal features into dominant modal features to obtain translated dominant modal features; fuse the dominant modal features and the translated dominant modal features to obtain the missing joint features, and execute S5.

[0057] When the audio modality is missing, the effect of translating the text modality or the image modality for emotion recognition is not much different, so this embodiment uses the mutual information score to determine which modality contains more emotional information, so as to let other modalities translate to it. This can better utilize the modality that works better in different situations, thereby helping to improve the accuracy of sentiment analysis.

[0058] First, context features of image modality data and text modality data are extracted respectively to obtain image features and text features. The method of extracting context features is the same as above, and will not be repeated here to save space.

[0059] Then, based on the joint probability and marginal probability between text features, image features and sentiment labels, the mutual information scores between text features, image features and sentiment labels are obtained respectively.

[0060] The operation of obtaining the mutual information score MI can be achieved by the following formula:

[0061] ,

[0062] MI ( f,l ) indicates characteristics f With emotional labels l The mutual information score between them is larger, indicating that the feature f With emotional labels l The greater the dependency between f is a text feature or an image feature, F is the total number of features, L is the total number of sentiment labels, p ( f,l ) is a feature f and emotional labels l The joint probability mass function of p ( f )and p ( l ) are the characteristics f and emotional labels l The marginal probability mass function of .

[0063] In order to further improve the accuracy of the mutual information score, after the mutual information score MI is obtained based on the above formula, the operation after obtaining the mutual information score includes: inputting the historical text features, image features and the mutual information scores with the emotional labels into the random forest regression model (corresponding random forest regression model) for learning and training respectively; when the emotional label is missing, the corresponding trained random forest regression model is used to predict the mutual information scores of the text features or image features as the optimized mutual information score for performing the comparison operation.

[0064] Among them, in the operation of obtaining the optimized mutual information score based on the random forest regression model: the trained random forest regression model processes several text features or image features to obtain a first mutual information score for each text feature or image feature; the first mutual information scores of all text features or image features are sorted in order from large to small, the first mutual information scores that occupy the first percentage (preferably 50%) of the sorting are averaged, and the first mutual information scores that occupy the second percentage (preferably 50%) are medianed, and the mean and the median are weighted averaged to obtain a second mutual information score as the optimized mutual information score of each text feature or image feature.

[0065] The first mutual information score can be obtained by the following formula:

[0066] ,

[0067] Y is the first mutual information score, B is the total number of decision trees in the random forest regression model, T b For the b The prediction output of a decision tree, X As input (text features or image features), the random forest regression model averages the prediction results of all decision trees to achieve the new input X In the random forest regression model, the prediction process of each decision tree is recorded as T b (X;θ b ) In this process, starting from the root node of the decision tree, the model parameters obtained in advance are θ b （ No. b The prediction parameters of the decision tree), including the decision threshold and the selected features, are input in order. X Guide the corresponding subtree until it finally reaches a leaf node. The output of the last leaf node becomes the decision tree for the given input X prediction results.

[0068] Next, after comparing the mutual information scores or optimized mutual information scores between the text features, image features and the sentiment labels, the modal feature corresponding to the highest value of the mutual information score between the sentiment labels is taken as the dominant modal feature.

[0069] Subsequently, other modal features (text features or image features) are translated into dominant modal features to obtain translated dominant modal features. The translation process is the same as the translation operation in S2, and will not be repeated here to save space.

[0070] Finally, the dominant modality feature and the translated dominant modality feature are fused to obtain the missing joint feature. The fusion operation can be achieved by weighted summing the dominant modality feature and the translated dominant modality feature, or by the fusion method in S2 above.

[0071] S4. When the audio modality exists, the complete joint features are processed using a neural network tool to obtain a complete joint feature vector. The complete joint feature vector is classified to obtain a multimodal emotion prediction result.

[0072] S5. When the audio modality does not exist, a neural network tool is used to approximate the missing joint features to the complete joint features with the standard complete joint features as the target until the loss function reaches the set value to obtain the missing feature vector. The missing feature vector is classified to obtain the multimodal emotion prediction result.

[0073] To verify the effect of the method of this embodiment, the following experiment was conducted.

[0074] Benchmark datasets. In the experiments, two well-known multimodal sentiment analysis benchmark datasets were used: CMU-MOSI and IEMOCAP.

[0075] CMU-MOSI: The CMU-MOSI (Carnegie Mellon University MultimodalOpinion Sentiment and Intensity) dataset is derived from 93 movie clips on YouTube and contains a total of 2,199 monologue images. In the CMU-MOSI dataset, each sample is assigned a sentiment score in the range of [-3,3] to quantify the intensity of the sentiment.

[0076] IEMOCAP: The IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset is based on emotional dialogue and communication between participants. It covers 5 sessions, each of which contains about 30 image segments, and each image contains at least 24 utterances. The emotional labels in the IEMOCAP dataset include: neutral, frustrated, angry, happy, sad, excited, surprised, fearful, disappointed, etc.

[0077] In the experiment, a three-classification emotion experiment was implemented on the CMU-MOSI dataset. Specifically, the emotion scores were divided into three labels: negative, neutral, and positive, which were achieved through the following mapping rules: the scores in the interval [-3, 0) were mapped to negative labels, the scores of 0 were mapped to neutral labels, and the scores in the interval (0, 3) were mapped to positive labels. At the same time, on the IEMOCAP dataset, a two-classification experiment was performed in the experiment, in which the emotion labels were divided into positive and negative categories: the labels [frustrated, angry, sad, fearful, disappointed] were classified as negative emotions; [happy, excited] were regarded as positive emotions.

[0078] Dataset preprocessing. In the image modality, the OpenFace2.0 toolbox is used to extract features from the face images in the image; this includes timestamps, confidence, successful recognition signs, eye movements, head postures, and facial movements. Each image sample eventually forms a high-dimensional feature vector with 709 features. For the text modality, feature extraction is performed using the pre-trained BERT model, which outputs text features with a dimension of 768, which can effectively capture the semantic information in the text. In the audio modality feature extraction, the Librosa library is used for processing, converting the audio signal to mono and resampling it to 16,000 Hz. The processing of the audio frame is based on 512 sample divisions, mainly extracting features including zero crossing rate, Mel frequency cepstral coefficients (MFCC), and constant Q transform (CQT). The dimension of the final generated audio feature vector is 33.

[0079] Experimental parameter settings. The experiments were performed on a personal computer equipped with the following hardware: Windows 10 operating system, Intel(R) Core(TM) i9- 10900K and Nvidia 3090 graphics processors, and 96GB memory. The model framework used in the experiment is based on TensorFlow 1.14.0 version, and the programming language is Python 3.7. Among them, the learning rate is set to 0.001, the batch size is adjusted to 32, and the hidden layer size is 300. In order to fine-tune the proposed model, the Adam optimization algorithm is selected in the experiment to reduce the value of the loss function. The experimental cycle is set to 20, and the loss weight is 0.1.

[0080] Evaluation indicators. Accuracy (Accuracy, Acc) and macro F1 score (Macro F1, M-F1) are used as evaluation indicators in the experiment.

[0081] in, Acc=N ture / N , N ture is the number of samples predicted correctly, N is the total number of samples; M-F1=2P·R / (P+ R) , P is the positive prediction value, R is the recall value.

[0082] Existing methods for experimental contrast. In order to prove the effectiveness of the method of this embodiment, several cutting-edge methods were selected in the experiment to compare the effects with the method of this embodiment. The existing methods include: AE, a generalized autoencoder framework, which aims to optimize the consistency between the output and input of the neural network and deal with linear and nonlinear autoencoding problems; CRA, a missing modality reconstruction model based on the cascaded residual structure of the autoencoder, which approximates the input data through a residual connection mechanism; MCTN, which promotes interaction between modalities through modal translation and effectively supports robust joint relationship learning; TransM, a multimodal feature fusion model based on end-to-end translation, which realizes multimodal interaction through cyclic conversion between modes; MMIN, a feature reconstruction model for processing missing modalities, which uses cascaded residual autoencoders and forward and backward imagination modules to convert available and missing modalities into each other; ICDN, which combines consistency and difference networks to achieve cross-modal interaction through cross-modal The modality converter maps the information of other modalities to the target modality, thereby realizing inter-modal interaction; MRAN, the model uses multimodality and missing index embedding to guide the reconstruction of missing modalities, and aligns image, audio and text features to meet the challenges of missing modalities; TATE_C, uses label-assisted transformer encoders to handle various uncertain missing modalities, and uses pre-trained models to guide the learning process of joint representation; MTMSA, modality translation network, converts image and audio modalities into text modalities to handle missing modalities and capture deep interactions between different modalities, and the model fully utilizes the advantages of text modality; TATE_J, based on the above research, assigns different weights to different modalities to make full use of each modality; SMCMSA, for missing modalities, through a pre-built complete modality sample database, the model selects samples similar to the missing modality for missing completion.

[0083] Experimental results. In the experiment, the performance of the TMMSA model is verified by implementing 3-classification on the CMU-MOSI dataset and 2-classification on the IEMOCAP dataset. The experiment consists of two parts. In the first part, the case of single-modal loss is considered. In the second part, the case of multi-modal loss is considered.

[0084] Experiments for single-modal missingness. In the experiments, the modal missingness rates were set to 0, 0.1, 0.2, 0.3, 0.4 and 0.5, respectively. The experimental results are shown in Tables 1 and 2. From Table 1, it can be found that for the CMU-MOSI dataset, at all missingness rates (0, 0.1, 0.2, 0.3, 0.4 and 0.5), the method of this embodiment outperforms the other 9 existing methods in two evaluation indicators (ACC and M-F1). In addition, compared with other existing methods, when the missingness rate is 0.3, the method of this embodiment improves the M-F1 index value by 0.77% to 5.57% and the ACC value by 1.74% to 10.98%. At the same time, from Table 2, it can be found that for the IEMOCAP dataset, at all missingness rates (0, 0.1, 0.2, 0.3, 0.4 and 0.5), the method of this embodiment outperforms the other 9 existing methods in two evaluation indicators (ACC and M-F1).

[0085] Table 1 Experimental results of different methods on the CMU-MOSI dataset when unimodal loss occurs

[0086] .

[0087] Table 2 Experimental results of different methods on the IEMOCAP dataset when single modality is missing

[0088] .

[0089] Experiments for multi-modal missingness. In the experiments, the modal missingness rates were set to 0, 0.1, 0.2, 0.3, 0.4 and 0.5 respectively. And on two public datasets CMU-MOSI and IEMOCAP, by setting different modal missingness rates, multiple modal uncertain missingness experiments were carried out, and the experimental results are shown in Tables 3 and 4. It can be found from Table 3 that for the dataset CMU-MOSI, at all missingness rates (0, 0.1, 0.2, 0.3, 0.4 and 0.5), the method of this embodiment outperforms the other 9 existing methods in two evaluation indicators (ACC and M-F1). In addition, compared with other existing methods, when the missing rate is 0.5, the method of this embodiment improves the M-F1 index value by 1.10% to 12.73%, and the ACC value by 1.00% to 11.59%. For the IEMOCAP dataset, at all missing rates (0, 0.1, 0.2, 0.3, 0.4, and 0.5), the method of this embodiment outperforms the other nine baseline models in two evaluation indicators (ACC and M-F1). In addition, it can be found from Table 4 that on the IEMOCAP dataset, compared with other baseline models, when the missing rate is 0.5, the method of this embodiment improves the M-F1 indicator value by 0.38% to 10.20% and the ACC value by 0.77% to 7.80%.

[0090] Table 3 Experimental results of different methods on the CMU-MOSI dataset when multimodality is missing

[0091] .

[0092] Table 4 Experimental results of different methods on the IEMOCAP dataset when multimodality is missing

[0093] .

[0094] Multi-classification verification experiment. In order to demonstrate the multi-classification performance of the method in this embodiment, the experiment selects the emotion labels of happiness, anger, sadness and neutrality to conduct a four-classification experiment on the IEMOCAP dataset. The emotion labels of happiness, anger, sadness, neutrality, frustration, excitement and surprise are selected to conduct a seven-classification experiment on the IEMOCAP dataset. The detailed distribution of the multi-class labels of the IEMOCAP dataset is shown in Table 5. The experimental results are shown in Table 5. Figure 2 and Figure 3 shown.

[0095] Table 5 Detailed distribution of IEMOCAP dataset

[0096] .

[0097] The experimental results of four types of sentiment classification are as follows Figure 2 As shown, from Figure 2 (a) and Figure 2 It can be found in (b) that when the missing rate is set to 0, the M-F1 and ACC of the method in this embodiment are 3.27% and 2.38% larger than those of the model ICDN, respectively. When the missing rate is 0.1, the M-F1 value of the method in this embodiment is 21.15% higher than that of the model TATE_J, and the ACC value of the method in this embodiment is 12.39% higher than that of the model TATE_J. When the missing rate is set to 0.5, compared with MTMSA, the method in this embodiment improves by 4.23% in M-F1 and 1.37% in ACC.

[0098] The experimental results of seven types of sentiment classification are as follows Figure 3 As shown, from Figure 3 (a) and Figure 3It can be found in (b) that when the missing rate is 0, the method of this embodiment has a 13.18% improvement in M-F1 and a 6.56% improvement in ACC compared with ICDN. When the missing rate is set to 0.1, the M-F1 value of the method of this embodiment is 17.87% higher than that of MRAN, and the ACC value of the method of this embodiment is 12.10% higher than that of MRAN. Compared with MTMSA, when the missing rate is set to 0.5, the method of this embodiment improves 6.47% in M-F1 and 3.07% in ACC. Based on the above experimental results, it can be concluded that the method of this embodiment has better performance in multi-classification of emotions.

[0099] Figure 2 and Figure 3 The results show that in the four-category sentiment analysis and seven-category sentiment analysis tasks, when the missing rate takes different values, the method of this embodiment is always better than the other five existing technologies in F1 and ACC.

[0100] According to the above experimental results, it can be seen that the method of this embodiment has better performance in the multi-classification sentiment classification task with uncertain modality loss.

[0101] This embodiment further provides a multimodal sentiment analysis system with the audio modality as the target modality, which is used to implement the above-mentioned multimodal sentiment analysis method with the audio modality as the target modality, including:

[0102] The audio modality existence judgment module is used to obtain the multimodal data to be analyzed and judge whether the audio modality exists. If so, the complete joint feature generation module is executed; if not, the missing joint feature generation module is executed;

[0103] A complete joint feature generation module is used to extract context features of audio modal data, image modal data and text modal data respectively to obtain audio features, image features and text features; translate the image features and text features into audio features respectively to obtain image translation features and text translation features; perform a fusion operation on the audio features, image translation features and text translation features to obtain a complete joint feature for executing the first multimodal emotion prediction result generation module;

[0104] The missing joint feature generation module extracts the context features of the image modal data and the text modal data respectively to obtain the image features and the text features; based on the joint probability and marginal probability between the text features, the image features and the emotion tags, the mutual information scores between the text features, the image features and the emotion tags are obtained respectively, and after comparison, the modal feature corresponding to the highest value of the mutual information score between the emotion tags is used as the dominant modal feature; other modal features are translated into dominant modal features to obtain translated dominant modal features; the dominant modal features and the translated dominant modal features are fused to obtain the missing joint features, which are used to execute the second multimodal emotion prediction result generation module;

[0105] The first multimodal emotion prediction result generating module is used to process the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal emotion prediction result;

[0106] The second multimodal emotion prediction result generation module is used to use a neural network tool, with the standard complete joint feature as the target, and the missing joint feature is approximated to the complete joint feature until the loss function reaches a set value, thereby obtaining a missing feature vector. The missing feature vector is classified to obtain a multimodal emotion prediction result.

[0107] This embodiment also provides a multimodal emotion analysis device with audio modality as the target modality, including a processor and a memory, wherein the processor implements the above-mentioned multimodal emotion analysis method with audio modality as the target modality when executing a computer program stored in the memory.

[0108] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned multimodal emotion analysis method with the audio modality as the target modality.

[0109] The present embodiment provides a multimodal sentiment analysis method with an audio modality as a target modality. It determines whether the audio modality exists according to the multimodal data to be analyzed. When the audio modality exists, the image modality and the text modality are translated into the audio modality, and the sentiment clues in the image features and the text features are added and fused to construct a more comprehensive and representative sentiment feature to obtain a complete joint feature, which is processed by a neural network to obtain a multimodal sentiment prediction result. When the audio modality does not exist, the mutual information score is used to determine which modality contains more sentiment information, so that other modalities can be translated to it, so that the modality with better effects in different situations can be better utilized to obtain the missing joint feature, and then the standard complete joint feature is used as the target, and the missing joint feature is approximated to the complete joint feature for neural network processing to obtain the multimodal sentiment prediction result. The method is flexible, the prediction and analysis process is more comprehensive and reasonable, and the sentiment prediction result is more accurate.

Claims

1. A multimodal sentiment analysis method with audio modality as target modality, characterized in that: The following steps are involved: S1, obtain the multimodal data to be analyzed, determine whether there is an audio mode, if so, execute S2, if not, execute S3; S2, respectively extracting context features of the audio modality data, the image modality data, and the text modality data to obtain audio features, image features, and text features; translating the image features and the text features into audio features to obtain image translation features and text translation features; The audio features, image translation features and text translation features are fused to obtain complete joint features, and S4 is executed; S3, respectively extracting context features of image modal data and text modal data to obtain image features and text features; based on the joint probability and marginal probability between text features, image features and emotional labels, respectively obtaining mutual information scores between text features, image features and emotional labels, and after comparison, taking the modal feature corresponding to the highest value of the mutual information score between the emotional labels as the dominant modal feature; translating other modal features into dominant modal features to obtain translated dominant modal features; The dominant modality feature and the translated dominant modality feature are fused to obtain the missing joint feature, and S5 is executed; S4. Processing the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal sentiment prediction result; S5. Use the neural network tool, take the standard complete joint feature as the target, approximate the missing joint feature to the complete joint feature, until the loss function reaches the set value, and obtain the missing feature vector. After the missing feature vector is classified, the multimodal emotion prediction result is obtained.

2. The multimodal sentiment analysis method with audio modality as target modality according to claim 1, characterized in that: In S3, based on the joint probability and marginal probability between the text features, image features and emotion tags, the mutual information scores between the text features, image features and emotion tags are obtained respectively. MI It is obtained by the following formula: ， MI ( f,l ) indicates characteristics f With emotional labels l The mutual information score between f is a text feature or an image feature, F is the total number of features, L is the total number of sentiment labels, p ( f,l ) is a feature f and emotional labels l The joint probability mass function of p ( f )and p ( l ) are the characteristics f and emotional labels l The marginal probability mass function of .

3. The multimodal sentiment analysis method with audio modality as target modality according to claim 1, characterized in that: In S3, the operation of obtaining the mutual information score includes: inputting the mutual information scores between the text features, the image features and the emotional tags into the random forest regression model for learning and training respectively, and when the emotional tags are missing, using the trained random forest regression model to predict the optimized mutual information scores of the text features or the image features, and the optimized mutual information scores are used to perform the comparison operation; In the operation of obtaining the optimized mutual information score based on the random forest regression model: the trained random forest regression model processes several text features or image features to obtain a first mutual information score for each text feature or image feature; the first mutual information scores of all text features or image features are sorted in descending order, the first mutual information scores that occupy the first percentage of the sorting are averaged, the first mutual information scores that occupy the second percentage of the sorting are medianed, the mean and the median are weighted averaged to obtain a second mutual information score as the optimized mutual information score for each text feature or image feature.

4. The multimodal sentiment analysis method with audio modality as target modality according to claim 1, characterized in that: In S2, the operation of extracting context features is specifically as follows: The audio modality data, the image modality data, or the text modality data is processed by multi-head attention and residual connection to obtain audio residual data, image residual data, or text residual data; The audio residual data or the image residual data or the text residual data and the audio attention data or the image attention data or the text attention data are subjected to layer normalization and nonlinear processing to obtain audio features or image features or text features.

5. The multimodal sentiment analysis method with audio modality as target modality according to claim 1, characterized in that: In S2, the operation of translating the image features into audio features to obtain the image translation features is specifically as follows: Based on the image features, the key item matrix and the value item matrix, an image key item vector and an image value item vector are obtained; based on the audio features and the query item matrix, an audio query vector is obtained; The audio query vector and the image key vector are subjected to dot product operation, scaling and linear processing, and then dot product operation is performed with the image value vector to obtain the initial image translation feature; The initial image translation features are processed by residual connection to obtain the initial image translation residual features; The initial image translation residual features and the initial image translation features are subjected to layer normalization and nonlinear processing to obtain image translation features.

6. The multimodal sentiment analysis method with audio modality as target modality according to claim 1, characterized in that: In S2, the fusion operation is specifically as follows: Based on the interactive relationship between audio modality data, image modality data, and text modality data, the audio-image interaction weight matrix, the audio-text interaction weight matrix, and the image-text interaction weight matrix are obtained; The audio-image interaction weight matrix and the image-text interaction weight matrix are respectively concatenated with the image translation features after dot product operations to obtain the initial fusion features of the image; The audio-image interaction weight matrix and the audio-text interaction weight matrix are concatenated with the audio features after dot product operations to obtain the initial audio fusion features. The image-text interaction weight matrix and the audio-text interaction weight matrix are concatenated with the text translation features after dot product operations to obtain the initial fusion features of the text; The image initial fusion features, the audio initial fusion features and the text initial fusion features are processed by weighted summation to obtain a complete joint feature.

7. A multimodal sentiment analysis system with audio modality as target modality, characterized in that: The method for implementing the multimodal sentiment analysis method with the audio modality as the target modality as claimed in claim 1 comprises: The audio modality existence judgment module is used to obtain the multimodal data to be analyzed and judge whether the audio modality exists. If so, the complete joint feature generation module is executed; if not, the missing joint feature generation module is executed; A complete joint feature generation module is used to extract context features of audio modal data, image modal data and text modal data respectively to obtain audio features, image features and text features; translate the image features and text features into audio features respectively to obtain image translation features and text translation features; perform a fusion operation on the audio features, image translation features and text translation features to obtain a complete joint feature for executing the first multimodal emotion prediction result generation module; The missing joint feature generation module extracts the context features of the image modal data and the text modal data respectively to obtain the image features and the text features; based on the joint probability and marginal probability between the text features, the image features and the emotion tags, the mutual information scores between the text features, the image features and the emotion tags are obtained respectively, and after comparison, the modal feature corresponding to the highest value of the mutual information score between the emotion tags is used as the dominant modal feature; other modal features are translated into dominant modal features to obtain translated dominant modal features; the dominant modal features and the translated dominant modal features are fused to obtain the missing joint features, which are used to execute the second multimodal emotion prediction result generation module; The first multimodal emotion prediction result generating module is used to process the complete joint features using a neural network tool to obtain a complete joint feature vector, and the complete joint feature vector is classified to obtain a multimodal emotion prediction result; The second multimodal emotion prediction result generation module is used to use a neural network tool, with the standard complete joint feature as the target, and the missing joint feature is approximated to the complete joint feature until the loss function reaches a set value, thereby obtaining a missing feature vector. The missing feature vector is classified to obtain a multimodal emotion prediction result.

8. A multimodal sentiment analysis device with audio modality as target modality, characterized in that: It comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, it implements the multimodal emotion analysis method with the audio modality as the target modality as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the multimodal sentiment analysis method with the audio modality as the target modality is implemented as described in any one of claims 1-6.

Images