Multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement

Abstract

The invention discloses a multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement, and the method comprises the steps: (1) carrying out the preprocessing of a public data set, and respectively extracting text, visual and acoustic features in the data set; (2) constructing an MICTE multi-modal sentiment analysis model, effectively fusing rich sentiment information in text modal information into a visual modal and an acoustic modal through a cross-modal text attention enhancement module CTEA, enhancing the relevance between modals at an input level through a mutual information maximization module MIM, further extracting key information related to tasks at a fusion level, and obtaining an MICTE multi-modal sentiment analysis model; eliminating redundant information; (3) training and verifying the MICTE model by using the training set and the verification set, and selecting the model with the best performance as a final model; and (4) inputting the test set into the finally trained MICTE model to obtain an emotion analysis result. According to the multi-modal sentiment analysis method, the relevance among different modals is deeply mined, and the quality of the fused modals is remarkably improved, so that the accuracy of multi-modal sentiment analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and specifically refers to a multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement (MICTE). Background Art

[0002] Sentiment analysis has always been one of the main research directions in the field of natural language processing. Early research mainly focused on sentiment analysis of text. In recent years, with the breakthrough development of social media, more and more people tend to express their emotions on the Internet in ways such as short videos. Multimodal data such as movies and short videos have emerged continuously. Researchers have begun to use several common modal data such as text, video (i.e., visual signals), and speech (i.e., acoustic signals) to achieve sentiment classification. These modalities complement each other and work together in the field of sentiment analysis. Deeply mining and understanding the sentiment information in multimodal data, namely the so-called multi-modal sentiment analysis, has become a popular research topic.

[0003] In the current field of multi-modal sentiment analysis, existing research generally focuses on mining the commonalities between different modal data and using the attention mechanism to extract common information to assist in sentiment analysis. However, different modal data contain varying degrees of redundant information unrelated to sentiment, and the content and ways of expressing sentiment have certain differences, especially for visual signals and acoustic signals. This makes it difficult for network models to accurately analyze useful information. The main challenge of the multi-modal sentiment analysis task is to solve the problem of how to obtain the representation features of single-modal data. In addition, multi-modal sentiment analysis also needs to explore the interaction between modalities, fuse different single-modal data representations, and complete the integration of sentiment information. Current fusion methods have achieved certain results, but there is still room for improvement. On the one hand, traditional methods usually achieve feature fusion by simply combining multi-modal features or performing linear mixing, and fail to fully explore the deep connections between modalities. On the other hand, these methods also cannot effectively control the information flow from input to the fusion process, resulting in problems such as information redundancy, omission, and noise during feature fusion. Therefore, how to effectively mine the correlation between different modalities and how to improve the quality of the fused modalities remains a difficult problem to be solved. Summary of the Invention

[0004] In order to overcome the deficiencies of the existing methods, the present invention proposes a multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement, which solves problems such as low correlation between modalities and poor quality of fused features, and is of great significance for improving the quality of multi-modal fusion.

[0005] The first aspect of the present invention is to provide a multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement, including the following steps:

[0006] (1) After preprocessing the public dataset, extract the text, visual, and acoustic features in the dataset respectively;

[0007] (2) Construct the MICTE multi-modal sentiment analysis model;

[0008] (3) Through the cross-modal text-enhanced attention module CTEA (Cross-modal Text-Enhanced Attention), integrate the text-modal information into the visual and acoustic modalities;

[0009] (4) Through the mutual information maximization module MIM (Mutual Information Maximization), extract the key information related to the task at the input level and the fusion level respectively;

[0010] (5) Use the training set and the validation set to train and validate the MICTE model;

[0011] (6) Input the test set into the finally trained MICTE model to obtain the sentiment analysis result.

[0012] Preferably, the specific steps of step 1 include:

[0013] The multi-modal sentiment analysis dataset is the CMU-MOSI dataset, which contains text-modal, visual-modal, and acoustic-modal information. Reasonably divide the dataset, including using the training set for model training, the validation set for parameter optimization and performance evaluation, and the test set for the final detection effect test. Use the BERT (Bidirectional Encoder Representations from Transformers) language model to extract text-modal features, and use the LSTM (unidirectional Long Short-Term Memory) network to extract acoustic-modal and visual-modal features. Subsequently, use a temporal convolutional network to capture the sequential features of the three modalities at each moment, and standardize the obtained hidden states to a unified size.

[0014] Preferably, the specific steps of step 2 include:

[0015] The specific design idea of the cross-modal text-enhanced attention module CTEA is:

[0016] Receiving text features and another feature as inputs using text-based multi-head attention, and taking the obtained text-based multi-head attention output as the input of the cross-modal Transformer;

[0017] The mutual information maximization module MIM is divided into input-level mutual information maximization and fusion-level mutual information maximization. The specific design idea is as follows:

[0018] Input-level mutual information maximization: By calculating the exact lower bound of mutual information, maximizing the mutual information between multi-modal input pairs. Using non-parametric methods to estimate the entropy of mutual information, and constructing Gaussian mixture models for negative and non-negative class samples respectively;

[0019] Fusion-level mutual information maximization: Adopting a scoring function to measure the correlation between the normalized prediction vector and the actual vector.

[0020] Preferably, step 3 specifically includes:

[0021] The core idea of this method is to use text as a medium, promote the effective interaction between the visual modality and the acoustic modality by calculating the attention weights between different modalities, thereby enhancing the information exchange between modalities, and helping to more comprehensively understand and analyze the emotional content in multi-modal data. The three extracted modality features are input into the cross-modal text-enhanced attention module CTEA, and through this module, the rich emotional information in the text is effectively integrated into the visual modality and the acoustic modality.

[0022] Preferably, step 4 specifically includes:

[0023] The core idea of this method is that by implementing a hierarchical mutual information maximization strategy at the input level and the fusion level, the model can more effectively capture the dependencies between different levels of modalities, exclude task-irrelevant noise, and enhance the features of each modality, thereby constructing a more detailed multi-modal feature representation.

[0024] Preferably, step 5 specifically includes:

[0025] The experimental environment is set on an RTX 4050 GPU, the operating system is Windows 11, the programming language is Python 3.11, and the deep learning framework is PyTorch 2.2.2.

[0026] The batch size of the MICTE model is 32, the learning rate is 1e-4, the number of layers of multi-head attention is 5, and the number of layers of cross-modal text-enhanced attention is 5.

[0027] In the training stage, the model is iteratively trained using the training data. The model continuously optimizes the model parameters through the overall loss until the loss function converges. After training, the binary classification accuracy Acc-2, seven-class classification accuracy Acc-7, mean absolute error MAE, Pearson correlation coefficient Corr, and F1 value of the validation set are used as the main evaluation metrics for the model performance to evaluate the performance of sentiment analysis, and the model with the best performance is selected as the final model.

[0028] The second aspect of the present invention relates to a multi-modal sentiment analysis device based on mutual information and cross-modal text enhancement, including a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement of the present invention.

[0029] The third aspect of the present invention relates to a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement of the present invention.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] Through the cross-modal text enhancement attention module CTEA, rich sentiment information in the text is effectively integrated into the visual modality and the acoustic modality, thereby enhancing the sentiment expression ability of non-text modalities; through the mutual information maximization module MIM, the correlation between modalities is enhanced at the input layer, and key information related to the task is further refined at the fusion layer, and redundant information is removed, thereby optimizing the representation of multi-modal features. This method deeply explores the correlation between different modalities, significantly improves the quality of the fusion modality, and thus improves the accuracy of multi-modal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0033] Figure 2 It is a structural diagram of the MICTE multi-modal sentiment analysis model designed in the embodiment of the present invention;

[0034] Figure 3 It is a structural diagram of the cross-modal text enhancement attention module of the present invention;

[0035] Figure 4 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0036] The present invention will be further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0037] Example 1

[0038] As Figure 1 shown, a multi-modal sentiment analysis method based on MICTE includes the following steps:

[0039] Step 1: After preprocessing the public dataset, extract the text, visual, and acoustic features in the dataset respectively.

[0040] Step 1.1: In this example, the CMU-MOSI dataset is used, and the content in it is collected from the video website YouTube. This dataset includes a total of 93 videos and 2,199 video clips. The videos include 89 volunteers from various cultural backgrounds, among which there are 41 females and 48 males. The language used in the videos is English. Each video clip is labeled with an emotion value from -3 to +3, reflecting the polarity (positive / negative) and intensity (absolute value) of the emotion. The present invention divides the 2,199 video materials into three different groups. Specifically, 1,284 videos are used as training data, 229 videos are used as validation data, and the remaining 686 videos are used as test data.

[0041] Step 1.2: Use the BERT language model to extract text modality features, use the LSTM network to extract acoustic modality features and visual modality features, and input the features of the three modalities obtained through processing into the subsequent cross-modal text enhanced attention module. The formula for feature extraction is as follows:

[0042]

[0043]

[0044] In the formula m ∈ {t, a, v} represents the original input features of modality m, m ∈ {t, a, v} represents the features of modality m extracted from the original input X m .

[0045] Subsequently, a temporal convolutional network Conv1D is used to capture the sequential features of the three modalities at each moment, and the obtained hidden state is normalized to a unified size for use in the subsequent cross-modal text enhanced attention module. The specific formula is as follows:

[0046]

[0047] Among them m ∈ {t, a, v} represents the features extracted from modality m by the temporal convolutional network, and k m represents the size of the convolutional kernel used for each corresponding modality m in the temporal convolutional network.

[0048] Step 2: Construct the MICTE multi-modal sentiment analysis model.

[0049] Step 2.1: The multi-modal sentiment analysis model based on MICTE is as Figure 2 shown. The design idea of the cross-modal text enhanced attention module CTEA is as follows:

[0050] Use the text-based multi-head attention to receive text features and another feature as inputs, and the output of the text-based multi-head attention obtained is used as the input of the cross-modal Transformer;

[0051] Step 2.2: The mutual information maximization module MIM is divided into input-level mutual information maximization and fusion-level mutual information maximization. The specific design idea is as follows:

[0052] Input-level mutual information maximization: By calculating the exact lower bound of mutual information, maximize the mutual information between multi-modal input pairs. Given that text data provides richer information, optimize the interaction boundaries between the two pairs of modalities, namely text and acoustics, and text and vision. Then use non-parametric methods to estimate the entropy of mutual information, and construct Gaussian mixture models for negative and non-negative class samples respectively. Finally, the loss function of input-level mutual information maximization can be obtained.

[0053] Fusion-level mutual information maximization: When maximizing the mutual information between the fusion result and the input modalities, it is necessary to optimize the output results generated by the fusion network. For this, a scoring function is used to measure the correlation between the normalized prediction vector and the actual vector. The positive and negative samples in noise contrast estimation refer to the pairings of single-modal representations and fusion representations with fusion relationships and the pairings of single-modal representations and fusion representations of other samples in the same batch respectively. Through noise contrast estimation, the scoring function can be calculated as the loss function of fusion-level mutual information maximization.

[0054] Step 3: Through the cross-modal text enhanced attention module CTEA, integrate the text modality information into the visual modality and the acoustic modality.

[0055] Step 3.1: The core idea of this method is to use text as a medium, and by calculating the attention weights between different modalities, promote the effective interaction between the visual modality and the acoustic modality, so as to enhance the information exchange between modalities, which helps to more comprehensively understand and analyze the emotional content in multi-modal data. Input the three modality features extracted into the cross-modal text enhanced attention module CTEA, and through this module, effectively integrate the rich emotional information in the text into the visual modality and the acoustic modality. Taking the cross-modal text enhanced attention structure that maps the acoustic modality to the visual modality as an example, the specific structure is as Figure 3 shown.

[0056] First, use text-based multi-head attention to receive acoustic features and text features as inputs, where the acoustic features provide the query Q, and the text features provide the key K and value V. The specific formula is as follows:

[0057]

[0058]

[0059] Where are the linear transformation weight matrices respectively.

[0060] The text-based multi-head attention consists of m groups, and the specific output formula is as follows:

[0061] H c = concat(Attention(Q, K, V)1, Attention(Q, K, V)2,... Attention(Q, K, V)5)W 0 (6)

[0062] Attention(Q, K, V) = softmax(QK T )V (7)

[0063] Where

[0064] Then, take H v as and the output H c of the obtained text-based multi-head attention together as the input of the 0th layer of the cross-modal Transformer. After that, the input of each layer is calculated based on the output of the previous layer. The specific calculation process of the i-th layer is as follows:

[0065]

[0066] Where FFN represents the feed-forward neural network, LN represents layer normalization, represents the multi-head attention of the i-th layer. After n layers of processing, the visual modality output F a→v that maps the acoustic modality is obtained. Similarly, the visual modality is attached to the acoustic modality using the same processing method to obtain F v→a .

[0067] Step 4: Through the mutual information maximization module, extract the task-related key information at the input level and the fusion level respectively.

[0068] Step 4.1: Input-level mutual information maximization: Calculate the exact lower bound of mutual information using the following method to maximize the mutual information between multi-modal input pairs. The specific formula is as follows:

[0069]

[0070] Where H(y) is the entropy of y, and q(y|x) is used to approximate p(y|x).

[0071] Given that the text data provides richer information, optimize the interaction boundaries between the two pairs of modalities: text and acoustics, and text and vision. Let The parameters μ and σ in the formula are determined by x, and are usually calculated by means of a regression multi-layer perceptron. The loss function for likelihood maximization is specifically as follows:

[0072]

[0073] Where N is the training batch size, and tv and ta are the sum of the probabilities of two predictors.

[0074] Then, use a non-parametric method to estimate H(Y), and construct Gaussian mixture models for negative and non-negative class samples respectively. The specific formula for H(Y) is as follows:

[0075]

[0076] Where ∑ is the covariance matrix, and det(∑) is the determinant of ∑.

[0077] The loss function for maximizing the input-level mutual information is as follows:

[0078]

[0079] Step 4.2: Maximize the fusion-level mutual information: When maximizing the mutual information between the fusion result and the input modality, it is necessary to optimize the output result Z = F(X t , X a , X v ) generated by the fusion network. For this, a scoring function is adopted to measure the correlation between the normalized prediction vector and the actual vector. The calculation formulas for the prediction vector and the actual vector are as follows:

[0080]

[0081] Where is a neural network with parameter , Z represents the fusion feature, and h m represents modality m. ||·||2 is the Euclidean norm, and it is divided by it to obtain a unit-length vector.

[0082] The CPC score is expressed as s(h m, Z), and this score is proportional to the significance of the result of modal fusion, reflecting the effectiveness in capturing basic information from different modalities during the fusion process. Noise Contrastive Estimation is denoted as E H , where the positive and negative samples respectively refer to the pairings of the single-modal representations and the fused representations with a fusion relationship, and the pairings of the single-modal representations and the fused representations of other samples in the same batch. The specific formula is as follows:

[0083]

[0084] The loss function for maximizing the fusion-level mutual information is as follows:

[0085]

[0086] Step 5: Use the training set and the validation set to train and validate the MICTE model.

[0087] Step 5.1: After the model's comprehensive processing, visual and acoustic feature representations combined with text information are obtained, and these features are fused with the text vector. Then, through a fully connected network, the fused features are mapped to the output space of sentiment classification to obtain sentiment labels Finally, the mean absolute error MAE between the sentiment label and the true label y is used as the loss function of the task. The specific formula is as follows:

[0088]

[0089] where N represents the batch size.

[0090] By calculating the weighted sum of these task losses as the overall loss of the model. The specific formula is as follows:

[0091] L all = L task + αL BA + βL CPC (19)

[0092] where α and β are parameters that adjust the influence of maximizing mutual information.

[0093] The experimental environment is set on an RTX 4050 GPU, with the operating system being Windows 11, the programming language being Python 3.11, and the deep learning framework being PyTorch 2.2.2.

[0094] The batch size of the model is 32, using Bert with a hidden size of 768, a learning rate of 5e-5, and a learning rate of 1e-4 for other parameters. The Adam optimizer based on stochastic gradient descent is used to train the model. The kernel size of the temporal convolutional network is 3, the hidden layer size of the LSTM is 32, the weight decay coefficient is 0.1, the number of layers of multi-head attention is 5, the number of layers of cross-modal text enhanced attention is 5, and both α and β in the overall loss function are 0.1.

[0095] In the training stage, the model is iteratively trained using the training data. The model continuously optimizes the model parameters through the overall loss until the loss function converges. After training is completed, the binary classification accuracy Acc-2, seven-class classification accuracy Acc-7, mean absolute error MAE, Pearson correlation coefficient Corr, and F1 value of the validation set are used as the main evaluation indicators of the model performance, and the model with the best performance is selected as the final model.

[0096] The specific indicators of the final model are: the binary classification accuracy Acc-2 is 86.93%, the seven-class classification accuracy Acc-7 is 48.31%, the mean absolute error MAE is 0.689, the Pearson correlation coefficient Corr is 0.808, and the F1 value is 86.91. These results demonstrate the effectiveness of the model in this example for multi-modal sentiment analysis.

[0097] Example 2

[0098] As Figure 4 , this example relates to a multi-modal sentiment analysis device based on mutual information and cross-modal text enhancement, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement of the present invention.

[0099] Example 3

[0100] This example relates to a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement of the present invention.

[0101] The above are the specific embodiments of the present invention and the technical principles applied. If changes are made according to the concept of the present invention and the functions and effects produced do not exceed the spirit covered by the description and the drawings, they should still fall within the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on mutual information and cross-modal text enhancement, characterized in that, It includes the following steps: (1) After preprocessing the public dataset, extract the text, visual, and acoustic features in the dataset respectively; (2) Construct the MICTE multi-modal sentiment analysis model; (3) Through the cross-modal text enhanced attention module CTEA, integrate the text modal information into the visual and acoustic modalities; (4) Through the mutual information maximization module MIM, refine the task-related key information at the input level and the fusion level respectively; (5) Use the training set and the validation set to train and validate the MICTE model; (6) Input the test set into the finally trained MICTE model to obtain the sentiment analysis result.

2. The multimodal sentiment analysis method based on mutual information and cross-modal text enhancement according to claim 1, wherein In step (1), use the BERT language model to extract the text modal features, and use the LSTM network to extract the acoustic and visual modal features: Among them $m\in\{t, a, v\}$ represents the original input feature of modality $m$, $m\in\{t, a, v\}$ represents the feature of modality $m$ extracted from the original input $X$, m where $X$ is the original input; Adopt a temporal convolutional network to capture the sequential features of the three modalities at each moment, and normalize the obtained hidden states to a unified size: where m ∈ {t, a, v} represents the features extracted by the temporal convolutional network from modality m, and k m represents the size of the convolutional kernel used for each corresponding modality m in the temporal convolutional network.

3. A multimodal sentiment analysis method based on mutual information and cross-modal text enhancement according to claim 1, characterized in that In step (2), the specific design idea of the cross-modal text enhanced attention module CTEA is: Use the text-based multi-head attention to receive the text features and another feature as inputs, and use the output of the text-based multi-head attention as the input of the cross-modal Transformer; The mutual information maximization module MIM is divided into input-level mutual information maximization and fusion-level mutual information maximization, and the specific design idea is: Input-level mutual information maximization: Maximize the mutual information between multi-modal input pairs by calculating the exact lower bound of the mutual information. Use a non-parametric method to estimate the entropy of the mutual information, and construct Gaussian mixture models for negative and non-negative class samples respectively; Fusion-level mutual information maximization: Adopt a scoring function to measure the correlation between the normalized prediction vector and the actual vector.

4. A multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement according to claim 1, characterized in that In step (3), the specific design method of the cross-modal text enhanced attention module CTEA is: Taking the cross-modal text enhanced attention structure that maps the acoustic modality to the visual modality as an example, first use the text-based multi-head attention to receive the acoustic features and text features as inputs: wherein are respectively linear transformation weight matrices; Then, take H v as and the output H c of the obtained text-based multi-head attention together as the input to the 0th layer of the cross-modal Transformer; the specific calculation process of the ith layer is shown in the following formula: where FFN represents the feed-forward neural network, and LN represents layer normalization, denotes the multi-head attention of the i-th layer; after n layers of processing, the visual modality output F that maps the acoustic modality is obtained a→v ; similarly, the visual modality is appended to the acoustic modality using the same processing method to obtain F v→a .

5. A multimodal sentiment analysis method based on mutual information and cross-modal text enhancement according to claim 1, characterized in that, In step (4), the specific design method of the mutual information maximization module MIM is: For input-level mutual information maximization, maximize the mutual information between multi-modal input pairs by calculating the exact lower bound of the mutual information: where H(y) is the entropy of y, and use q(y|x) to approximate p(y|x); The loss function of input-level mutual information maximization is as follows: For fusion-level mutual information maximization, adopt a scoring function to measure the correlation between the normalized prediction vector and the actual vector: where and are the unit actual vector and the unit predicted vector respectively, and E H is noise contrast estimation; The loss function of fusion-level mutual information maximization is as follows:

6. A multimodal sentiment analysis method based on mutual information and cross-modal text enhancement according to claim 1, characterized in that In the step (5), the mean absolute error MAE between the sentiment label and the true label y is used as the loss function of the task: where N represents the batch size; Calculate the weighted sum of these task losses as the overall loss of the model; the specific formula is as follows: L all = L task + αL BA + βL CPC (14) where α and β are parameters that adjust the influence of maximizing mutual information; In the training stage, the model is iteratively trained using training data. The model continuously optimizes the model parameters through the overall loss until the loss function converges. After training is completed, the binary classification accuracy Acc-2, seven-class classification accuracy Acc-7, mean absolute error MAE, Pearson correlation coefficient Corr, and F1 value of the validation set are used as the main evaluation indicators of the model performance, and the model with the best performance is selected as the final model.

7. A multi-modal sentiment analysis device based on mutual information and cross-modal text enhancement, characterized in that, It includes a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, A program is stored thereon. When the program is executed by a processor, it implements the multi-modal sentiment analysis method based on mutual information and cross-modal text enhancement according to any one of claims 1-6.

Images