Multi-modal sentiment analysis method based on non-homogeneous hierarchical fusion

Abstract

The invention discloses a multi-modal sentiment analysis method based on non-homogeneous hierarchical fusion, which comprises the following steps: S1, constructing a non-homogeneous hierarchical fusion network model which comprises a multi-modal word correction module, a Transform aggregation module, a cross attention fusion module and a function-based attention fusion module; s2, modal feature extraction is carried out through a multi-modal word correction module; s3, fusion of the visual mode and the voice mode is executed through a Transform aggregation module, and a video and voice fusion mode is obtained; s4, executing fusion of the text mode and the video and audio fusion mode through a cross attention fusion module; s5, dynamically adjusting the weight of each dimension in each modal representation through a function-based attention fusion module; and S6, predicting a final emotion regression label through regression analysis calculation. The emotion semantic information fusion method solves the problems of relatively large information density difference among voice, vision and texts, relatively high multi-mode isomerism difficulty and relatively poor effect in an emotion semantic information fusion method in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal sentiment analysis, and in particular relates to a multimodal sentiment analysis method based on heterogeneous hierarchical fusion. Background Art

[0002] Different modalities can provide complementary sentiment semantic information to each other. Existing fusion methods include those based on simple operations, graph neural networks, attention mechanisms, and routing, but they ignore the differences in information density among the three modalities, namely speech, vision, and text, that is, vision and speech have low-level semantic features, while text has high-level semantic features; the SPT model proposed by Cheng et al. and the PMR model proposed by Lv et al. both use cross-modal attention to establish multimodal sequences. They strengthen the target modality from the source modality by learning the directional pairwise attention between the source modalities. However, the multimodal information used in multimodal sentiment analysis is often heterogeneous. In multimodal sentiment analysis, how to effectively fuse the differences in information density between multimodal information, and the multimodal information used is often heterogeneous, a key issue is how to design a multimodal fusion scheme to effectively fuse heterogeneous data, so as to learn multimodal representations that include more sentiment-related information while maintaining the consistency and differentiated information of each modality.

[0003] Therefore, it is necessary to design a multimodal sentiment analysis method based on heterogeneous hierarchical fusion to solve the above problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal sentiment analysis method based on non-homogeneous hierarchical fusion. The method aims to solve the problems existing in the prior art of sentiment semantic information fusion methods, such as large differences in information density among speech, vision and text, high difficulty of multimodal heterogeneity and poor effect. The method has the characteristics of being able to learn multimodal representations including more sentiment-related information while maintaining the consistency and differentiated information of each modality.

[0005] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0006] The multimodal sentiment analysis method based on heterogeneous hierarchical fusion includes the following steps:

[0007] S1, construct a non-homogeneous hierarchical fusion network model, including a multimodal word correction module, a Transformer aggregation module, a cross-attention fusion module, and a function-based attention fusion module;

[0008] S2, modal feature extraction through the multimodal word correction module;

[0009] S3, performs the fusion of visual modality and speech modality through the Transformer aggregation module to obtain the visual-sound fusion modality;

[0010] S4, performs the fusion of text modality and audio-visual fusion modality through the cross-attention fusion module;

[0011] S5, dynamically adjusts the weight of each dimension in each modality representation through a feature-based attention fusion module;

[0012] S6, predicts the final sentiment regression label through regression analysis calculation.

[0013] Preferably, in step S2, extracting modal features by using a multimodal word correction module includes:

[0014] Text modality feature extraction in non-isomorphic fusion network models only modifies the text input token embedding through the multimodal word correction module, which makes it easy to adapt to the BERT-based multimodal feature fusion model; therefore, the BERT model is used to extract the text modality feature from the improved word embedding. Encode and use the representation of [CLS] as text representation, represented as s t , calculated as follows:

[0015] s t =BERT textual (z t );

[0016] In the formula, BERT textual Indicates the operation of the text BERT model; z t Represent word embedding;

[0017] For both visual and speech modalities, two independent BiGRU networks are used to encode visual and speech features, as shown below:

[0018] s v =BiGRU visual (x v );

[0019] s a =BiGRU audio (x a );

[0020] In the formula, s v For visual representation, s a For acoustic representation, BiGRU visual Represents the BIGRU network operation for visual features, BiGRU audio represents the BIGRU network operation for speech features; x v and x aIndicates the currently input text and sound;

[0021] The results of unimodal feature extraction have different dimensions, namely d t ≠d a ≠d v , then a fully connected layer is used to map the results of unimodal feature extraction to the same dimension d m , the calculation is as follows:

[0022] ω t = relu(W t s t +b t );

[0023] ω a = relu(W a s a +b a );

[0024] ω v = relu(W v s v +b v );

[0025] In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the speech feature extraction network, It is a trainable parameter of the visual feature extraction network; relu represents the fully connected layer operation; Indicates d m ×d t The real matrix of Indicates d m -dimensional real matrix;

[0026] For visual and speech modalities, in order to prevent too many parameters, which will affect the next step of aggregation network, a unidirectional LSTM network is used to extract visual and speech features:

[0027] X v =sLSTM visual (x v );

[0028] X a =sLSTM audio (x a );

[0029] In the formula, sLSTM visual and sLSTM audio Respectively represent the unidirectional LSTM network operations for vision and speech, X vRepresents the visual feature X extracted by sLSTM a Represents the speech features extracted by sLSTM.

[0030] Preferably, in step S3, considering that visual and speech modalities are low-level signal features, while text has high-level semantic features. In order to make up for the difference in information density between different modalities and enable them to be fused fairly, a low-level feature fusion mechanism of a Transformer aggregation module is designed; the fusion of visual modality and speech modality is performed through the Transformer aggregation module, and the visual and audio fusion modality obtained includes:

[0031] S301, since the feature dimensions of the visual and speech modalities after feature extraction by the sLSTM network are inconsistent, the extracted visual and speech features are unified in dimension through the time domain convolution network and further extracted. The calculation formula is as follows:

[0032]

[0033] In the formula, Conv1D represents the time domain convolutional network operation, k v and k a Represents the convolution kernel size of the time domain convolutional network of the visual modality and speech modality, respectively, and f v and f a Represent the sequence lengths of the visual modality and the audio modality, respectively. Represents the length of the visual modality sequence multiplied by a real matrix of d dimensions Represents the length of the speech modality sequence multiplied by a d-dimensional real matrix;

[0034] S302 uses a Transformer encoder to allow attention to flow independently within a single modality for unimodal representation learning. The calculation formula is as follows:

[0035] T v = Transformer(X v );

[0036] T a = Transformer(X a );

[0037] Where Transformer represents the Transformer encoder operation; T v Represents input X v The output result T obtained by Transformer a Represents input X a Output result obtained through Transformer;

[0038] S303, the underlying feature fusion is performed through the Transformer aggregation network. This module forces one modality to organize and condense its own information before sharing information with another modality. The core idea is to introduce The dimension of B is much smaller than that of the speech modality and the visual modality, and the calculation formula is as follows:

[0039]

[0040] In the formula, ∥ represents the concatenation of dimensions, Represents the visual module feature T v With auxiliary modal feature T f Splicing, generating new features through Transformer, Indicates that the language modal feature T a and the updated auxiliary modal features Splicing, generating new features through Transformer, T va Represents the updated language modality features and further updated auxiliary modal features Splicing to get the final fusion features.

[0041] The aggregation module is updated twice, once using visual information and once using speech information. This operation allows the model to compress information from each modality and only share necessary information, improving or maintaining multimodal fusion performance while reducing computational complexity.

[0042] Preferably, in step S4, the cross-attention fusion module uses the information of the source modality to strengthen the target modality by learning the directional pairwise attention between the source modality and the target modality; considering that the high-level semantic features of the audio-visual fusion, i.e., the fusion of vision and speech, have been obtained, the cross-attention operation is used to realize the mutual reinforcement of the text modality and the audio-visual fusion; performing the fusion of the text modality and the audio-visual fusion modality through the cross-attention fusion module includes:

[0043] S401, using the fully connected layer to make the dimensions of the text modality feature vector and the audio-visual fusion modality feature vector consistent, to facilitate attention calculation, the calculation is as follows:

[0044] X t =W t X t +b t ;

[0045] X va =W va T va +b va ;

[0046] In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the audio-visual fusion feature extraction network, X va Represents the obtained audio-visual modal feature vector X t represents the obtained text modality feature vector;

[0047] S402, calculating the relationship between the text modality feature and the audio-visual fusion modality feature to capture their correlation:

[0048] In order to minimize the heterogeneity between modalities, a learnable weight matrix is ​​set The calculation is as follows:

[0049]

[0050] Where Z represents the cross-correlation matrix, W represents the mutual weight between text and audio-visual fusion features, K represents the feature dimension of text and audio-visual fusion features, represents the transpose of the audio-visual fusion modality feature vector;

[0051] S403, the cross-correlation matrix Z represents the correlation measure between the text and the audio-visual fusion features. The magnitude of the correlation coefficient of the matrix Z is positively correlated with the correlation between the text modality features and the audio-visual fusion modality features corresponding to the subsequence. By applying Z and Z T The column-wise softmax is used to calculate the text and audio-visual fusion features. The calculation is as follows:

[0052]

[0053]

[0054] Where i and j represent the i-th row and j-th column of the cross-correlation matrix Z, respectively, T represents the softmax temperature coefficient, and A t and A va represents the cross attention weight;

[0055] S404, since the weight W is learned based on the correlation between text and audio-visual fusion features, the attention weight of each modality is guided by the other modality, thereby effectively utilizing the complementary nature of the text and audio-visual fusion modalities; after obtaining the cross-attention weight value, the attention map of the text modality and audio-visual fusion modality features is obtained using the cross-attention weight value, making it more comprehensive and discriminative; the calculation is as follows:

[0056]

[0057] In the formula, A t and Ava are the cross attention weights of text and audio-visual fusion features, respectively, and their values ​​are calculated in step S403. Represents the text modality features obtained by cross-attention weighting It represents the audio-visual fusion modality features obtained by cross-attention;

[0058] Add the reweighted attention map to the corresponding feature to get the attended feature, which is calculated as follows:

[0059]

[0060] In the formula, X att,t Represents the obtained focused text feature X att,va Indicates the obtained audio-visual fusion feature of interest; tanh represents the nonlinear operation of the hyperbolic tangent function;

[0061] X att,t and X att,va Connect and get the final cross-attention fusion representation. The calculation is as follows:

[0062]

[0063] In the formula, Represents the final cross-attention fusion representation.

[0064] Preferably, in step S5, a feature-based attention mechanism is designed to dynamically adjust the weight of each dimension in each modal representation. This method can establish correlations between each representation dimension and determine key features to enhance the multimodal fusion process. The joint representation generated by this module is ultimately used to generate sentiment predictions. Dynamically adjusting the weight of each dimension in each modal representation through a feature-based attention fusion module includes:

[0065] S501, X obtained in step S4 is converted into t , X va and Concatenate into a multimodal representation H:

[0066]

[0067] S502, calculate the weighted feature attention matrix W by the following formula att :

[0068] W att =Sigmoid(W2(ReLU(W1*H)));

[0069] In the formula, W1 and W2 are learnable parameters, Sigmoid means that W is limited by the activation function calculation. att The value in is between 0 and 1. Represents 3×d m A real matrix of ;

[0070] In this way, the importance of each feature channel can be obtained and then applied to enhance the contribution of key features to the multimodal fusion process;

[0071] S503, concatenate into a multimodal representation H and W att Multiply them together to get the final output H ′ , the calculation is as follows:

[0072] H ′ =W att *H;

[0073] Where W att represents the weighted feature attention matrix.

[0074] Preferably, in step S6, the regression analysis calculation includes:

[0075] In the regression analysis, a non-linear layer is used to predict the final sentiment regression label:

[0076] score = ReLU(W3H ′ +b3);

[0077] In the formula, ReLU means prediction through nonlinear layer; are the trainable parameters of the prediction network, Represents 1×d H The real matrix of Represents a one-dimensional real number matrix, and score represents the final regression prediction result.

[0078] The beneficial effects of the present invention are as follows:

[0079] 1. This paper designs a non-homogeneous multimodal fusion network NHHF to realize the interaction of three modalities with different information density. At the same time, this paper also takes into account that when the multimodal sentiment analysis model is deployed in a real environment, the text is usually obtained from the automatic speech recognition model, which solves the key problem of efficiently fusing heterogeneous data in multimodal sentiment analysis, thereby learning a multimodal representation containing more sentiment-related information while maintaining the consistency and differentiation information of each modality.

[0080] 2. The present invention designs a multimodal word correction module MWRM, which can use a language model to estimate the potential position of sentiment words in ASR text by capturing syntactic and semantic information;

[0081] 3. The present invention designs the Transformer aggregation module TFA. Since vision and speech have low-level signal features, while text has high-level semantic features. In order to enhance the low-level features of speech and visual modalities to high-level features. This module first organizes and compresses the information of the two modalities, and then the integrated information must be passed through the aggregation module to achieve complementary feature learning to compensate for the difference in information density. Finally, the audio-visual fusion feature is obtained through this module;

[0082] 4. The present invention designs a cross attention fusion module CAF, which realizes the mutual reinforcement of text modality and audio-visual fusion. It captures relevant complementary information by using the mutual relationship matrix to achieve accurate emotion recognition. The attention based on mutual correlation helps capture the semantic correlation between text features and audio-visual fusion features;

[0083] 5. The present invention designs a function-based attention fusion module FWAF. Regarding the FWAF module, the representations of the three modalities are initially integrated through a simple vector concatenation operation. Subsequently, a feature-based attention mechanism is designed to dynamically adjust the weight of each dimension in each modality representation. This approach is able to establish correlations between each representation dimension and identify key features to enhance the multimodal fusion process. The joint representation generated by this module is finally used to generate sentiment predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the model framework of the present invention;

[0085] Figure 2 This is a comparison chart of loss rates under different optimizers in Example 2 of the present invention. DETAILED DESCRIPTION

[0086] Embodiment 1:

[0087] like Figure 1 As shown, the multimodal sentiment analysis method based on non-homogeneous hierarchical fusion includes the following steps:

[0088] S1, construct a non-homogeneous hierarchical fusion network model, including a multimodal word correction module, a Transformer aggregation module, a cross-attention fusion module, and a function-based attention fusion module;

[0089] S2, modal feature extraction through the multimodal word correction module;

[0090] S3, performs the fusion of visual modality and speech modality through the Transformer aggregation module to obtain the visual-sound fusion modality;

[0091] S4, performs the fusion of text modality and audio-visual fusion modality through the cross-attention fusion module;

[0092] S5, dynamically adjusts the weight of each dimension in each modality representation through a feature-based attention fusion module;

[0093] S6, predicts the final sentiment regression label through regression analysis calculation.

[0094] Preferably, in step S2, extracting modal features by using a multimodal word correction module includes:

[0095] Text modality feature extraction in non-isomorphic fusion network models only modifies the text input token embedding through the multimodal word correction module, which makes it easy to adapt to the BERT-based multimodal feature fusion model; therefore, the BERT model is used to extract the text modality feature from the improved word embedding. Encode and use the representation of [CLS] as text representation, represented as s t , calculated as follows:

[0096] s t =BERT textual (z t );

[0097] In the formula, BERT textual Indicates the operation of the text BERT model; z t Represent word embedding;

[0098] For both visual and speech modalities, two independent BiGRU networks are used to encode visual and speech features, as shown below:

[0099] s v =BiGRU visual (x v );

[0100] s a =BiGRU audio (x a );

[0101] In the formula, s v For visual representation, s a For acoustic representation, BiGRU visual Represents the BIGRU network operation for visual features, BiGRU audio represents the BIGRU network operation for speech features; x v and x a Indicates the currently input text and sound;

[0102] The results of unimodal feature extraction have different dimensions, namely d t ≠d a ≠d v, then a fully connected layer is used to map the results of unimodal feature extraction to the same dimension d m , the calculation is as follows:

[0103] ω t = relu(W t s t +b t );

[0104] ω a = relu(W a s a +b a );

[0105] ω v = relu(W v s v +b v );

[0106] In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the speech feature extraction network, It is a trainable parameter of the visual feature extraction network; relu represents the fully connected layer operation; Indicates d m ×d t The real matrix of Indicates d m -dimensional real matrix;

[0107] For visual and speech modalities, in order to prevent too many parameters, which will affect the next step of aggregation network, a unidirectional LSTM network is used to extract visual and speech features:

[0108] X v =sLSTM visual (x v );

[0109] X a =sLSTM audio (x a );

[0110] In the formula, sLSTM visual and sLSTM audio Respectively represent the unidirectional LSTM network operations for vision and speech, X v Represents the visual feature X extracted by sLSTM a Represents the speech features extracted by sLSTM.

[0111] Preferably, in step S3, considering that visual and speech modalities are low-level signal features, while text has high-level semantic features. In order to make up for the difference in information density between different modalities and enable them to be fused fairly, a low-level feature fusion mechanism of a Transformer aggregation module is designed; the fusion of visual modality and speech modality is performed through the Transformer aggregation module, and the visual and audio fusion modality obtained includes:

[0112] S301, since the feature dimensions of the visual and speech modalities after feature extraction by the sLSTM network are inconsistent, the extracted visual and speech features are unified in dimension through the time domain convolution network and further extracted. The calculation formula is as follows:

[0113]

[0114] In the formula, Conv1D represents the time domain convolutional network operation, k v and k a Represents the convolution kernel size of the time domain convolutional network of the visual modality and speech modality, respectively, and f v and f a Represent the sequence lengths of the visual modality and the audio modality, respectively. Represents the length of the visual modality sequence multiplied by a real matrix of d dimensions Represents the length of the speech modality sequence multiplied by a d-dimensional real matrix;

[0115] S302 uses a Transformer encoder to allow attention to flow independently within a single modality for unimodal representation learning. The calculation formula is as follows:

[0116] T v = Transformer(X v );

[0117] T a = Transformer(X a );

[0118] Where Transformer represents the Transformer encoder operation; T v Represents input X v The output result T obtained by Transformer a Represents input X a Output result obtained through Transformer;

[0119] S303, the underlying feature fusion is performed through the Transformer aggregation network. This module forces one modality to organize and condense its own information before sharing information with another modality. The core idea is to introduce The dimension of B is much smaller than that of the speech modality and the visual modality, and the calculation formula is as follows:

[0120]

[0121] In the formula, ∥ represents the concatenation of dimensions, Represents the visual module feature T v With auxiliary modal feature T f Splicing, generating new features through Transformer, Indicates that the language modal feature T a and the updated auxiliary modal features Splicing, generating new features through Transformer, T va Represents the updated language modality features and further updated auxiliary modal features Splicing to get the final fusion features.

[0122] The aggregation module is updated twice, once using visual information and once using speech information. This operation allows the model to compress information from each modality and only share necessary information, improving or maintaining multimodal fusion performance while reducing computational complexity.

[0123] Preferably, in step S4, the cross-attention fusion module uses the information of the source modality to strengthen the target modality by learning the directional pairwise attention between the source modality and the target modality; considering that the high-level semantic features of the audio-visual fusion, i.e., the fusion of vision and speech, have been obtained, the cross-attention operation is used to realize the mutual reinforcement of the text modality and the audio-visual fusion; performing the fusion of the text modality and the audio-visual fusion modality through the cross-attention fusion module includes:

[0124] S401, using the fully connected layer to make the dimensions of the text modality feature vector and the audio-visual fusion modality feature vector consistent, to facilitate attention calculation, the calculation is as follows:

[0125] X t =W t X t +b t ;

[0126] X va =W va T va +b va ;

[0127] In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the audio-visual fusion feature extraction network, X vaRepresents the obtained audio-visual modal feature vector X t represents the obtained text modality feature vector;

[0128] S402, calculating the relationship between the text modality feature and the audio-visual fusion modality feature to capture their correlation:

[0129] In order to minimize the heterogeneity between modalities, a learnable weight matrix is ​​set The calculation is as follows:

[0130]

[0131] Where Z represents the cross-correlation matrix, W represents the mutual weight between text and audio-visual fusion features, K represents the feature dimension of text and audio-visual fusion features, represents the transpose of the audio-visual fusion modality feature vector;

[0132] S403, the cross-correlation matrix Z represents the correlation measure between the text and the audio-visual fusion features. The magnitude of the correlation coefficient of the matrix Z is positively correlated with the correlation between the text modality features and the audio-visual fusion modality features corresponding to the subsequence. By applying Z and Z T The column-wise softmax is used to calculate the text and audio-visual fusion features. The calculation is as follows:

[0133]

[0134] Where i and j represent the i-th row and j-th column of the cross-correlation matrix Z, respectively, T represents the softmax temperature coefficient, and A t and A va represents the cross attention weight;

[0135] S404, since the weight W is learned based on the correlation between text and audio-visual fusion features, the attention weight of each modality is guided by the other modality, thereby effectively utilizing the complementary nature of the text and audio-visual fusion modalities; after obtaining the cross-attention weight value, the attention map of the text modality and audio-visual fusion modality features is obtained using the cross-attention weight value, making it more comprehensive and discriminative; the calculation is as follows:

[0136]

[0137] In the formula, A t and A va are the cross attention weights of text and audio-visual fusion features, respectively, and their values ​​are calculated in step S403. Represents the text modality features obtained by cross-attention weighting It represents the audio-visual fusion modality features obtained by cross-attention;

[0138] Add the reweighted attention map to the corresponding feature to get the attended feature, which is calculated as follows:

[0139]

[0140] In the formula, X att,t Represents the obtained focused text feature X att,va Indicates the obtained audio-visual fusion feature of interest; tanh represents the nonlinear operation of the hyperbolic tangent function;

[0141] X att,t and X att,va Connect and get the final cross-attention fusion representation. The calculation is as follows:

[0142]

[0143] In the formula, Represents the final cross-attention fusion representation.

[0144] Preferably, in step S5, a feature-based attention mechanism is designed to dynamically adjust the weight of each dimension in each modal representation. This method can establish correlations between each representation dimension and determine key features to enhance the multimodal fusion process. The joint representation generated by this module is ultimately used to generate sentiment predictions. Dynamically adjusting the weight of each dimension in each modal representation through a feature-based attention fusion module includes:

[0145] S501, X obtained in step S4 is converted into t , X va and Concatenate into a multimodal representation H:

[0146]

[0147] S502, calculate the weighted feature attention matrix W by the following formula att :

[0148] W att =Sigmoid(W2(ReLU(W1*H)));

[0149] In the formula, W1 and W2 are learnable parameters, Sigmoid means that W is limited by the activation function calculation. att The value in is between 0 and 1. Represents 3×d m A real matrix of ;

[0150] In this way, the importance of each feature channel can be obtained and then applied to enhance the contribution of key features to the multimodal fusion process;

[0151] S503, concatenate into a multimodal representation H and W att Multiply them together to get the final output H ′ , the calculation is as follows:

[0152] H ′ =W att *H;

[0153] Where W att represents the weighted feature attention matrix.

[0154] Preferably, in step S6, the regression analysis calculation includes:

[0155] In the regression analysis, a non-linear layer is used to predict the final sentiment regression label:

[0156] score = ReLU(W3H ′ +b3);

[0157] In the formula, ReLU means prediction through nonlinear layer; are the trainable parameters of the prediction network, Represents 1×d H The real matrix of Represents a one-dimensional real number matrix, and score represents the final regression prediction result.

[0158] Embodiment 2:

[0159] This embodiment designs a variety of comparative experiments of existing technologies to verify the advancement of this model.

[0160] To ensure fairness, the same parameters were used in the experiments on all three datasets. When training the regression model, the L1 norm loss function was selected as the loss function used in this example, and Adam was used as the optimizer with a learning rate of 1×10 -5 The batch size is 32, the text dropout is 0.3, the language dropout is 0.1, and the visual dropout is 0.1. The entire method is implemented based on PyTorch, and all experiments are run on NVIDIA GeForce RTX 3080 GPU.

[0161] In order to verify the effectiveness of the model, this section compares the NHHF model with the following benchmark methods:

[0162] MulT: A Multimodal Transformer is proposed to achieve interactions between multimodal sequences across different time steps using cross-modal attention.

[0163] Self-MM: Learning rich unimodal representations by training one multimodal task and three unimodal subtasks.

[0164] SWRM: A sentiment word-aware multimodal word correction model is proposed to dynamically complete the damaged sentiment semantics using multimodal sentiment information.

[0165] MMIM: It uses multi-task learning to solve the problems of cross-modal information interaction and multimodal feature fusion effect by learning to maximize information.

[0166] TETA: It uses label encoding techniques to assist the Transformer network to cover all missing cases of uncertain modalities through supervised joint representation learning.

[0167] In this embodiment, the performance of the model on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets is evaluated using various metrics, including accuracy (Acc2), F1 score (F1) in binary classification, mean absolute error (MAE), and correlation (Corr).

[0168] Specifically, Acc2 and F1 scores are calculated in two ways: negative / non-negative (Non0Acc, Non0-F1) and negative / positive (Has0-Acc, Has0-F1).

[0169] Table 1 below lists the percentage of sentiment word replacement errors in the MOSI dataset using the three ASR APIs. The percentage of sentiment word replacement errors for MOSI-IBM is 17.6%, which means that approximately 17 out of 100 utterances have sentiment word replacement errors.

[0170] Table 1: Proportion of sentiment word replacement errors in three real-world datasets

[0171] data set percentage MOSI-SpeechBrain 26.5％ MOSI-IBM 17.6％ MOSI-iFlytek 10.6％

[0172] The experimental results of different models are shown in Table 2.

[0173] According to the Has0-F1 values ​​in the table, the following conclusions are drawn: Compared with the MulT model based on cross-modal attention, the NHHF model achieved an improvement of 5.04%, 4.61% and 4.88% on the MOSI-SpeechBrain, MOSI-IBM and MOSI-iFlytek datasets, respectively. The reason behind this performance improvement is that MulT ignores the difference in information density between different modalities. Compared with the MMIM model and Self-MM model using multi-task learning methods, the NHHF model achieved an improvement of 5.04%, 4.61% and 4.88% on the MOSI-SpeechBrain, MOSI-IBM and MOSI-iFlytek datasets, respectively. Although multi-task learning can share the correlation information between multiple modalities and can complement each other, it will lead to reduced model performance in some cases where the modal correlation is poor. The NHHF model improves the fusion effect of vision and speech and compensates for the heterogeneity between modalities. Compared with the TATE model, the NHHF model achieved an improvement of 1.81%, 1.68% and 1.64% on the MOSI-SpeechBrain, MOSI-IBM and MOSI-iFlytek datasets, respectively. When the probability of sentiment words being replaced in the text is low, the gap with the NHHF model is relatively low, but the TATE model takes into account the situation of modal missing, but does not pay attention to the heterogeneity between different modalities, and there are problems of insufficient fusion and information redundancy in the fusion process. Compared with the SWRM model, the NHHF model achieved an improvement of 1.81%, 1.68% and 1.64% on the MOSI-SpeechBrain, MOSI-IBM and MOSI-iFlytek datasets, respectively. This is because NHHF takes into account the difference in information density between different modalities in multimodal processing. In the process of multimodal fusion, the modality with low-level signal features is enhanced to high-level features, and then information interaction can be carried out fairly with the text with high-level semantic features.

[0174] Table 2: Comparative experimental results

[0175]

[0176]

[0177] like Figure 2As shown in the figure, the performance of the loss rate reduction after using different optimizers is shown. It can be seen from the figure that after 410 iterations, all optimizers can reduce the loss rate of the model. Among them, SGD has the worst effect, slow convergence speed, and high loss rate in the later stage of training, showing large fluctuations; Momentum has improved convergence speed compared to SGD, but it is still not as good as other advanced optimizers. RMSprop has a stable convergence speed, but the loss rate does not reach the lowest in the later stage of training. AdaGrad performs well in the early stage of training, but the convergence speed slows down due to the gradual decrease in learning rate. RAdam, NAdam and Adam perform relatively well, combining the advantages of momentum and adaptive learning rate, while improving stability. However, Adam's loss rate drops lower on average, so Adam is selected as the optimizer of the loss function.

[0178] Furthermore, the contribution of each component in the model is evaluated through ablation experiments, as shown in Table 3. The architecture contains several different variants. NHHF represents the complete model proposed in this embodiment. sLSTM (Audio), sLSTM (Visual) and BERT (Text) represent the single-modal tasks of speech, vision and text respectively. From the experimental results, it can be seen that in the single-modality, text still dominates, and the accuracy of emotion recognition is much higher than that of vision and speech; secondly, it can be seen that the importance of multi-modality is improved to a certain extent through multi-modal recognition of emotions. w / o MWRM means not using the multimodal word correction module, and no processing is done on the sentiment words that are incorrectly recognized by ASR. From the experimental results, it can be found that the performance of the model has dropped significantly, but it is better than the baseline model, which proves the effectiveness and superiority of the fusion network of NHHF; w / o TFA means the model Transformer aggregation module. From the experimental results, it can be found that it is necessary to make up for the difference in information density between modalities, and information interaction under fair conditions can achieve better results; w / o CAF and w / o FWAF respectively mean not using the cross attention module and not using the function-based attention fusion module. From the experimental results, it can be seen that both are necessary to improve the interaction ability between modalities, thereby improving the performance of the model.

[0179] Table 3: Ablation experiment results of MOSI-SpeechBrain dataset

[0180] Models Has0-Acc↑ Non0-Acc↑ MAE↓ NHHF 76.82 76.74 87.23 sLSTM(Audio) 54.69 55.23 138.28 sLSTM(Visual) 54.25 53.36 144.12 BERT(Text) 73.32 75.09 91.22 w / o MVRM 75.20 75.78 89.98 w / o CAF 75.66 77.29 89.76 w / o TFA 76.02 76.14 89.22 w / o FWAF 76.26 76.34 88.84

[0181] As can be seen from the above, this embodiment discloses a multimodal sentiment analysis architecture based on a non-homogeneous hierarchical fusion network, which is mainly composed of MWRM module, TFA module, CAF module and FWAF module. The MWRM module effectively alleviates the error of sentiment word recognition in speech recognition; at the same time, the TFA module makes up for the difference in information density between different modalities; then, the CAF module improves the fusion strategy of the correlation between text feature information and audio-visual fusion feature information; finally, the feature attention mechanism is used through the FWAF module to dynamically adjust the representation weight of each modality; the effectiveness and reliability of the model of this embodiment are proved by experimental results.

Claims

1. A multimodal sentiment analysis method based on heterogeneous hierarchical fusion, characterized in that: The following steps are involved: S1, construct a non-homogeneous hierarchical fusion network model, including a multimodal word correction module, a Transformer aggregation module, a cross-attention fusion module, and a function-based attention fusion module; S2, modal feature extraction through the multimodal word correction module; S3, performs the fusion of visual modality and speech modality through the Transformer aggregation module to obtain the visual-sound fusion modality; S4, performs the fusion of text modality and audio-visual fusion modality through the cross-attention fusion module; S5, dynamically adjusts the weight of each dimension in each modality representation through a feature-based attention fusion module; S6, predicts the final sentiment regression label through regression analysis calculation.

2. The multimodal sentiment analysis method based on heterogeneous hierarchical fusion according to claim 1 is characterized in that: In step S2, extracting modal features through the multimodal word correction module includes: Using the BERT model to improve word embedding Encode and use the representation of [CLS] as text representation, represented as s t , calculated as follows: s t =BERT textual (z t ); In the formula, BERT textual Indicates the operation of the text BERT model; z t Represent word embedding; For both visual and speech modalities, two independent BiGRU networks are used to encode visual and speech features, as shown below: s v =BiGRU visual (x v ); s a =BiGRUU audio (x a ); In the formula, s v For visual representation, s a For acoustic representation, BiGRU visual Represents the BIGRU network operation for visual features, BiGRU audio represents the BIGRU network operation for speech features; x v and x a Indicates the currently input text and sound; The results of unimodal feature extraction have different dimensions, namely d t ≠d a ≠d v , then a fully connected layer is used to map the results of unimodal feature extraction to the same dimension d m , the calculation is as follows: ω t =relu(W t st +b t ); ω a =relu(W a s a +b a ); ω v =relu(W v s v +b v ); In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the speech feature extraction network, It is a trainable parameter of the visual feature extraction network; relu represents the fully connected layer operation; Indicates d m ×d t A real matrix of Indicates d m -dimensional real matrix; A unidirectional LSTM network is used to extract visual and speech features: X v =sLSTM visual (x v ); X a =sLSTM audio (x a ); In the formula, sLSTM visual and sLSTM audio Respectively represent the unidirectional LSTM network operations for vision and speech, X v Represents the visual feature X extracted by sLSTM a Represents the speech features extracted by sLSTM.

3. The multimodal sentiment analysis method based on heterogeneous hierarchical fusion according to claim 1 is characterized in that: In step S3, the visual modality and the speech modality are fused through the Transformer aggregation module, and the obtained visual and audio fusion modality includes: S301, since the feature dimensions of the visual and speech modalities after feature extraction by the sLSTM network are inconsistent, the extracted visual and speech features are unified in dimension through the time domain convolution network and further extracted. The calculation formula is as follows: In the formula, Conv1D represents the time domain convolutional network operation, k v and k a Represents the convolution kernel size of the time domain convolutional network of the visual modality and the speech modality, respectively, and f v and f a Represent the sequence lengths of the visual modality and the audio modality, respectively. Represents the length of the visual modality sequence multiplied by a real matrix of d dimensions Represents the length of the speech modality sequence multiplied by a d-dimensional real matrix; S302 uses a Transformer encoder to allow attention to flow independently within a single modality for unimodal representation learning. The calculation formula is as follows: T v =Transformer(X v ); T a =Transformer(X a ); Where Transformer represents the Transformer encoder operation; T v Represents input X v The output result T obtained by Transformer a Represents input X a Output result obtained through Transformer; S303, the underlying feature fusion is performed through the Transformer aggregation network, and the input sequence is introduced The dimension of B is smaller than that of the speech modality and the visual modality, and the calculation formula is as follows: In the formula, || represents the concatenation of dimensions, Represents the visual module feature T v With auxiliary modal feature T f Splicing, generating new features through Transformer, Indicates that the language modal feature T a and the updated auxiliary modal features Splicing, generating new features through Transformer, T va Represents the updated language modality features and further updated auxiliary modal features Splicing to get the final fusion features.

4. The multimodal sentiment analysis method based on heterogeneous hierarchical fusion according to claim 1, characterized in that: In step S4, performing the fusion of the text modality and the audio-visual fusion modality through the cross attention fusion module includes: S401, using the fully connected layer to make the dimensions of the text modality feature vector and the audio-visual fusion modality feature vector consistent, to facilitate attention calculation, the calculation is as follows: X t =W t X t +b t ; X va =W va T va +b va ; In the formula, is the trainable parameter of the text feature extraction network, is the trainable parameter of the audio-visual fusion feature extraction network, X va Represents the obtained audio-visual modal feature vector X t represents the obtained text modality feature vector; S402, calculating the relationship between the text modality feature and the audio-visual fusion modality feature to capture their correlation: In order to minimize the heterogeneity between modalities, a learnable weight matrix is ​​set The calculation is as follows: Where Z represents the cross-correlation matrix, W represents the mutual weight between text and audio-visual fusion features, K represents the feature dimension of text and audio-visual fusion features, represents the transpose of the audio-visual fusion modality feature vector; S403, the cross-correlation matrix Z represents the correlation measure between the text and the audio-visual fusion features. The magnitude of the correlation coefficient of the matrix Z is positively correlated with the correlation between the text modality features and the audio-visual fusion modality features corresponding to the subsequence. By applying Z and Z T The column-wise softmax is used to calculate the text and audio-visual fusion features. The calculation is as follows: Where i and j represent the i-th row and j-th column of the cross-correlation matrix Z, respectively, T represents the softmax temperature coefficient, and A t and A va represents the cross attention weight; S404, after obtaining the cross attention weight value, the attention map of the text modality and the audio-visual fusion modality features is obtained using the cross attention weight value to make it more comprehensive and discriminative; the calculation is as follows: In the formula, A t and A va are the cross attention weights of text and audio-visual fusion features, respectively, and their values ​​are calculated in step S403. Represents the text modality features obtained by cross-attention weighting It represents the audio-visual fusion modality features obtained by cross-attention; Add the reweighted attention map to the corresponding feature to get the attended feature, which is calculated as follows: In the formula, X att,t Represents the obtained focused text feature X att,va Indicates the obtained audio-visual fusion feature of interest; tanh represents the nonlinear operation of the hyperbolic tangent function; X att,t and X att,va Connect and get the final cross-attention fusion representation. The calculation is as follows: In the formula, Represents the final cross-attention fusion representation.

5. The multimodal sentiment analysis method based on heterogeneous hierarchical fusion according to claim 4 is characterized in that: In step S5, the weight of each dimension in each modality representation is dynamically adjusted through the function-based attention fusion module, including: S501, X obtained in step S4 is converted into t , X va and Concatenate into a multimodal representation H: S502, calculate the weighted feature attention matrix W by the following formula att : W att =Sigmoid(W2(ReLU(W1*H))); In the formula, W1 and W2 are learnable parameters, Sigmoid means that W is limited by the activation function calculation. att The value in is between 0 and 1. Represents 3×d m A real matrix of ; S503, concatenate into a multimodal representation H and W att Multiply them together to get the final output H′, which is calculated as follows: H′=W att *H; Where W att represents the weighted feature attention matrix.

6. The multimodal sentiment analysis method based on heterogeneous hierarchical fusion according to claim 1, characterized in that: In step S6, the regression analysis calculation includes: In the regression analysis, a non-linear layer is used to predict the final sentiment regression label: score = ReLU(W3H′+b3); In the formula, ReLU means prediction through nonlinear layer; are the trainable parameters of the prediction network, Represents 1×d H A real matrix of Represents a one-dimensional real number matrix, and score represents the final regression prediction result.

Images