Multi-modal sentiment analysis method based on hierarchical feature fusion and multi-task learning

Abstract

The invention belongs to the technical field of multi-modal information processing, and particularly relates to a multi-modal sentiment analysis method based on hierarchical feature fusion and multi-task learning, which comprises the following steps: S1, establishing a multi-modal sentiment analysis model framework; s2, preparing a data set and preprocessing multi-modal audio-visual data; s3, inputting the single-mode data into a single-mode hierarchical feature extraction fusion network (UFEN), and obtaining a single-mode multilevel space-time attention fusion feature and an emotion analysis subtask result; s4, inputting different single-mode emotion representations into a multi-mode cross fusion network (MCFN) to obtain complementary features of different-mode emotion information; s5, inputting the emotion information complementary features into a feature reconstruction network and a prediction layer for emotion prediction; and S6, performing multi-task learning in combination with a plurality of sub-task results, and constructing a multi-task dynamic loss function training model. The spatial-temporal features of the multi-modal data are effectively extracted, multi-modal complementary feature interaction is achieved, and the accuracy and generalization of multi-modal sentiment analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information processing, and specifically to a multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning. Background Art

[0002] Multimodal sentiment analysis technology extracts and fuses subjective expressions with emotional colors and multiple sensory data as carriers, giving machines the ability to perceive and understand human emotions. It has been widely used in smart cockpits, public opinion analysis, Internet education, intelligent customer service, medical health, criminal interrogation, human-computer interaction and other fields. Different modalities have unique emotional characteristics. Fusion of multiple modalities can make full use of the complementary and transmission characteristics of information. However, multimodal information is also heterogeneous, and the semantic information between different modal data is not completely consistent. In this case, simple modal addition often cannot achieve the desired effect, which puts higher requirements on the feature extraction and fusion network of multimodal sentiment analysis.

[0003] Multimodal sentiment analysis mainly includes two main processes: single-modal internal sentiment feature extraction and multi-modal information interaction. Intra-modal sentiment feature extraction is to obtain a higher-dimensional single-modal sentiment representation. It focuses on the spatial or temporal information in a single modal information and is the basis for improving the complementary characteristics of multi-modal information. Multi-modal information interaction focuses on information fusion and provides more accurate and robust results for sentiment analysis tasks. Common methods for extracting intra-modal sentiment features are generally recurrent neural networks such as LSTM and GRU. These methods can effectively capture sequence features, but cannot balance the relationship between intra-modal and inter-modal interactions. Only using RNN cannot obtain the multi-scale information expected by inter-modal interaction. The common Transformer method in the multi-modal interaction process over-focuses on fusion features and ignores intra-modal sentiment features, resulting in limited sentiment analysis accuracy. Current multimodal sentiment analysis models mainly face the following problems: (1) Many single-modal feature information cannot be effectively extracted, and direct fusion can easily ignore these unique sentiment information. (2) The complementarity of multimodal sentiment features is not fully utilized, and the prediction results are more likely to be biased towards the semantic information represented by a certain modality. (3) When the sentiment tendencies of the various modalities are inconsistent, it is difficult for the model to balance the feature relationship between the modalities and between the modalities. Therefore, we propose a model that takes into account both the intra-modal feature extraction and the interaction of inter-modal sentiment information to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the above problems, the present invention provides a multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning, aiming to balance the feature relationship between intra-modality and inter-modality, better utilize unimodal spatiotemporal features and multimodal complementary features, and improve the accuracy and generalization of multimodal sentiment analysis.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] The multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning includes the following steps:

[0009] S1. Establish a multimodal sentiment analysis model framework, including data preprocessing, single-modal hierarchical feature extraction fusion network, multimodal cross-fusion network, sentiment feature reconstruction network, sentiment prediction layer and multi-task learning optimization algorithm;

[0010] S2. Prepare the data set and preprocess the multimodal audio-visual data input into the data set, including the discourse-level word vector embedding of the text modality, the extraction of the acoustic Mel-frequency cepstral coefficients of the audio modality, and the facial image alignment of the visual modality. After preprocessing, the data of each modality is represented as where i∈{t,a,v}, represents text, audio and visual modalities respectively;

[0011] S3, input the preprocessed unimodal data into the unimodal hierarchical feature extraction fusion network respectively, the process is as follows:

[0012] (1) The preprocessed text, audio, and visual unimodal data are respectively input into the bidirectional gated recurrent unit to obtain contextual temporal features;

[0013] (2) Input the contextual temporal features into a 3-layer 3×3 deep separable convolutional neural network and a 1×1 convolutional neural network to extract spatiotemporal features at different levels;

[0014] (3) Input different levels of spatiotemporal features into the self-attention layer to extract the attention weights and multiply them to obtain the corresponding level of spatiotemporal attention features;

[0015] (4) Concatenate and fuse different levels of spatiotemporal attention features to form multi-level spatiotemporal attention fusion features;

[0016] (5) Input the multi-level spatiotemporal attention fusion features into the linear prediction layer for unimodal sentiment analysis;

[0017] The contextual temporal feature extraction in S3 adopts a bidirectional gated recurrent unit to capture and enhance the time series dependency of unimodal data, and the output is the forward and backward fusion hidden representation X at the final moment. iBi-GRU , where i∈{t,a,v}, denotes text, audio and visual modalities respectively;

[0018] The multi-level spatiotemporal feature extraction in S3 adopts a 3-layer 3×3 depth-separable convolutional network and a 1×1 convolutional neural network. By fusing channel convolution and point-by-point convolution, spatiotemporal feature joint extraction is achieved and the amount of calculation is reduced. The hierarchical spatiotemporal feature process is expressed as:

[0019]

[0020] Where DwConv(·) represents a deep separable convolutional network, Conv(·) represents a common convolutional neural network, where l∈{1,2,3} represents the number of layers of the two convolutional networks, K represents the convolution kernel size of the layer, and F represents the number of convolution kernels of the layer. represents element addition;

[0021] The multi-level spatiotemporal attention feature extraction and fusion in S3 refers to: inputting spatiotemporal features of different levels into the self-attention layer, obtaining the attention weights and multiplying them, which is expressed as:

[0022]

[0023] where Attention(·) represents the self-attention process, la,l∈{1,2,3}, represents element multiplication, and the multi-level spatiotemporal attention feature fusion process is expressed as

[0024] The unimodal sentiment analysis in S3 refers to inputting the multi-level spatiotemporal attention fusion features into the linear prediction layer, which is expressed as:

[0025] Y i =LeakyReLU(W i X i f +b i )

[0026] The activation function uses LeakyReLU, W i is the trainable weight matrix of the corresponding modality, b i is the bias term, Y i are different unimodal labels predicted;

[0027] S4: Input the multi-level fusion features of different modalities into the multimodal cross-fusion network, which includes six multi-head cross-modal attention layers, namely and The cross-modal complementary feature extraction and fusion, the multi-modal cross-fusion network processing process is as follows:

[0028] (1) Input two single-modal multi-level spatiotemporal attention fusion features, assign the query to the first modal feature, assign the key and value to the other modal feature, perform multiple linear projections on the query, key and value in the attention mechanism with different initial parameters, and extract cross-modal attention features for the features projected in different subspaces. The process is expressed as:

[0029]

[0030] The activation function uses softmax, d k is the set dimension of Query and Key, n is the number of heads set in the cross-modal attention layer, and CrossAtt(·) represents the cross-modal attention process;

[0031] (2) Multiple subspace cross-modal features are concatenated and linearly projected again, expressed as:

[0032] MHCA(Q i ,K j ,V j )=Concatenet(head1,head2,…,head n )W

[0033] Where W is the trainable projection weight, MHCA(·) represents the multi-head cross-modal attention feature extraction process;

[0034] (3) The outputs of the six multi-head cross-modal attention layers are concatenated to obtain the complementary features of multimodal sentiment information as the output of the multimodal cross fusion network MCFN out , expressed as:

[0035]

[0036] S5. The complementary features of multimodal sentiment information are simultaneously input into the feature reconstruction network and the multimodal prediction layer for sentiment prediction. The output of the multimodal prediction layer is represented as Y m =ReLU(W m MFN out +b m ), the activation function uses ReLU, where W m is the weight matrix, b m is the bias term, and the prediction result of the feature reconstruction network is expressed as Y m '=softmax(W de Output de +b de ), the activation function uses softmax, where W de is the weight matrix, b de is the bias term;

[0037] S6. Combine the unimodal sentiment analysis results, feature reconstruction network prediction results and multimodal sentiment prediction results to perform multi-task learning optimization, construct a multi-task dynamic loss function and train the model to obtain the optimal model.

[0038] Furthermore, in S2, a data set is prepared and multimodal audio-visual data inputted into the data set is preprocessed. The prepared data set includes a Chinese video multimodal sentiment analysis data set CH-SIMS, an English online sharing website audio-visual data set CMU-MOSI, and an English online sharing website expanded audio-visual data set CMU-MOSEI. The data set is divided into a training set, a test set, and a validation set. For the multimodal audio-visual data preprocessing process, a BERT pre-trained model is used to perform word embedding of the text modality and map the information to the corresponding vector space. The Librosa toolkit is used to extract the acoustic Mel-frequency cepstral coefficients of the audio modality. The Openface2.0 toolkit is used to perform facial frame image alignment.

[0039] Furthermore, in S5, the complementary features of the emotional information are input into the feature reconstruction network for emotional prediction, and the process is as follows:

[0040] (1) Inputting the multimodal emotion complementary features into the encoder network, which includes a masked self-attention layer, a normalization layer, a forward propagation layer, and a linear prediction layer;

[0041] (2) The output of the masked self-attention layer is added to the complementary feature input and used as the input of the normalization layer, expressed as:

[0042] N en =LN(MCFN out +MA(MCFN out ))

[0043] Where N en is the output of the normalization layer, MA(·) represents the mask self-attention calculation process, and LN(·) represents the layer normalization process;

[0044] (3) The result of the encoder network normalization layer is input into the forward propagation layer. The output of the forward propagation layer is added to the output of the normalization layer as the output of the encoder network. en , expressed as:

[0045] Output en =N en +FF(N en )

[0046] Where FF(·) represents the forward propagation process, which includes two linear transformations and ReLU activation, expressed as:

[0047] FF(N en )=ReLU((N en W1+b1)W2+b2)

[0048] (4) The encoder output and complementary features are input to the decoder network. The decoder network includes a multi-head attention layer, a normalization layer, a forward propagation layer, and a linear prediction layer. The output of the multi-head attention layer and the complementary features are fused as the input of the normalization layer of the decoder network, which is expressed as:

[0049] N de =LN(MCFN out +MHA(MCFN out ,Output en ))

[0050] Where MHA(·) represents the multi-head attention process, in which the query is assigned to the complementary feature input, the key and value are assigned to the output of the encoder, and LN(·) represents the layer normalization process;

[0051] (5) The normalized result of the decoder network is input into the forward propagation layer, and the forward propagation layer result is added to the normalized result as the output of the decoder network. de , expressed as:

[0052] Output de =N de +FF(N de )

[0053] (6) Output the decoder network output de Input to the linear prediction layer to obtain the prediction result Y of the feature reconstruction network m ';

[0054] The masked self-attention layer in S5 includes self-attention calculation and masking process, and the self-attention calculation result is added to the mask matrix and then activated, which is expressed as:

[0055] MA(MCFN out )=softmax((QK Τ +M)V)

[0056] Where Q, K, and V represent the Query, Key, and Value in the self-attention process, respectively. M represents the padding mask matrix. When the feature matrix is ​​in a valid position, the padding mask matrix is ​​assigned a value of 0. When the feature matrix is ​​in a padding position, the padding mask matrix is ​​assigned a value of -∞.

[0057] Furthermore, step S4 inputs the multi-level spatiotemporal attention fusion features of different modalities into a multimodal cross fusion network (MCFN), which includes the following processes: (1) performing multiple linear projections on the Query, Key and Value in the attention mechanism with different initial parameters and extracting cross-modal attention features of the features projected in different subspaces respectively; (2) concatenating multiple subspace cross-modal features and linearly projecting them again; (3) concatenating the outputs of six multi-head cross-modal attention layers to obtain multimodal emotional complementary features as the output of the multimodal cross fusion network.

[0058] Furthermore, in S6, a multi-task dynamic loss function is constructed and the model is trained to obtain the optimal model. It is expressed as:

[0059]

[0060] in They represent the loss functions between the results of the unimodal sentiment analysis subtask, the results of the multimodal sentiment prediction subtask, the results of the feature reconstruction network prediction subtask and the true labels of the data. The loss function calculation process includes the Huber loss function and the focus loss optimization based on the power exponential weight, which is expressed as:

[0061]

[0062] in represents the loss of different subtasks, Represents the predicted labels of each subtask, y n represents the true label of the dataset sample, N represents the number of training samples, α is the balance factor of the subtask dynamic loss function, and γ is the adjustment factor used to change the focus strength of the loss function. It is the calculation process of Huber loss function, expressed as:

[0063]

[0064] Where δ is the Huber loss function smoothing parameter, which is used to dynamically adjust the similarity with the mean square error (MSE) and the mean absolute error (MAE), and obtain the optimal model by minimizing the multi-task loss function during the training process.

[0065] (III) Beneficial effects

[0066] Compared with the prior art, the present invention provides a multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning, which has the following beneficial effects:

[0067] Based on the idea of ​​multi-level emotional information fusion, the present invention constructs a unimodal hierarchical feature extraction fusion network, which includes deep separable convolution and self-attention mechanism, and focuses on more representative unimodal emotional information by extracting spatiotemporal features and features of different scales.

[0068] The present invention constructs a multimodal cross-fusion network and introduces a multi-head cross-modal attention layer to extract complementary sentiment features of different modal subspaces and fuse them into multimodal sentiment complementary features, thereby reducing the impact of unreasonable weight distribution of existing attention methods and improving the sentiment correlation and modal fusion effect of multimodal data.

[0069] The present invention constructs a feature reconstruction network to avoid overfitting caused by excessive extraction of sentiment information and improve the generalization of the network. It introduces multi-task learning optimization to avoid deviation in prediction results caused by inconsistent modal sentiment. It uses dynamic loss functions of multiple subtasks to improve the robustness of the overall model and achieve accurate and stable multimodal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the method model proposed by the present invention;

[0071] Figure 2 Schematic diagram of the unimodal feature extraction fusion network (UFEN) in the model of the method of the present invention;

[0072] Figure 3 Schematic diagram of the feature reconstruction network in the model of the method of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Unless otherwise specified, all technical and scientific terms used in this document have the same meanings as those generally understood by technicians in the technical field of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] Example

[0075] like Figure 1 As shown, the multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning proposed in the embodiment of the present invention has the following specific steps:

[0076] S1. The multimodal sentiment analysis model framework includes data preprocessing, unimodal hierarchical feature extraction fusion network (UFEN), multimodal cross fusion network (MCFN), sentiment feature reconstruction network, sentiment prediction layer and multi-task learning optimization algorithm. Each module is connected in sequence, and the model framework input is the audio-visual data of the dataset.

[0077] S2. Prepare the dataset and preprocess the multimodal audio-visual data of the dataset input, including discourse-level word vector embedding for text modality, acoustic Mel-frequency cepstral coefficient extraction for audio modality, and facial image alignment for visual modality.

[0078] The prepared datasets include the Chinese video multimodal sentiment analysis dataset CH-SIMS, the English online sharing website audio-visual dataset CMU-MOSI, and the English online sharing website expanded audio-visual dataset CMU-MOSEI. The datasets are divided into training set, test set, and validation set. The BERT pre-trained model is used to embed words in the text mode and map the information to the corresponding vector space. The Librosa toolkit is used to extract the acoustic Mel frequency cepstral coefficients of the audio mode. The Openface2.0 toolkit is used to align facial frame images. The pre-processed data is represented as follows: where i∈{t,a,v} represents text, audio and visual modalities respectively.

[0079] S3. Input the preprocessed unimodal data into the unimodal hierarchical feature extraction fusion network (UFEN) to obtain multi-level spatiotemporal attention fusion features of different modalities and unimodal sentiment analysis subtask results.

[0080] like Figure 2 As shown in the figure, the unimodal feature extraction fusion network (UFEN) mainly includes the following processing steps:

[0081] (1) Input the preprocessed text, audio and visual unimodal data into the bidirectional gated recurrent unit to obtain contextual temporal features The process is expressed as:

[0082] X i Bi-GRU =BiGRU(X i )

[0083] Where i∈{t,a,v} represents text, audio and visual modalities respectively, and BIGRU(·) represents the bidirectional gated recurrent unit extraction process, which includes forward and reverse information hidden state extraction and state concatenation, and is used to capture and enhance the time series dependencies of unimodal data, expressed as:

[0084]

[0085] Where Xt Represents the input information of the current time step, h t-1 is the GRU output of the previous time step. The direction of the arrow indicates the direction of data processing. GRU (·) contains the calculation of parameters such as reset gate, update gate, hidden state, etc. t It is the output after the concatenation of the forward and reverse hidden states.

[0086] (2) The contextual temporal features are input into a three-layer 3×3 deep separable convolutional neural network and a 1×1 convolutional neural network to extract spatiotemporal features at different levels. The spatiotemporal features are jointly extracted and the amount of calculation is reduced by fusing channel convolution and point-by-point convolution. The hierarchical spatiotemporal feature process is expressed as:

[0087]

[0088] Where DwConv(·) represents a deep separable convolutional network, Conv(·) represents a common convolutional neural network, where l∈{1,2,3} represents the number of layers of the two convolutional networks, K represents the convolution kernel size of the layer, and F represents the number of convolution kernels of the layer. Represents element-wise addition.

[0089] (3) Input the spatiotemporal features at different levels into the self-attention layer to extract the attention weights and multiply them to obtain the corresponding hierarchical spatiotemporal attention features, expressed as:

[0090]

[0091] where la,l∈{1,2,3}, represents element-wise multiplication, and Attention(·) represents the self-attention calculation process.

[0092] (4) The spatiotemporal attention features at different levels are concatenated and fused to form a multi-level spatiotemporal attention fusion feature, which is expressed as:

[0093]

[0094] (5) The multi-level spatiotemporal attention fusion features are input into the linear prediction layer for unimodal sentiment analysis, which is expressed as:

[0095] Y i =LeakyReLU(W i X i f +b i )

[0096] Where W i is the trainable weight matrix of the corresponding modality, b i is the bias term, Y iFor different unimodal labels predicted, LeakyReLU(·) is the activation function used to avoid overfitting of the unimodal sentiment analysis subtask results, expressed as:

[0097]

[0098] S4. Input the multi-level spatiotemporal attention fusion features of different modalities into the multimodal cross fusion network (MCFN) to obtain the emotional complementary features of different modal subspaces, and concatenate the emotional complementary features of the subspaces to obtain the multimodal emotional information complementary features.

[0099] The Multimodal Cross-Fusion Network (MCFN) consists of six multi-head cross-modal attention layers, divided into and The cross-modal complementary feature extraction mainly includes the following processing steps:

[0100] (1) Use different initial parameters to perform multiple linear projections on the Query, Key, and Value in the attention mechanism and extract cross-modal attention features from the features projected in different subspaces, expressed as:

[0101]

[0102] The activation function uses softmax, d k is the set dimension of Query and Key, n is the number of heads set in the cross-modal attention layer, and CrossAtt(·) represents the cross-modal attention calculation process. First, the input unimodal information i and j are linearly mapped, which can be expressed as:

[0103]

[0104] Where W is a trainable weight matrix, b is a bias term, and X in two-dimensional space is i,n Defined as Query, X j,n Defined as Key and Value, expressed as:

[0105]

[0106] (2) Multiple subspace cross-modal features are concatenated and linearly projected again, expressed as:

[0107] MHCA(Q i ,K j ,V j )=Concatenet(head1,head2,…,head n )W

[0108] Where W is the projection weight and MHCA(·) represents the whole process of multi-head cross-modal attention.

[0109] (3) The outputs of the six multi-head cross-modal attention layers are concatenated to obtain multimodal complementary features as the output of the multimodal cross-fusion network, expressed as:

[0110]

[0111] S5. Input the complementary features of multimodal emotional information into the feature reconstruction network and the multimodal prediction layer to perform emotional prediction to obtain subtask results. The output of the multimodal prediction layer is Y m , the output of the feature reconstruction network is Y m '.

[0112] The calculation process of the multimodal prediction layer is expressed as:

[0113] Y m =ReLU(W m MFN out +b m )

[0114] The activation function uses ReLU, W m is the weight matrix, b m is the bias term.

[0115] like Figure 3 As shown in the figure, the feature reconstruction network includes an encoder network and a decoder network. Inputting the complementary features of emotional information into the feature reconstruction network for emotion prediction includes the following process:

[0116] (1) The multimodal emotion complementary features are input into the encoder network, which consists of a masked self-attention layer, a normalization layer, a forward propagation layer, and a linear prediction layer.

[0117] Masked self-attention includes self-attention calculation and masking process. The self-attention calculation result is added to the mask matrix and activated, which is expressed as MA(MCFN out )=softmax((QK T +M)V), where Q, K, and V represent the Query, Key, and Value in the self-attention process, respectively, and M represents the padding mask matrix. When the feature matrix is ​​in a valid position, the padding mask matrix is ​​assigned a value of 0. When the feature matrix is ​​in a padding position, the padding mask matrix is ​​assigned a value of -∞. The mask matrix generation is expressed as:

[0118]

[0119] (2) The output of the masked self-attention layer is added to the complementary feature input and used as the input of the normalization layer, expressed as:

[0120] N en =LN(MCFN out +MA(MCFN out ))

[0121] Where N en is the output of the normalization layer, MA(·) represents the mask self-attention calculation process, LN(·) represents the layer normalization process, and the input of the layer normalization is z=(z1,z2,…,z n ), n represents the number of layer normalized neurons, then the specific calculation process of LN(·) is expressed as:

[0122]

[0123] Where μ represents the mean of all neurons in the layer normalization process, σ represents the standard deviation of all neurons in the layer normalization process, and y i is the output result of layer normalization, W and b represent the scaling and bias coefficients respectively, and the calculation process of μ and σ is:

[0124]

[0125] (3) The result of the encoder network normalization layer is input into the forward propagation layer. The output of the forward propagation layer is added to the output of the normalization layer as the final output of the encoder network. en , expressed as:

[0126] Output en =N en +FF(N en )

[0127] Where FF(·) represents the forward propagation process, which includes two linear transformations and ReLU activation, expressed as:

[0128] FF(N en )=ReLU((N en W1+b1)W2+b2)

[0129] (4) The encoder output and complementary features are used as the input of the multi-head attention layer in the decoder network. The output of the multi-head attention layer is fused with the complementary features as the input of the normalization layer of the decoder network, which is expressed as:

[0130] N de =LN(MCFN out +MHN(MCFN out ,Output en ))

[0131] Where MHA(·) represents the multi-head attention process, in which the query is assigned to the complementary feature input, the key and value are assigned to the output of the encoder, and LN(·) represents the layer normalization process.

[0132] (5) The normalized result of the decoder network is input into the forward propagation layer, and the forward propagation layer result is added to the normalized result as the output of the decoder network. de , expressed as:

[0133] Output de =N de +FF(N de )

[0134] (6) The decoder network output result Ooutput de Input to the linear prediction layer to obtain the prediction result Y of the feature reconstruction network m ', expressed as:

[0135] Y m '=softmax(W de Output de +b de )

[0136] The activation function uses softmax, where W de is the weight matrix, b de is the bias term.

[0137] S6. Combine the results of the unimodal sentiment analysis subtask, the feature reconstruction network prediction subtask, and the multimodal sentiment prediction subtask to perform multi-task learning optimization and construct a multi-task dynamic loss function. Minimize during training And update the model parameters to obtain the optimal model.

[0138] Constructed multi-task dynamic loss function The calculation process is expressed as:

[0139]

[0140] in They represent the loss functions between the results of the unimodal sentiment analysis subtask, the results of the multimodal sentiment prediction subtask, the results of the feature reconstruction network prediction subtask and the true labels of the data. The loss function calculation process includes the Huber loss function and the focus loss optimization based on the power exponential weight, which is expressed as:

[0141]

[0142] in represents the loss of different subtasks, Represents the predicted labels of each subtask, y n represents the true label of the dataset sample, N represents the number of training samples, α is the balancing factor of the subtask dynamic loss function, which is used to control the scaling of the subtask loss, and γ is the adjustment factor, which is used to dynamically control the focus strength of the loss function. It is the calculation process of Huber loss function, expressed as:

[0143]

[0144] Where δ is the smoothing parameter of the Huber loss function, which is used to dynamically adjust the similarity with the mean square error (MSE) and the mean absolute error (MAE). By minimizing the multi-task loss function during the training process, the risk of gradient explosion due to outliers can be reduced, and the robustness and accuracy of the multi-task model can be balanced.

[0145] The Adam optimizer is used in the overall training process of the model framework. The exponential moving average of the gradient is introduced to accelerate convergence. The exponential moving average of the squared gradient is introduced to scale the learning rates of different parameters, adaptively adjust the learning rate of each parameter, and effectively handle the sparse gradient problem. The initial first-order and second-order moment estimates are expressed as:

[0146]

[0147] Where m t is the first-order moment estimate, v t is the second-order moment estimate, β1 and β2 are the first-order and second-order momentum factors, g t Represents the current gradient, and the Adam optimizer deviation correction process is expressed as:

[0148]

[0149] For the bias-corrected first-order and second-order moment estimates, the parameter update process is expressed as:

[0150]

[0151] Where θ is the parameter value, α is the learning rate set by the model, and ∈ is the bias correction term used to avoid division by zero errors.

[0152] Experimental verification and analysis:

[0153] The data sets used for experimental verification and analysis in this example are CH-SIMS, CMU-MOSI, and CMU-MOSEI data sets:

[0154] CH-SIMS: Contains 2281 carefully edited Chinese video clips from 60 film and television videos. The video clips have rich backgrounds and a wide range of age. The audio and text parts are in Mandarin Chinese, and the length of the clips is not less than 1 second and not more than 10 seconds. The annotation tasks include 2 categories (positive, negative), 3 categories (positive, negative, neutral) and 5 categories (negative, weakly negative, neutral, weak positive, positive). The dataset has multimodal labels and independent unimodal labels.

[0155] CMU-MOSI: 2,199 video clips ranging from 2 to 5 minutes in length were collected through online sharing websites. The dataset contains video, audio, and text data, and each video clip is labeled with a [-3, 3] sentiment intensity label.

[0156] CMU-MOSEI: 23,453 annotated video clips were collected through online video sharing websites. These video clips are from 1,000 different speakers and 250 topics. Each video clip was annotated using a variety of methods, including a Likert scale of [-3, 3] and an Ekman sentiment scale of [0, 3].

[0157] The sample quantity distribution and prediction label range of different data sets for training, validation, and testing during the experimental verification process are shown in Table 1.

[0158] Table 1 Detailed statistical information of the experimental data set

[0159]

[0160] This embodiment uses two types of evaluation indicators, classification and regression, to experimentally verify the proposed method model. The evaluation indicators include sentiment binary classification accuracy (Acc-2), sentiment seven-classification accuracy (Acc-7), F1 value (F1-Score), mean absolute error (MAE) and Pearson correlation (Corr). For the CH-SIMS data set, the evaluation indicators of sentiment three-classification accuracy (Acc-3) and sentiment five-classification accuracy (Acc-5) are added. Since there are two different ways to express the binary classification accuracy of sentiment analysis, negative / non-negative and negative / positive. Therefore, the experimental results of this embodiment are distinguished using the segmentation mark - / -, where the scores on the left are negative / non-negative and the scores on the right are negative / positive.

[0161] The entire experimental process of this embodiment is based on the deep learning framework of NVIDIA 3060Ti GPU, Python3.9, Pytorch1.12.1 and Cuda11.6. The Adam optimizer is used to optimize the model parameters. The hyperparameters set for different data sets are shown in Table 2.

[0162] Table 2 Hyperparameters of the method model

[0163]

[0164] In order to verify the creativity and practicality of the present invention, a benchmark model comparison experiment and a model ablation experiment were designed. The benchmark model comparison experiment selected commonly used method models for multimodal sentiment analysis, including EF-LSTM, LF-DNN, TFN, MFN, Graph-MFN, MulT, etc. The experimental results of the comparison between the method of the present invention and other sentiment analysis benchmark model methods are shown in Tables 3 to 5.

[0165] Table 3. Comparison experiments of benchmark models on CMU-MOSI dataset

[0166]

[0167] Table 4 Benchmark model comparison experiments on the CMU-MOSEI dataset

[0168]

[0169] Table 5 Benchmark model comparison experiments on CH-SIMS dataset

[0170]

[0171] It can be seen from the experimental results of the benchmark model comparison that the method proposed in the present invention achieves the best results on the three multimodal sentiment analysis data sets compared with the benchmark model method, and has a significant improvement on some complex evaluation indicators such as Acc-5 and Acc-7, indicating that the hierarchical feature extraction fusion network, multimodal cross fusion network and multi-task learning optimization method can extract multi-layer information and weight coefficients of a single modality, make full use of contextual information, take into account global and local sentiment representation, extract complementary sentiment features between modalities while reducing redundant information interference.

[0172] In order to verify the effectiveness of each module in the method of the present invention, this embodiment conducts ablation experiments based on the CMU-MOSI and CH-SIMS datasets to verify the number of tasks in multi-task learning and the role of each network layer. The sentiment analysis results of different subtasks in the multi-task learning optimization process are shown in Table 6.

[0173] Table 6 Experimental results of multimodal sentiment analysis on different tasks. M, A, T, V represent multimodal, audio, text, and visual tasks, respectively.

[0174]

[0175] From the results, we can see that the introduction of unimodal subtasks can significantly improve the performance of the model. As the number of subtasks increases, the performance of the model tends to be better. Overall, the gain brought by the visual modality is the most obvious, because the visual modality has a lot of image information, including movements and micro-expressions in different facial regions.

[0176] The ablation experimental results of each network layer of the method of the present invention are shown in Table 7.

[0177] Table 7 Ablation experiments on CH-SIMS and CMU-MOSI datasets

[0178]

[0179] From the results, it can be seen that after removing specific network layers or modules, the evaluation indicators of the method models all show a certain degree of decline, indicating that the method models contribute to accurate multimodal sentiment analysis results.

[0180] In summary, the multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning proposed in the present invention not only has the ability to extract unimodal sentiment information, but also can explore the complementary features between different modalities. The unimodal feature extraction fusion network (UFEN) in the method can extract multi-layer spatiotemporal fusion information and weight coefficients of the unimodal, make full use of contextual information, take into account global and local sentiment representations, and reduce redundant information interference. The multimodal cross fusion network (MCFN) in the method can extract complementary sentiment information and joint representations between modalities. The method uses self-attention and multi-head attention at the same time to deeply extract the interaction of human sentiment information in different modalities and dimensions. By adding a multi-task learning method and using the prediction results of multiple subtasks to construct a dynamic focus loss function to optimize the model parameters, the model can obtain stable analysis results even when facing difficult samples with unbalanced representations, and is verified and analyzed through a series of experiments.

[0181] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning, characterized in that: The following steps are involved: S1. Establish a multimodal sentiment analysis model framework, including data preprocessing, single-modal hierarchical feature extraction fusion network, multimodal cross-fusion network, sentiment feature reconstruction network, sentiment prediction layer and multi-task learning optimization algorithm; S2. Prepare the data set and preprocess the multimodal audio-visual data input into the data set, including the discourse-level word vector embedding of the text modality, the extraction of the acoustic Mel-frequency cepstral coefficients of the audio modality, and the facial image alignment of the visual modality. After preprocessing, the data of each modality is represented as where i∈{t,a,v}, represents text, audio and visual modalities respectively; S3, input the preprocessed unimodal data into the unimodal hierarchical feature extraction fusion network respectively, the process is as follows: (1) The preprocessed text, audio, and visual unimodal data are respectively input into the bidirectional gated recurrent unit to obtain contextual temporal features; (2) Input the contextual temporal features into a 3-layer 3×3 deep separable convolutional neural network and a 1×1 convolutional neural network to extract spatiotemporal features at different levels; (3) Input different levels of spatiotemporal features into the self-attention layer to extract the attention weights and multiply them to obtain the corresponding level of spatiotemporal attention features; (4) Concatenate and fuse different levels of spatiotemporal attention features to form multi-level spatiotemporal attention fusion features; (5) Input the multi-level spatiotemporal attention fusion features into the linear prediction layer for unimodal sentiment analysis; The contextual temporal feature extraction in S3 adopts a bidirectional gated recurrent unit to capture and enhance the time series dependency of unimodal data, and the output is the forward and backward fusion hidden representation X at the final moment. i Bi-GRU , where i∈{t,a,v}, denotes text, audio and visual modalities respectively; The multi-level spatiotemporal feature extraction in S3 adopts a 3-layer 3×3 depth-separable convolutional network and a 1×1 convolutional neural network. By fusing channel convolution and point-by-point convolution, spatiotemporal feature joint extraction is achieved and the amount of calculation is reduced. The hierarchical spatiotemporal feature process is expressed as: Where DwConv(·) represents a depthwise separable convolutional network, Conv(·) represents a common convolutional neural network, l∈{1,2,3} represents the number of layers of the two convolutional networks, K represents the convolution kernel size of the layer, and F represents the number of convolution kernels of the layer. represents element addition; The multi-level spatiotemporal attention feature extraction and fusion in S3 refers to: inputting spatiotemporal features of different levels into the self-attention layer, obtaining the attention weights and multiplying them, which is expressed as: where Attention(·) represents the self-attention process, la,l∈{1,2,3}, represents element multiplication, and the multi-level spatiotemporal attention feature fusion process is expressed as The unimodal sentiment analysis in S3 refers to inputting the multi-level spatiotemporal attention fusion features into the linear prediction layer, which is expressed as: Y i =LeakyReLU(W i X i f +b i ) The activation function uses LeakyReLU, W i is the trainable weight matrix of the corresponding modality, b i is the bias term, Y i are different unimodal labels predicted; S4: Input the multi-level fusion features of different modalities into the multimodal cross-fusion network, which includes six multi-head cross-modal attention layers, namely and The cross-modal complementary feature extraction and fusion, the multi-modal cross-fusion network processing process is as follows: (1) Input two single-modal multi-level spatiotemporal attention fusion features, assign the query to the first modal feature, assign the key and value to the other modal feature, perform multiple linear projections on the query, key and value in the attention mechanism with different initial parameters, and extract cross-modal attention features for the features projected in different subspaces. The process is expressed as: The activation function uses softmax, d k is the set dimension of Query and Key, n is the number of heads set in the cross-modal attention layer, and CrossAtt(·) represents the cross-modal attention process; (2) Multiple subspace cross-modal features are concatenated and linearly projected again, expressed as: MHCA(Q i ,K j ,V j )=Concatenet(head1,head2,…,head n )W Where M is the trainable projection weight, MHCA(·) represents the multi-head cross-modal attention feature extraction process; (3) The outputs of the six multi-head cross-modal attention layers are concatenated to obtain the complementary features of multimodal sentiment information as the output of the multimodal cross fusion network MCFN out , expressed as: S5. The complementary features of multimodal sentiment information are simultaneously input into the feature reconstruction network and the multimodal prediction layer for sentiment prediction. The output of the multimodal prediction layer is represented as Y m =ReLU(W m MFN out +b m ), the activation function uses ReLU, where W m is the weight matrix, b m is the bias term, and the prediction result of the feature reconstruction network is expressed as Y m '=softmax(W de Output de +b de ), the activation function uses softmax, where W de is the weight matrix, b de is the bias term; S6. Combine the unimodal sentiment analysis results, feature reconstruction network prediction results and multimodal sentiment prediction results to perform multi-task learning optimization, construct a multi-task dynamic loss function and train the model to obtain the optimal model.

2. The multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning according to claim 1, characterized in that: In the S2, a data set is prepared and the multimodal audio-visual data inputted into the data set is preprocessed. The prepared data set includes a Chinese video multimodal sentiment analysis data set CH-SIMS, an English online sharing website audio-visual data set CMU-MOSI, and an English online sharing website expanded audio-visual data set CMU-MOSEI. The data set is divided into a training set, a test set, and a validation set. For the multimodal audio-visual data preprocessing process, a BERT pre-trained model is used to perform word embedding of the text modality and map the information to the corresponding vector space. The Librosa toolkit is used to extract the acoustic Mel-frequency cepstral coefficients of the audio modality. The Openface2.0 toolkit is used to perform facial frame image alignment.

3. The multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning according to claim 1, characterized in that: In S5, the complementary features of the emotional information are input into the feature reconstruction network for emotional prediction, and the process is as follows: (1) Inputting the multimodal emotion complementary features into the encoder network, which includes a masked self-attention layer, a normalization layer, a forward propagation layer, and a linear prediction layer; (2) The output of the masked self-attention layer is added to the complementary feature input and used as the input of the normalization layer, expressed as: N en =LN(MCFN out +MA(MCFN out )) Where N en is the output of the normalization layer, MA(·) represents the mask self-attention calculation process, and LN(·) represents the layer normalization process; (3) The result of the encoder network normalization layer is input into the forward propagation layer. The output of the forward propagation layer is added to the output of the normalization layer as the output of the encoder network. en , expressed as: Output en =N en +FF(N en ) Where FF(·) represents the forward propagation process, which includes two linear transformations and ReLU activation, expressed as: <h2 style=";text-align:left;direction:ltr">FF(N<h2 style=";text-align:left;direction:ltr"> en <h2 style=";text-align:left;direction:ltr"> )=ReLU((N<h2 style=";text-align:left;direction:ltr"> en <h2 style=";text-align:left;direction:ltr"> W1+b1)W2+b2) (4) The encoder output and complementary features are input to the decoder network. The decoder network includes a multi-head attention layer, a normalization layer, a forward propagation layer, and a linear prediction layer. The output of the multi-head attention layer and the complementary features are fused as the input of the normalization layer of the decoder network, which is expressed as: N de =LN(MCFN out +MHA(MCFN out ,Outpu ten )) Where MHA(·) represents the multi-head attention process, in which the query is assigned to the complementary feature input, the key and value are assigned to the output of the encoder, and LN(·) represents the layer normalization process; (5) Normalize the decoder network result N de Input to the forward propagation layer, the forward propagation layer result is added to the normalized result as the output of the decoder network Output de , expressed as: Output de =N de +FF(N de ) (6) Output the decoder network output de Input to the linear prediction layer to obtain the prediction result Y of the feature reconstruction network m '; The masked self-attention layer in S5 includes self-attention calculation and masking process, and the self-attention calculation result is added to the mask matrix and then activated, which is expressed as: MA(MCFN out )=softmax((QK Τ +M)V) Where Q, K, and V represent the Query, Key, and Value in the self-attention process, respectively. M represents the padding mask matrix. When the feature matrix is ​​in a valid position, the padding mask matrix is ​​assigned a value of 0. When the feature matrix is ​​in a padding position, the padding mask matrix is ​​assigned a value of -∞.

4. The multimodal sentiment analysis method based on hierarchical feature fusion and multi-task learning according to claim 1, characterized in that: In S6, a multi-task dynamic loss function is constructed and the model is trained to obtain the optimal model. The constructed multi-task dynamic loss function It is expressed as: in They represent the loss functions between the results of the unimodal sentiment analysis subtask, the results of the multimodal sentiment prediction subtask, the results of the feature reconstruction network prediction subtask and the true labels of the data. The loss function calculation process includes the Huber loss function and the focus loss optimization based on the power exponential weight, which is expressed as: in represents the loss of different subtasks, Represents the predicted labels of each subtask, y n represents the true label of the dataset sample, N represents the number of training samples, α is the balance factor of the subtask dynamic loss function, and γ is the adjustment factor used to change the focus strength of the loss function. It is the calculation process of Huber loss function, expressed as: Where δ is the Huber loss function smoothing parameter, which is used to dynamically adjust the similarity with the mean square error (MSE) and the mean absolute error (MAE), and obtain the optimal model by minimizing the multi-task loss function during the training process.

Images