A multimodal sentiment analysis method and system based on multidimensional perception

Abstract

The present invention discloses a multimodal sentiment analysis method and system based on multidimensional perception, which relates to the field of sentiment analysis technology. The method comprises the following steps: inputting the multimodal sentiment data to be identified into a trained multimodal sentiment analysis model, the multimodal sentiment analysis model comprising a feature extraction network, a transmembrane interaction network, a multidimensional perception network, and a sentiment prediction network; extracting features from the multimodal sentiment data to be identified through the feature extraction network, constructing aligned text features, aligned audio features, and aligned video features; inputting the aligned text features, aligned audio features, and aligned video features into the transmembrane interaction network for feature interaction, and determining interactive fusion features; extracting features from the interactive fusion features along multiple feature dimensions based on the multidimensional perception network to generate hybrid multimodal features; and employing a sentiment prediction network to perform sentiment scoring on the hybrid multimodal features, and outputting a target sentiment score. The above scheme helps to improve the accuracy of sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on multi-dimensional perception. Background Art

[0002] In the application scenario of smart classrooms, multimodal sentiment analysis can monitor students' emotional state in classroom learning in real time based on their emotional data. The sources of emotional data are very rich, mainly including three modalities: video, audio and text. Video data mainly reflects students' visual clues such as facial expressions, eyes, and body movements. Audio data records students' auditory information such as voice intonation, vocal rhythm, and volume changes. Text data usually includes text content such as students' oral answers, discussion speeches or written assignments. Students' emotional state not only directly affects the learning atmosphere in the classroom, but also significantly affects students' understanding and absorption of knowledge. Therefore, multimodal sentiment analysis provides strong support for improving teaching effectiveness and student satisfaction.

[0003] Existing multimodal sentiment analysis methods mainly use LSTM or Transformer to further process the features of multiple modalities into one-dimensional tensors with the same feature dimension, and then directly splice the feature tensors of the three modalities. Since there is more noise and relatively less effective information in video and audio, the video and audio may be introduced into noise after splicing the same-dimensional features. It is easy for valuable sentiment information to be ignored or lost during the fusion process, resulting in low sentiment analysis accuracy. Summary of the Invention

[0004] The present invention provides a multimodal sentiment analysis method and system based on multidimensional perception, which improves the technical problem that existing multimodal sentiment analysis methods easily lose sentiment information, resulting in low sentiment analysis accuracy.

[0005] The first aspect of the present invention provides a multimodal sentiment analysis method based on multidimensional perception, comprising:

[0006] Acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network, and an emotion prediction network;

[0007] Extracting features of the multimodal emotion data to be identified through the feature extraction network to construct aligned text features, aligned audio features, and aligned video features;

[0008] Inputting the aligned text features, the aligned audio features, and the aligned video features into the transmembrane state interaction network for feature interaction to determine an interactive fusion feature;

[0009] Extracting features from the interactive fusion features along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features;

[0010] The emotion prediction network is used to perform emotion scoring on the hybrid multimodal features, and a target emotion score is output.

[0011] Optionally, the multimodal emotion data to be identified includes text emotion data, audio emotion data, and visual emotion data; and the step of extracting features from the multimodal emotion data to be identified by the feature extraction network to construct aligned text features, aligned audio features, and aligned video features includes:

[0012] Extracting initial text features of text sentiment data based on the text extraction model;

[0013] After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features;

[0014] The video extraction model extracts the initial video features of the video emotion data, and then performs temporal feature extraction based on the Bi-LSTM layer to generate intermediate video features.

[0015] The initial text features, the intermediate audio features, and the intermediate video features are respectively input into a one-dimensional temporal convolution layer for feature alignment, and aligned text features, aligned audio features, and aligned video features are correspondingly output.

[0016] Optionally, inputting the aligned text features, the aligned audio features, and the aligned video features into the transmembrane state interaction network for feature interaction to determine the interactive fusion features includes:

[0017] Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct an interactive audio and video feature;

[0018] A cross-attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive audio and video features as a key matrix and a value matrix to determine the interactive text audio and video features;

[0019] Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features;

[0020] A cross attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive video and audio features as a key matrix and a value matrix to determine the interactive text video and audio features;

[0021] The aligned text features, the interactive text audio and video features, and the interactive text video and audio features are spliced ​​to determine interactive fusion features.

[0022] Optionally, extracting the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features includes:

[0023] Extracting features from the interactive fusion features along the sequence axis using a multi-layer perceptron to determine sequence features;

[0024] A multi-layer perceptron is used to perform feature transformation on the sequence features along the modal axis to generate modal sequence features;

[0025] The modal sequence features are processed along the channel axis based on a multi-layer perceptron to output mixed multi-modal features.

[0026] Optionally, the employing the emotion prediction network to perform emotion scoring on the hybrid multimodal features and outputting a target emotion score comprises:

[0027] Flattening the hybrid multimodal feature using a flattening layer to determine a flattened feature;

[0028] Inputting the flattened features into the self-attention layer for feature enhancement to generate enhanced features;

[0029] Perform a linear transformation on the enhanced features and output a target sentiment score.

[0030] Optionally, it also includes:

[0031] Obtain training multimodal emotion dataset and true emotion score set;

[0032] Inputting the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and outputting a set of predicted sentiment scores;

[0033] Calculating a loss function value based on the predicted emotion score set and the true emotion score set;

[0034] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, and the trained multimodal sentiment analysis model is determined.

[0035] A second aspect of the present invention provides a multimodal sentiment analysis system based on multidimensional perception, comprising:

[0036] A data acquisition module is used to acquire multimodal emotion data to be identified and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and an emotion prediction network;

[0037] A feature extraction module is used to extract features of the multimodal emotion data to be identified through the feature extraction network, and construct aligned text features, aligned audio features, and aligned video features;

[0038] a feature interaction module, configured to input the aligned text features, the aligned audio features, and the aligned video features into the transmembrane state interaction network for feature interaction, and determine an interactive fusion feature;

[0039] A multi-dimensional perception module, configured to extract features of the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features;

[0040] The sentiment analysis module is used to use the sentiment prediction network to perform sentiment scoring on the hybrid multimodal features and output a target sentiment score.

[0041] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of the above items.

[0042] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the multimodal sentiment analysis method based on multidimensional perception as described in any one of the above items.

[0043] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method based on multidimensional perception as described in any one of the above items.

[0044] It can be seen from the above technical solutions that the present invention has the following advantages:

[0045] The above scheme of the present invention provides a multimodal sentiment analysis method based on multidimensional perception, including: obtaining multimodal sentiment data to be identified, inputting the multimodal sentiment data to be identified into a trained multimodal sentiment analysis model; the multimodal sentiment analysis model includes a feature extraction network, a trans-membrane interaction network, a multi-dimensional perception network and a sentiment prediction network; extracting features from the multimodal sentiment data to be identified through the feature extraction network, constructing aligned text features, aligned audio features and aligned video features; inputting the aligned text features, aligned audio features and aligned video features into the trans-membrane interaction network for feature interaction, and determining interactive fusion features; extracting features from the interactive fusion features along multiple feature dimensions based on the multi-dimensional perception network in turn, generating mixed multimodal features; using the sentiment prediction network to perform sentiment scoring on the mixed multimodal features, and outputting a target sentiment score. Based on the above scheme, the expressive power of emotional information of different modalities is improved through cross-modal interaction, and the dependency between modalities and between single modality and multimodality is enhanced along multiple feature dimensions by applying multi-dimensional perception, reducing the loss of emotional information during the fusion process, and helping to improve the accuracy of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of the steps of a multimodal sentiment analysis method based on multi-dimensional perception provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the structure of a multimodal sentiment analysis model provided in an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a transmembrane interaction network provided by an embodiment of the present invention;

[0050] Figure 4 A structural block diagram of a multimodal sentiment analysis system based on multi-dimensional perception provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention provide a multimodal sentiment analysis method and system based on multi-dimensional perception, which are used to improve the technical problem that existing multimodal sentiment analysis methods easily lose sentiment information, resulting in low sentiment analysis accuracy.

[0052] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method based on multi-dimensional perception provided by an embodiment of the present invention.

[0054] The present invention provides a multimodal sentiment analysis method based on multidimensional perception, comprising:

[0055] Step 101: Obtain multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network, and an emotion prediction network.

[0056] It should be noted that the network architecture of the multimodal sentiment analysis model is as follows Figure 2 As shown, it can be divided into feature extraction network, transmembrane interaction network, multi-dimensional perception network and emotion prediction network. Model training of the built multimodal emotion analysis model to be trained can determine the trained multimodal emotion analysis model. When the multimodal emotion data to be identified is obtained, the trained multimodal emotion data to be identified is input for emotion prediction analysis.

[0057] Step 102: extract features from the multimodal emotion data to be identified through a feature extraction network, and construct aligned text features, aligned audio features, and aligned video features.

[0058] It should be noted that the multimodal emotion data to be identified generally involves three main modalities: text modality (text, t), video modality (video, v), and audio modality (audio, a), including text emotion data, audio emotion data, and visual emotion data. The feature extraction network includes a text extraction model, an audio extraction model, a video extraction model, a Bi-LSTM layer, and a one-dimensional temporal convolution layer. Step 102 includes the following substeps:

[0059] S11. Extract initial text features of text sentiment data based on the text extraction model.

[0060] It should be noted that, in the text modality part, the text sentiment data is extracted through the text extraction model to obtain the initial text features;

[0061] In one implementation, this embodiment uses the Bert model as the text extraction model. Bert is a pre-trained language model based on the Transformer architecture. It was proposed by Google in 2018. Compared with the traditional unidirectional language model, Bert uses a bidirectional Transformer encoder, which can better understand the semantics and context in the context. Bert adopts a pre-training-fine-tuning strategy. In the pre-training stage, Bert uses a large-scale text corpus for unsupervised learning, and uses a masked language model (MLM) and next sentence prediction (Next , and the BERT model can learn text representation through tasks such as Sentence Prediction (NSP). The pre-trained BERT model can already have a good representation of the text. In the fine-tuning stage, BERT can be fine-tuned under the pre-trained parameters by performing supervised training on specific tasks, which can adapt to the needs of specific fields or tasks and can efficiently extract the sentiment features in the text. In specific implementation, the BERT model of this embodiment can be BERT-base-uncased, and the Tokenizer in the model is used to segment the text, and a start tag "[CLS]" is inserted before each sentence after segmentation, and an end tag "[SEP]" is inserted after each sentence. The sequence length of the text sentiment data is set to 50, and the part of the sentence length less than 50 is filled with a mask. The processed text sentiment data is then processed by the BERT model to output features with a tensor dimension of [50, 768], where 768 is the feature dimension of a single word in the text.

[0062] S12. After extracting the initial audio features of the audio emotion data using the audio extraction model, the features are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features.

[0063] It should be noted that in the audio modality part, an audio extraction model is first used to extract features from the audio emotion data to obtain initial audio features. In specific implementation, the audio extraction model can be Librosa. After feature extraction, each speech is converted into a tensor with a tensor dimension of [500, 5], where 500 is the audio sequence length and 5 is the audio feature dimension. The initial audio features are then input into the Bi-LSTM layer to obtain the global and local dependencies of the sequence, and the intermediate audio features are output.

[0064] S13. Extract the initial video features of the video emotion data through the video extraction model, and perform temporal feature extraction based on the Bi-LSTM layer to generate intermediate video features.

[0065] It should be noted that in the video modality part, the video extraction model is first used to extract features from the video emotion data to determine the initial video features. In one implementation, the video extraction model is OpenFace, which extracts frame information from the video at a set frequency. For each frame image, a face detection algorithm is used to detect the face in the image and detect the facial key points. The facial features are calculated from the key point positions to obtain the OpenFace feature with a tensor dimension of [375, 20], where 375 is the video sequence length and 20 is the video feature dimension. Then, the Bi-LSTM layer is also used to obtain the global and local dependencies of the sequence to generate intermediate video features.

[0066] S14: Input the initial text features, intermediate audio features, and intermediate video features into the one-dimensional temporal convolution layer for feature alignment, and output aligned text features, aligned audio features, and aligned video features accordingly.

[0067] It should be noted that after determining the initial text features, intermediate audio features, and intermediate video features, a one-dimensional temporal convolution layer (Conv1D) is used to perform a convolution operation to extract the local important features in the one-dimensional sequence features. At the same time, the features of the three modalities can be projected to the same dimension to obtain aligned text features. , align audio features and aligned video features .

[0068] Step 103: Input the aligned text features, aligned audio features, and aligned video features into a transmembrane state interaction network for feature interaction to determine interactive fusion features.

[0069] Step 103 includes the following sub-steps:

[0070] Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive audio and video features;

[0071] The aligned text feature projection is used as the query matrix, and the interactive audio and video feature projection is used as the key matrix and value matrix, and then a cross attention operation is performed to determine the interactive text audio and video features;

[0072] Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features;

[0073] The aligned text feature projection is used as the query matrix, and the interactive video and audio feature projection is used as the key matrix and value matrix, and then a cross-attention operation is performed to determine the interactive text video and audio features;

[0074] Aligned text features, interactive text audio and video features, and interactive text video and audio features are spliced ​​to determine the interactive fusion features.

[0075] The cross-modal interactive attention mechanism enhances the effectiveness of video and audio modalities, reduces noise, and enhances the visual and sound modalities. The cross-modal interactive attention mechanism enables the visual and sound modalities to learn from each other and then obtain text information separately, which strengthens the information of the visual and sound modalities and makes the fused tensor have a more emotional expression.

[0076] The attention weights between two modalities are calculated through the cross-modal interactive attention mechanism, which promotes one modality to receive information from another modality, blending the video and audio information with each other, and then blending them with the text information respectively. Integrating text information into video and audio features improves the effectiveness of video and audio features.

[0077] It should be noted that the cross-membrane interaction network includes the cross-membrane transformer and the concatenation layer. The cross-membrane transformer includes Each cross-modal layer includes a layer normalization unit (LayerNorm), a multi-head attention unit (Multi-Head Attention) and a feedforward neural network unit (FeedForward); the multi-head attention unit is represented as , is the query matrix, is the bond matrix, As a value matrix, the output of the multi-head attention unit can be obtained by the following equation:

[0078] ;

[0079] Where, is the output of the multi-head attention unit, is the number of heads of the multi-head attention unit, is the first Size, is the parameter matrix of the multi-head attention unit, For cross attention, is the softmax activation function, is the transpose of the key matrix, is the scaling factor;

[0080] Among them, the query matrix , key matrix Sum Matrix It is obtained by projecting the features of different modes, where: , , , For modal , For modal , For modal Multi-head attention input features, is the query projection matrix of the modality, For modal Multi-head attention input features, For modal The key projection matrix, For modal The query projection matrix, The length of the sequence of input features for multi-head attention, is the characteristic dimension of the bond matrix, is the characteristic dimension of the value matrix, The feature dimension of the input features for multi-head attention;

[0081] Generally speaking, the features of the text modality contain more emotion-related information than the features of the audio modality and video modality, and can better reflect the most basic and intuitive modality of the speaker's emotions. This embodiment uses a cross-modal interactive attention mechanism to interact the features of the video modality and the audio modality, and then integrates them with the text modality to enhance the effective emotional information in the modality; in specific implementation, such as Figure 3 As shown: First, align the audio features (The first cross-modal layer is represented as ), align video features (The first cross-modal layer is represented as ) is mapped to the query matrix to perform mutual mapping transformation between audio mode and video mode in the cross-modal interaction layer to obtain the potential adaptation between each other and determine the interactive audio and video features and interactive video and audio features ; Then, in order to obtain a mapping from audio modality or video modality to text modality, T layers of cross-modal layers are stacked in the transmodal transformer, and the interactive audio-video features and interactive video-audio features are respectively generated into new key-value pairs with the original representation of audio modality and visual modality, and are respectively encoded with multi-head attention fusion with text modality, catalyzing better interaction between text, audio and video modalities to transform audio modality to text modality. For example, the specific processing process can be expressed as: , , , where Aligned text features for the first cross-modal layer , The input features of the cross-modal transformer from audio modality or video modality to text modality in the first cross-modal layer, Transformer from audio or video mode to text mode The multi-head attention of the cross-modal layer, is layer normalization, Transformer from audio or video mode to text mode The intermediate features of the layer, is a feedforward network, Transformer from audio or video mode to text mode The output features of each cross-modal layer are used as the input of the next cross-modal layer; finally, the aligned text features and the interactive text audio and video features and interactive text video and audio features obtained through cross-modal interaction processing are spliced ​​to output the interactive fusion features.

[0082] Step 104: Extract the interactive fusion features along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features.

[0083] Step 104 includes the following sub-steps:

[0084] The interactive fusion features are extracted along the sequence axis by using a multi-layer perceptron to determine the sequence features;

[0085] A multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features;

[0086] The modal sequence features are processed along the channel axis based on a multi-layer perceptron and mixed multimodal features are output.

[0087] It should be noted that the interactive fusion features are fed into the multi-dimensional perception network for processing. The multi-dimensional perception network is a multimodal feature processing structure. Unlike traditional methods that exchange cross-modal information at the sequence level or channel level, the multi-dimensional perception network of this embodiment sequentially and separately performs mixing of multimodal features on multiple feature dimensions. The mixed multimodal features obtained in this way can more effectively transmit and fuse emotional information in different modalities such as text, audio, and vision, retain valuable emotional information, and effectively solve the problem of information loss and neglect in the fusion process of multimodal data.

[0088] In specific implementation, Figure 2 As shown, multiple feature dimensions include sequence axis (L axis), modal axis (M axis) and channel axis (D axis). The multi-dimensional perception network includes multiple multi-layer perceptrons (MLPs). Each multi-layer perceptron includes two fully connected layers and a nonlinear activation function. The fully connected layer can be regarded as two affine transformations.

[0089] First, the interactive fusion features are extracted along the sequence axis by the first multi-layer perceptron (MLP onaxis-L) to determine the sequence features:

[0090] ;

[0091] in, ;

[0092] Where, For interactive fusion features, is the current processing axis index, is the modal index, is the channel index, is a nonlinear activation function, For layer normalization, the output tensor of the first multilayer perceptron is the sequence feature Can be considered as a set of sequence vectors , is the fully connected layer, is the learnable weight matrix, is the input feature of the multilayer perceptron, is the learnable bias matrix, is a hyperparameter, i.e., the reduced dimension on the L axis; the formula of the fully connected layer shows that all Shared Parameters and ;

[0093] Secondly, a second multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features:

[0094] ;

[0095] Where, is the sequence index, and the output tensor of the second multilayer perceptron is the modal sequence feature can be considered as a set of modal sequence vectors , is the reduced dimension on the M axis;

[0096] Finally, a third multi-layer perceptron is used to process the modal sequence features along the channel axis and output mixed multimodal features:

[0097] ;

[0098] Where, the output tensor of the third multilayer perceptron is the mixed multimodal feature can be considered as a set of mixed multimodal vectors , is the reduced dimension on the D axis.

[0099] Step 105: Use the sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output the target sentiment score.

[0100] Step 105 includes the following sub-steps:

[0101] A flattening layer is used to flatten the mixed multimodal features and determine the flattened features;

[0102] The flattened features are input into the self-attention layer for feature enhancement to generate enhanced features;

[0103] Perform linear transformation on the enhanced features and output the target sentiment score.

[0104] It should be noted that the mixed multimodal features are passed to the sentiment prediction network to perform sentiment scoring. In the specific implementation, the sentiment prediction network includes a flattening layer (Flatten), a self-attention layer (SA-Encoder), and a linear transformation layer;

[0105] First, the mixed multimodal features are flattened into flattened features of a one-dimensional tensor through a flattening layer ;

[0106] It is then sent to the self-attention layer for self-attention enhancement. The self-attention layer is responsible for improving the emotional information expression ability of the features after multimodal fusion and capturing long-distance dependencies. It uses the self-attention mechanism in the Transformer architecture to allow the model to consider the information of all positions in the sequence at the same time when processing sequence data. Each feature vector will calculate the degree of correlation between it and all other feature vectors in the sequence to generate attention weights. These attention weights reflect the importance and mutual relationship between different features, allowing the model to automatically focus on the information that is most critical to the sentiment analysis task. In this way, based on the self-attention mechanism, the model can not only extract local features, but also capture global contextual information to capture the subtle connections between text, audio and video modalities. These connections are crucial for understanding the complexity of emotions and the inherent connections between multimodal data. Therefore, the enhanced features output by the self-attention layer are a feature representation containing rich contextual information, which provides more accurate and detailed feature input for the subsequent sentiment analysis module; the calculation process of the self-attention mechanism can be expressed as: , where For self-attention, is the self-attention query matrix, is the self-attention key matrix, is the self-attention value matrix, The dimension of the self-attention key matrix is ​​used to scale the dot product to prevent the vanishing gradient problem, and the softmax activation function ensures that each output feature vector is a weighted sum of the input features, with the weight determined by the similarity between the query and the key;

[0107] Finally, the enhanced features obtained through the self-attention mechanism are linearly transformed in the linear transformation layer to obtain the final target sentiment score.

[0108] Optionally, the training process of the multimodal sentiment analysis model includes:

[0109] Obtain training multimodal emotion dataset and true emotion score set;

[0110] Input the training multimodal sentiment dataset into the multimodal sentiment analysis model to be trained, and output a set of predicted sentiment scores;

[0111] Calculate the loss function value based on the predicted emotion score set and the true emotion score set;

[0112] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, and the trained multimodal sentiment analysis model is determined.

[0113] It should be noted that for regression tasks, in this embodiment, the mean absolute error (MAE) can be used as the loss function to calculate the loss function value:

[0114] ;

[0115] Where, is the mean absolute error loss function, is the number of samples, For the samples, To predict sentiment scores, is the true sentiment score; MAE is used to calculate the absolute error between the prediction and the true value. Compared with high-order errors, MAE has a greater impact on small errors, enabling the model to achieve better accuracy on subtle emotions. Therefore, MAE is often used as a key performance indicator for sentiment analysis.

[0116] To verify the effectiveness of our solution, we conducted experiments on the MOSI and MOSEI datasets, which are commonly used in multimodal sentiment analysis. Both datasets use [-3, 3] to annotate the target's sentiment intensity. The comparison results with existing models are shown in Table 1:

[0117] Table 1 Model comparison results

[0118]

[0119] The TFN, MFM, MISA, MULT, Self-mm, MMIM, and CMHFM data in Table 1 are derived from other published papers. The scores of our method measured on the processed datasets are shown in the table. After experimental verification, our method has achieved certain results.

[0120] In an embodiment of the present invention, the expressive power of emotional information of different modalities is improved based on cross-modal interaction, the dependency between modalities and between single modality and multimodality is enhanced along multiple feature dimensions through multi-dimensional perception, the loss of emotional information in the fusion process is reduced, and the computational burden is greatly reduced on the basis of improving the accuracy of emotional analysis.

[0121] See also Figure 4 , Figure 4 A structural block diagram of a multimodal sentiment analysis system based on multi-dimensional perception provided by an embodiment of the present invention.

[0122] The present invention provides a multimodal sentiment analysis system based on multidimensional perception, comprising:

[0123] The data acquisition module 401 is used to acquire multimodal emotion data to be identified and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network, and an emotion prediction network;

[0124] A feature extraction module 402 is configured to extract features from the multimodal emotion data to be identified through a feature extraction network, and construct aligned text features, aligned audio features, and aligned video features;

[0125] A feature interaction module 403 is used to input the aligned text features, aligned audio features, and aligned video features into a transmembrane state interaction network for feature interaction, and determine an interactive fusion feature;

[0126] A multi-dimensional perception module 404 is configured to extract interactive fusion features along multiple feature dimensions based on a multi-dimensional perception network to generate hybrid multimodal features;

[0127] The sentiment analysis module 405 is used to use the sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output a target sentiment score.

[0128] Optionally, the multimodal emotion data to be identified includes text emotion data, audio emotion data, and visual emotion data; the feature extraction module 402 is specifically used to:

[0129] Extracting initial text features of text sentiment data based on the text extraction model;

[0130] After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features;

[0131] The video extraction model extracts the initial video features of the video emotion data, and then performs temporal feature extraction based on the Bi-LSTM layer to generate intermediate video features.

[0132] The initial text features, intermediate audio features, and intermediate video features are respectively input into the one-dimensional temporal convolution layer for feature alignment, and the aligned text features, aligned audio features, and aligned video features are output accordingly.

[0133] Optionally, the feature interaction module 403 is specifically configured to:

[0134] Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive audio and video features;

[0135] The aligned text feature projection is used as the query matrix, and the interactive audio and video feature projection is used as the key matrix and value matrix, and then a cross attention operation is performed to determine the interactive text audio and video features;

[0136] Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features;

[0137] The aligned text feature projection is used as the query matrix, and the interactive video and audio feature projection is used as the key matrix and value matrix, and then a cross-attention operation is performed to determine the interactive text video and audio features;

[0138] Aligned text features, interactive text audio and video features, and interactive text video and audio features are spliced ​​to determine the interactive fusion features.

[0139] Optionally, the multi-dimensional perception module 404 is specifically configured to:

[0140] The interactive fusion features are extracted along the sequence axis by using a multi-layer perceptron to determine the sequence features;

[0141] A multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features;

[0142] The modal sequence features are processed along the channel axis based on a multi-layer perceptron and mixed multimodal features are output.

[0143] Optionally, the sentiment analysis module 405 is specifically configured to:

[0144] A flattening layer is used to flatten the mixed multimodal features and determine the flattened features;

[0145] The flattened features are input into the self-attention layer for feature enhancement to generate enhanced features;

[0146] Perform linear transformation on the enhanced features and output the target sentiment score.

[0147] Optionally, a model training module is also included for:

[0148] Obtain training multimodal emotion dataset and true emotion score set;

[0149] Input the training multimodal sentiment dataset into the multimodal sentiment analysis model to be trained, and output a set of predicted sentiment scores;

[0150] Calculate the loss function value based on the predicted emotion score set and the true emotion score set;

[0151] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, and the trained multimodal sentiment analysis model is determined.

[0152] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method based on multidimensional perception as in any of the above embodiments.

[0153] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal sentiment analysis method based on multi-dimensional perception as in any of the above embodiments are implemented.

[0154] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on multi-dimensional perception as in any of the above embodiments.

[0155] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0160] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on multidimensional perception, characterized in that: include: Acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; The multimodal sentiment analysis model includes a feature extraction network, a cross-modal interaction network, a multi-dimensional perception network and a sentiment prediction network; Extracting features of the multimodal emotion data to be identified through the feature extraction network to construct aligned text features, aligned audio features, and aligned video features; Inputting the aligned text features, the aligned audio features, and the aligned video features into the cross-modal interaction network for feature interaction, and determining interactive fusion features; Extracting features from the interactive fusion features along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features; Using the emotion prediction network to perform emotion scoring on the hybrid multimodal features, and outputting a target emotion score; Inputting the aligned text features, the aligned audio features, and the aligned video features into the cross-modal interaction network for feature interaction and determining interactive fusion features includes: Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct an interactive audio and video feature; A cross-attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive audio and video features as a key matrix and a value matrix to determine the interactive text audio and video features; Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features; A cross attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive video and audio features as a key matrix and a value matrix to determine the interactive text video and audio features; The aligned text features, the interactive text audio and video features, and the interactive text video and audio features are spliced ​​to determine interactive fusion features.

2. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The multimodal emotion data to be identified includes text emotion data, audio emotion data, and visual emotion data; and the feature extraction network is used to extract features from the multimodal emotion data to be identified, and to construct aligned text features, aligned audio features, and aligned video features, including: Extracting initial text features of text sentiment data based on the text extraction model; After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features; The video extraction model extracts the initial video features of the video emotion data, and then performs temporal feature extraction based on the Bi-LSTM layer to generate intermediate video features. The initial text features, the intermediate audio features, and the intermediate video features are respectively input into a one-dimensional temporal convolution layer for feature alignment, and aligned text features, aligned audio features, and aligned video features are correspondingly output.

3. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The extracting the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features includes: Extracting features from the interactive fusion features along the sequence axis using a multi-layer perceptron to determine sequence features; A multi-layer perceptron is used to perform feature transformation on the sequence features along the modal axis to generate modal sequence features; The modal sequence features are processed along the channel axis based on a multi-layer perceptron to output mixed multi-modal features.

4. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The sentiment prediction network is used to perform sentiment scoring on the hybrid multimodal features and output a target sentiment score, including: Flattening the hybrid multimodal feature using a flattening layer to determine a flattened feature; Inputting the flattened features into the self-attention layer for feature enhancement to generate enhanced features; Perform a linear transformation on the enhanced features and output a target sentiment score.

5. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: Also includes: Obtain training multimodal emotion dataset and true emotion score set; Inputting the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and outputting a set of predicted sentiment scores; Calculating a loss function value based on the predicted emotion score set and the true emotion score set; If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, and the trained multimodal sentiment analysis model is determined.

6. A multimodal sentiment analysis system based on multidimensional perception, characterized in that: include: A data acquisition module is used to acquire multimodal emotion data to be identified and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; The multimodal sentiment analysis model includes a feature extraction network, a cross-modal interaction network, a multi-dimensional perception network and a sentiment prediction network; A feature extraction module is used to extract features of the multimodal emotion data to be identified through the feature extraction network, and construct aligned text features, aligned audio features, and aligned video features; a feature interaction module, configured to input the aligned text features, the aligned audio features, and the aligned video features into the cross-modal interaction network for feature interaction, and determine an interactive fusion feature; A multi-dimensional perception module, configured to extract features of the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features; A sentiment analysis module, configured to perform sentiment scoring on the hybrid multimodal features using the sentiment prediction network and output a target sentiment score; The feature interaction module is specifically used to: Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct an interactive audio and video feature; A cross-attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive audio and video features as a key matrix and a value matrix to determine the interactive text audio and video features; Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features; A cross attention operation is performed after projecting the aligned text features as a query matrix and projecting the interactive video and audio features as a key matrix and a value matrix to determine the interactive text video and audio features; The aligned text features, the interactive text audio and video features, and the interactive text video and audio features are spliced ​​to determine interactive fusion features.

7. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method based on multidimensional perception are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method based on multidimensional perception are implemented.

Images