Multi-modal sentiment analysis method and system based on multi-dimensional perception

Abstract

The invention discloses a multi-modal sentiment analysis method and system based on multi-dimensional perception, and relates to the technical field of sentiment analysis, and the method comprises the steps: inputting to-be-recognized multi-modal sentiment data into a trained multi-modal sentiment analysis model, the multi-modal sentiment analysis model comprises a feature extraction network, a transmembrane interaction network, a multi-dimensional sensing network and a sentiment prediction network; performing feature extraction on the to-be-recognized multi-modal emotion data through the feature extraction network, and constructing aligned text features, aligned audio features and aligned video features; inputting the aligned text features, the aligned audio features and the aligned video features into a cross-membrane interactive network for feature interaction, and determining interactive fusion features; carrying out feature extraction on the interactive fusion feature along a plurality of feature dimensions in sequence based on a multi-dimensional sensing network to generate a mixed multi-modal feature; and performing emotion scoring on the mixed multi-modal features by using an emotion prediction network, and outputting a target emotion score. Based on the above scheme, improvement of sentiment analysis precision is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method and system based on multi-dimensional perception. Background Art

[0002] In the application scenarios of smart classrooms, multimodal sentiment analysis can monitor students' emotional state in classroom learning in real time based on their emotional data. The sources of emotional data are very rich, mainly including three modes: video, audio and text. Video data mainly reflects students' visual clues such as facial expressions, eyes, and body movements. Audio data records students' auditory information such as voice intonation, vocal rhythm, and volume changes. Text data usually includes text content such as students' oral answers, discussion speeches, or written assignments. Students' emotional state not only directly affects the learning atmosphere in the classroom, but also significantly affects students' understanding and absorption of knowledge. Therefore, multimodal sentiment analysis provides strong support for improving teaching effectiveness and student satisfaction.

[0003] In the existing multimodal sentiment analysis methods, the features of multiple modalities are further processed into one-dimensional tensors with the same feature dimension through LSTM or Transformer, and then the feature tensors of the three modalities are directly spliced. Since there is more noise and relatively less effective information in video and audio, the video and audio may be introduced with noise after the same-dimensional features are spliced, and valuable sentiment information is easily ignored or lost in the fusion process, resulting in low sentiment analysis accuracy. Summary of the invention

[0004] The present invention provides a multimodal sentiment analysis method and system based on multi-dimensional perception, which improves the technical problem that the existing multimodal sentiment analysis method is prone to lose sentiment information, resulting in low sentiment analysis accuracy.

[0005] The first aspect of the present invention provides a multimodal sentiment analysis method based on multidimensional perception, comprising:

[0006] Acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and an emotion prediction network;

[0007] Extracting features of the multimodal emotion data to be identified through the feature extraction network, and constructing aligned text features, aligned audio features, and aligned video features;

[0008] Inputting the aligned text features, the aligned audio features and the aligned video features into the transmembrane state interaction network for feature interaction, and determining an interactive fusion feature;

[0009] Based on the multi-dimensional perception network, feature extraction is performed on the interactive fusion features along multiple feature dimensions in sequence to generate hybrid multimodal features;

[0010] The emotion prediction network is used to perform emotion scoring on the mixed multimodal features, and a target emotion score is output.

[0011] Optionally, the multimodal emotion data to be identified includes text emotion data, audio emotion data and visual emotion data; and extracting features from the multimodal emotion data to be identified by the feature extraction network to construct aligned text features, aligned audio features and aligned video features includes:

[0012] Extracting initial text features of text sentiment data based on a text extraction model;

[0013] After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features;

[0014] The initial video features of the video emotion data are extracted through the video extraction model, and the temporal features are extracted based on the Bi-LSTM layer to generate intermediate video features;

[0015] The initial text features, the intermediate audio features and the intermediate video features are respectively input into a one-dimensional temporal convolution layer for feature alignment, and aligned text features, aligned audio features and aligned video features are correspondingly output.

[0016] Optionally, the step of inputting the aligned text features, the aligned audio features and the aligned video features into the transmembrane state interaction network for feature interaction to determine the interactive fusion features comprises:

[0017] Mapping the aligned audio features into a query matrix and mapping the aligned video features into a key matrix and a value matrix, and then performing a cross attention operation to construct an interactive audio and video feature;

[0018] The aligned text feature is projected as a query matrix, and the interactive audio and video feature is projected as a key matrix and a value matrix, and then a cross attention operation is performed to determine the interactive text audio and video features;

[0019] Mapping the aligned video features into a query matrix and mapping the aligned audio features into a key matrix and a value matrix, and then performing a cross attention operation to construct interactive video and audio features;

[0020] The aligned text feature projection is used as a query matrix, and the interactive video and audio feature projection is used as a key matrix and a value matrix, and then a cross attention operation is performed to determine the interactive text video and audio features;

[0021] The aligned text features, the interactive text audio and video features, and the interactive text video and audio features are spliced ​​to determine interactive fusion features.

[0022] Optionally, extracting the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features includes:

[0023] Extracting the interactive fusion features along the sequence axis through a multi-layer perceptron to determine sequence features;

[0024] A multi-layer perceptron is used to perform feature transformation on the sequence features along the modal axis to generate modal sequence features;

[0025] Based on the multi-layer perceptron, feature processing is performed on the modal sequence features along the channel axis to output mixed multi-modal features.

[0026] Optionally, the adopting the emotion prediction network to perform emotion scoring on the mixed multimodal features and outputting a target emotion score comprises:

[0027] Using a flattening layer to flatten the hybrid multimodal feature to determine a flattened feature;

[0028] Inputting the flattened features into the self-attention layer for feature enhancement to generate enhanced features;

[0029] The enhanced features are linearly transformed and the target sentiment score is output.

[0030] Optionally, it also includes:

[0031] Obtain training multimodal sentiment dataset and true sentiment score set;

[0032] Inputting the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and outputting a predicted sentiment score set;

[0033] Calculating a loss function value according to the predicted emotion score set and the true emotion score set;

[0034] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, thereby determining a trained multimodal sentiment analysis model.

[0035] A second aspect of the present invention provides a multimodal sentiment analysis system based on multidimensional perception, comprising:

[0036] A data acquisition module, used to acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and an emotion prediction network;

[0037] A feature extraction module, used to extract features of the multimodal emotion data to be identified through the feature extraction network, and construct aligned text features, aligned audio features and aligned video features;

[0038] A feature interaction module, used for inputting the aligned text features, the aligned audio features and the aligned video features into the transmembrane state interaction network for feature interaction, and determining an interactive fusion feature;

[0039] A multi-dimensional perception module, used to extract features of the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multi-modal features;

[0040] The sentiment analysis module is used to use the sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output a target sentiment score.

[0041] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of the above items.

[0042] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the multimodal sentiment analysis method based on multidimensional perception as described in any one of the above items.

[0043] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method based on multi-dimensional perception as described in any one of the above.

[0044] It can be seen from the above technical solutions that the present invention has the following advantages:

[0045] The above scheme of the present invention provides a multimodal sentiment analysis method based on multidimensional perception, including: obtaining multimodal sentiment data to be identified, inputting the multimodal sentiment data to be identified into a trained multimodal sentiment analysis model; the multimodal sentiment analysis model includes a feature extraction network, a trans-membrane interaction network, a multi-dimensional perception network and a sentiment prediction network; extracting features of the multimodal sentiment data to be identified through the feature extraction network, constructing aligned text features, aligned audio features and aligned video features; inputting the aligned text features, aligned audio features and aligned video features into the trans-membrane interaction network for feature interaction, and determining interactive fusion features; extracting features of the interactive fusion features along multiple feature dimensions in turn based on the multi-dimensional perception network to generate mixed multimodal features; using the sentiment prediction network to perform sentiment scoring on the mixed multimodal features, and outputting the target sentiment score. Based on the above scheme, the expressive power of emotional information of different modalities is improved through cross-modal interaction, and the dependency between modalities and between single modalities and multimodalities is enhanced along multiple feature dimensions by applying multi-dimensional perception, reducing the loss of emotional information during the fusion process, and helping to improve the accuracy of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flowchart of a multi-modal sentiment analysis method based on multi-dimensional perception provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the structure of a multimodal sentiment analysis model provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a transmembrane interaction network provided by an embodiment of the present invention;

[0050] Figure 4 A structural block diagram of a multimodal sentiment analysis system based on multi-dimensional perception provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention provide a multimodal sentiment analysis method and system based on multi-dimensional perception, which are used to improve the technical problem that the existing multimodal sentiment analysis method is prone to lose sentiment information, resulting in low sentiment analysis accuracy.

[0052] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method based on multi-dimensional perception is provided in an embodiment of the present invention.

[0054] The present invention provides a multi-modal sentiment analysis method based on multi-dimensional perception, comprising:

[0055] Step 101, obtain multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a trans-membrane interaction network, a multi-dimensional perception network and an emotion prediction network.

[0056] It should be noted that the network architecture of the multimodal sentiment analysis model is as follows Figure 2 As shown, it can be divided into feature extraction network, transmembrane interaction network, multi-dimensional perception network and emotion prediction network. Model training is performed on the built multimodal emotion analysis model to be trained to determine the trained multimodal emotion analysis model. When the multimodal emotion data to be identified is obtained, the trained multimodal emotion data to be identified is input for emotion prediction analysis.

[0057] Step 102: extract features from the multimodal emotion data to be identified through a feature extraction network, and construct aligned text features, aligned audio features, and aligned video features.

[0058] It should be noted that the multimodal emotion data to be identified usually involves three main modes: text mode (text, t), video mode (video, v) and audio mode (audio, a), including text emotion data, audio emotion data and visual emotion data. The feature extraction network includes a text extraction model, an audio extraction model, a video extraction model, a Bi-LSTM layer and a one-dimensional time convolution layer. Step 102 includes the following sub-steps:

[0059] S11. Extract initial text features of text sentiment data based on the text extraction model.

[0060] It should be noted that, in the text modality part, the text sentiment data is subjected to feature extraction through the text extraction model to obtain the initial text features;

[0061] In one implementation, this embodiment uses the Bert model as the text extraction model. Bert is a pre-trained language model based on the Transformer architecture. It was proposed by Google in 2018. Compared with the traditional unidirectional language model, Bert uses a bidirectional Transformer encoder, which can better understand the semantics and context in the context. Among them, Bert adopts a pre-training-fine-tuning strategy. In the pre-training stage, Bert uses a large-scale text corpus for unsupervised learning, and uses a masked language model (MLM) and a next sentence prediction (Next The pre-trained BERT model can already have a good representation of the text. In the fine-tuning stage, Bert can be fine-tuned under the pre-trained parameters by performing supervised training on specific tasks, which can adapt to the needs of specific fields or tasks and can efficiently extract the sentiment features in the text. In specific implementation, the Bert model of this embodiment can be Bert-base-uncased, and the Tokenizer in the model is used to segment the text, and a start tag "[CLS]" is inserted before each sentence after segmentation, and an end tag "[SEP]" is inserted after each sentence. The sequence length of the text sentiment data is set to 50, and the part of the sentence length less than 50 is filled with a mask. Then, the processed text sentiment data is processed by the BERT model to output features with a tensor dimension of [50, 768], where 768 is the feature dimension of a single word in the text.

[0062] S12. After extracting the initial audio features of the audio emotion data using the audio extraction model, the features are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features.

[0063] It should be noted that, in the audio modality part, the audio extraction model is first used to extract features from the audio emotion data to obtain initial audio features. In specific implementation, the audio extraction model can be Librosa. After feature extraction, each speech is converted into a tensor with a tensor dimension of [500, 5], where 500 is the length of the audio sequence and 5 is the audio feature dimension. The initial audio features are then input into the Bi-LSTM layer to obtain the global and local dependencies of the sequence, and the intermediate audio features are output.

[0064] S13. Extract the initial video features of the video emotion data through the video extraction model, and perform temporal feature extraction based on the Bi-LSTM layer to generate intermediate video features.

[0065] It should be noted that, in the video modality part, the video extraction model is first used to extract features from the video emotion data to determine the initial video features. In one implementation, the video extraction model is OpenFace, which extracts frame information from the video at a set frequency. For each frame, a face detection algorithm is used to detect the face in the image, detect the face key points, and calculate the facial features from the key point positions to obtain the OpenFace feature with a tensor dimension of [375, 20], where 375 is the video sequence length and 20 is the video feature dimension. Then, the Bi-LSTM layer is used to obtain the global and local dependencies of the sequence to generate intermediate video features.

[0066] S14, inputting the initial text features, the intermediate audio features and the intermediate video features into the one-dimensional temporal convolution layer respectively for feature alignment, and outputting aligned text features, aligned audio features and aligned video features accordingly.

[0067] It should be noted that after determining the initial text features, intermediate audio features, and intermediate video features, a one-dimensional temporal convolutional layer (Conv1D) is used to perform a convolution operation to extract the local important features in the one-dimensional sequence features. At the same time, the features of the three modalities can be projected to the same dimension to obtain aligned text features. , align audio features and align video features .

[0068] Step 103: input the aligned text features, aligned audio features and aligned video features into a transmembrane state interaction network for feature interaction to determine interactive fusion features.

[0069] Step 103 includes the following sub-steps:

[0070] Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive audio and video features;

[0071] After the aligned text feature projection is used as the query matrix and the interactive audio and video feature projection is used as the key matrix and the value matrix, a cross attention operation is performed to determine the interactive text audio and video features;

[0072] Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features;

[0073] The aligned text feature projection is used as the query matrix, and the interactive video and audio feature projection is used as the key matrix and the value matrix, and then a cross attention operation is performed to determine the interactive text video and audio features;

[0074] Aligned text features, interactive text audio and video features, and interactive text video and audio features are spliced ​​to determine the interactive fusion features.

[0075] The cross-modal interactive attention mechanism is used to enhance the effectiveness of video and audio modalities, reduce noise, and enhance the text information after the visual and sound modalities learn from each other. The information of the visual and sound modalities is strengthened and the fused tensor has a more emotional expression.

[0076] The attention weights between two modalities are calculated through the cross-modal interactive attention mechanism, which promotes one modality to receive information from another modality, blending the video and audio information with each other, and then blending them with the text information respectively. Integrating text information into video and audio features improves the effectiveness of video and audio features.

[0077] It should be noted that the cross-membrane interaction network includes the cross-membrane transformer and the concatenation layer. Each cross-modal layer includes a layer normalization unit (LayerNorm), a multi-head attention unit (Multi-Head Attention) and a feedforward neural network unit (FeedForward); the multi-head attention unit is represented as , is the query matrix, is the key matrix, As a value matrix, the output of the multi-head attention unit can be obtained by the following equation:

[0078] ;

[0079] In the formula, is the output of the multi-head attention unit, is the number of heads of the multi-head attention unit, is the first Head, is the parameter matrix of the multi-head attention unit, For cross attention, is the softmax activation function, is the transpose of the key matrix, is the scaling factor;

[0080] Among them, the query matrix , key matrix Sum Matrix It is obtained by projecting the features of different modes, where: , , , For modal , For modal , For modal The multi-head attention input features, is the query projection matrix of the modality, For modal The multi-head attention input features, For modal The key projection matrix, For modal The query projection matrix, The length of the sequence of input features for multi-head attention, is the characteristic dimension of the key matrix, is the characteristic dimension of the value matrix, The feature dimension of the input features for multi-head attention;

[0081] Generally speaking, the features of the text modality contain more emotion-related information than the features of the audio modality and the video modality, and can better reflect the most basic and intuitive modality of the speaker's emotions. This embodiment uses a cross-modal interactive attention mechanism to interact the features of the video modality and the audio modality, and then blends them with the text modality to enhance the effective emotional information in the modality; in specific implementation, such as Figure 3 As shown: First, align the audio features (The first cross-modal layer is represented as ), align video features (The first cross-modal layer is represented as ) is mapped to the query matrix to perform mutual mapping transformation between audio mode and video mode in the cross-modal interaction layer to obtain the potential adaptation between each other and determine the interactive audio and video features and interactive video and audio features ; Then, in order to obtain a mapping from the audio modality or the video modality to the text modality, T layers of cross-modal layers are stacked in the Transformer, and the interactive audio-video features and the interactive video-audio features are respectively generated into new key-value pairs with the original representations of the audio modality and the visual modality, and are respectively encoded by multi-head attention fusion with the text modality, catalyzing a better interaction between the text, audio and video modalities to transform the audio modality to the text modality. For example, the specific processing process can be expressed as: , , , where Aligned text features for the first cross-modal layer , The input features of the cross-modal transformer from audio modality or video modality to text modality in the first cross-modal layer, Transformer from audio or video mode to text mode The multi-head attention of the cross-modal layer, is layer normalization, Transformer from audio or video mode to text mode The intermediate features of the layer, is a feed-forward network, Transformer from audio or video mode to text mode The output features of each cross-modal layer are used as the input of the next cross-modal layer; finally, the aligned text features and the interactive text audio-video features and interactive text video-audio features obtained through cross-modal interaction processing are concatenated to output the interactive fusion features.

[0082] Step 104: extract the interactive fusion features along multiple feature dimensions in sequence based on the multi-dimensional perception network to generate hybrid multimodal features.

[0083] Step 104 includes the following sub-steps:

[0084] The interactive fusion features are extracted along the sequence axis by a multi-layer perceptron to determine the sequence features;

[0085] A multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features;

[0086] Based on the multi-layer perceptron, the modal sequence features are processed along the channel axis and mixed multi-modal features are output.

[0087] It should be noted that the interactive fusion features are fed to the multi-dimensional perception network for processing. The multi-dimensional perception network is a multi-modal feature processing structure. Different from the traditional method of exchanging cross-modal information at the sequence level or channel level, the multi-dimensional perception network of this embodiment sequentially and separately performs mixing of multi-modal features on each dimension on multiple feature dimensions. The mixed multi-modal features obtained in this way can more effectively transmit and fuse the emotional information in different modes such as text, audio, and vision, retain valuable emotional information, and effectively solve the problem of information loss and neglect in the fusion process of multi-modal data.

[0088] In specific implementation, Figure 2 As shown, multiple feature dimensions include sequence axis (L axis), modal axis (M axis) and channel axis (D axis), and the multi-dimensional perception network includes multiple multi-layer perceptrons (MLPs). Each multi-layer perceptron includes two fully connected layers and a nonlinear activation function. The fully connected layers can be regarded as two affine transformations.

[0089] First, the interactive fusion features are extracted along the sequence axis through the first multi-layer perceptron (MLP onaxis-L) to determine the sequence features:

[0090] ;

[0091] in, ;

[0092] In the formula, For interactive fusion features, is the current processing axis index, is the modal index, is the channel index, is a nonlinear activation function, is layer normalization, the output tensor of the first multi-layer perceptron is the sequence feature can be considered as a set of sequence vectors , is the fully connected layer, is the learnable weight matrix, is the input feature of the multilayer perceptron, is the learnable bias matrix, is a hyperparameter, i.e., the reduced dimension on the L axis; the formula of the fully connected layer shows that all Shared Parameters and ;

[0093] Secondly, a second multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features:

[0094] ;

[0095] In the formula, is the sequence index, and the output tensor of the second multilayer perceptron is the modal sequence feature can be considered as a set of modal sequence vectors , is the reduced dimension on the M axis;

[0096] Finally, a third multi-layer perceptron is used to process the modal sequence features along the channel axis and output mixed multimodal features:

[0097] ;

[0098] In the formula, the output tensor of the third multi-layer perceptron is the mixed multimodal feature can be considered as a set of mixed multimodal vectors , is the reduced dimension on the D axis.

[0099] Step 105: Use the sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output the target sentiment score.

[0100] Step 105 includes the following sub-steps:

[0101] The mixed multimodal features are flattened using a flattening layer to determine the flattened features;

[0102] The flattened features are input into the self-attention layer for feature enhancement to generate enhanced features;

[0103] Perform a linear transformation on the enhanced features and output the target sentiment score.

[0104] It should be noted that the mixed multimodal features are passed to the sentiment prediction network to perform sentiment scoring. In the specific implementation, the sentiment prediction network includes a flattening layer (Flatten), a self-attention layer (SA-Encoder), and a linear transformation layer;

[0105] First, the mixed multimodal features are flattened into flattened features of a one-dimensional tensor through a flattening layer ;

[0106] Then it is sent to the self-attention layer for self-attention enhancement. The self-attention layer is responsible for improving the emotional information expression ability of the features after multimodal fusion and capturing long-distance dependencies. It uses the self-attention mechanism in the Transformer architecture to allow the model to consider the information of all positions in the sequence at the same time when processing sequence data. Each feature vector will calculate the degree of correlation between it and all other feature vectors in the sequence to generate attention weights. These attention weights reflect the importance and relationship between different features, so that the model can automatically focus on the most critical information for the sentiment analysis task. In this way, based on the self-attention mechanism, the model can not only extract local features, but also capture global context information to capture the subtle connections between text, audio and video modalities. These connections are crucial to understanding the complexity of emotions and the inherent connections of multimodal data. Therefore, the enhanced features output by the self-attention layer are a feature representation containing rich context information, which provides more accurate and detailed feature input for the subsequent sentiment analysis module; the calculation process of the self-attention mechanism can be expressed as: , where For self-attention, is the self-attention query matrix, is the self-attention key matrix, is the self-attention value matrix, is the dimension of the self-attention key matrix for scaling dot products to prevent the vanishing gradient problem, and the softmax activation function ensures that each output feature vector is a weighted sum of the input features, with the weight determined by the similarity between the query and the key;

[0107] Finally, the enhanced features obtained through the self-attention mechanism are transformed once in the linear transformation layer to obtain the final target sentiment score.

[0108] Optionally, the training process of the multimodal sentiment analysis model includes:

[0109] Obtain training multimodal sentiment dataset and true sentiment score set;

[0110] Input the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and output a set of predicted sentiment scores;

[0111] Calculate the loss function value based on the predicted sentiment score set and the true sentiment score set;

[0112] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, thereby determining the trained multimodal sentiment analysis model.

[0113] It should be noted that for the regression task, in this embodiment, the mean absolute error (MAE) can be used as the loss function to calculate the loss function value:

[0114] ;

[0115] In the formula, is the mean absolute error loss function, is the number of samples, For the samples, To predict the sentiment score, is the true sentiment score; MAE is used to calculate the absolute error between the prediction and the true value. Compared with high-order errors, MAE has a greater impact on small errors, enabling the model to achieve better accuracy on subtle emotions. Therefore, MAE is often used as a key performance indicator for sentiment analysis.

[0116] To verify the effectiveness of the scheme, we conducted experiments on the MOSI and MOSEI datasets commonly used in multimodal sentiment analysis. Both datasets use [-3, 3] to mark the target's sentiment intensity. The comparison results with the existing models are shown in Table 1:

[0117] Table 1 Model comparison results

[0118]

[0119] The TFN, MFM, MISA, MULT, Self-mm, MMIM, and CMHFM data in Table 1 are derived from other published papers. The scores of our method on the processed data sets are shown in the table. After experimental verification, our method has achieved certain results.

[0120] In the embodiments of the present invention, the expressive power of emotional information of different modalities is improved based on cross-modal interaction, the dependency between modalities and between single modality and multi-modality is enhanced along multiple feature dimensions through multi-dimensional perception, the loss of emotional information in the fusion process is reduced, and the computational burden is greatly reduced on the basis of improving the accuracy of emotional analysis.

[0121] See also Figure 4 , Figure 4 A structural block diagram of a multimodal sentiment analysis system based on multi-dimensional perception provided in an embodiment of the present invention.

[0122] The present invention provides a multi-modal sentiment analysis system based on multi-dimensional perception, comprising:

[0123] The data acquisition module 401 is used to acquire the multimodal emotion data to be identified, and input the multimodal emotion data to be identified into the trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and an emotion prediction network;

[0124] A feature extraction module 402 is used to extract features from the multimodal emotion data to be identified through a feature extraction network, and construct an aligned text feature, an aligned audio feature, and an aligned video feature;

[0125] A feature interaction module 403 is used to input the aligned text features, aligned audio features and aligned video features into the transmembrane state interaction network for feature interaction and determine the interactive fusion features;

[0126] A multi-dimensional perception module 404 is used to extract interactive fusion features along multiple feature dimensions in sequence based on a multi-dimensional perception network to generate hybrid multi-modal features;

[0127] The sentiment analysis module 405 is used to use a sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output a target sentiment score.

[0128] Optionally, the multimodal emotion data to be identified includes text emotion data, audio emotion data and visual emotion data; the feature extraction module 402 is specifically used for:

[0129] Extracting initial text features of text sentiment data based on a text extraction model;

[0130] After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features;

[0131] The initial video features of the video emotion data are extracted through the video extraction model, and the temporal features are extracted based on the Bi-LSTM layer to generate intermediate video features;

[0132] The initial text features, intermediate audio features, and intermediate video features are respectively input into the one-dimensional temporal convolution layer for feature alignment, and the aligned text features, aligned audio features, and aligned video features are output accordingly.

[0133] Optionally, the feature interaction module 403 is specifically used for:

[0134] Mapping the aligned audio features into a query matrix and the aligned video features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive audio and video features;

[0135] After the aligned text feature projection is used as the query matrix and the interactive audio and video feature projection is used as the key matrix and the value matrix, a cross attention operation is performed to determine the interactive text audio and video features;

[0136] Mapping the aligned video features into a query matrix and the aligned audio features into a key matrix and a value matrix, and then performing a cross-attention operation to construct interactive video and audio features;

[0137] The aligned text feature projection is used as the query matrix, and the interactive video and audio feature projection is used as the key matrix and the value matrix, and then a cross attention operation is performed to determine the interactive text video and audio features;

[0138] Aligned text features, interactive text audio and video features, and interactive text video and audio features are spliced ​​to determine the interactive fusion features.

[0139] Optionally, the multi-dimensional perception module 404 is specifically used for:

[0140] The interactive fusion features are extracted along the sequence axis by a multi-layer perceptron to determine the sequence features;

[0141] A multi-layer perceptron is used to transform the sequence features along the modal axis to generate modal sequence features;

[0142] Based on the multi-layer perceptron, the modal sequence features are processed along the channel axis and mixed multi-modal features are output.

[0143] Optionally, the sentiment analysis module 405 is specifically used for:

[0144] The mixed multimodal features are flattened using a flattening layer to determine the flattened features;

[0145] The flattened features are input into the self-attention layer for feature enhancement to generate enhanced features;

[0146] Perform a linear transformation on the enhanced features and output the target sentiment score.

[0147] Optionally, a model training module is also included, for:

[0148] Obtain training multimodal sentiment dataset and true sentiment score set;

[0149] Input the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and output a set of predicted sentiment scores;

[0150] Calculate the loss function value based on the predicted sentiment score set and the true sentiment score set;

[0151] If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, thereby determining the trained multimodal sentiment analysis model.

[0152] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of a multimodal sentiment analysis method based on multi-dimensional perception as described in any of the above embodiments.

[0153] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of a multimodal sentiment analysis method based on multi-dimensional perception as in any of the above embodiments are implemented.

[0154] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a multimodal sentiment analysis method based on multi-dimensional perception as in any of the above embodiments.

[0155] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0160] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on multi-dimensional perception, characterized in that: include: Acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; The multimodal sentiment analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and a sentiment prediction network; Extracting features of the multimodal emotion data to be identified through the feature extraction network, and constructing aligned text features, aligned audio features, and aligned video features; Inputting the aligned text features, the aligned audio features and the aligned video features into the transmembrane state interaction network for feature interaction, and determining an interactive fusion feature; Extracting features of the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multimodal features; The emotion prediction network is used to perform emotion scoring on the mixed multimodal features, and a target emotion score is output.

2. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The multimodal emotion data to be identified includes text emotion data, audio emotion data and visual emotion data; the feature extraction of the multimodal emotion data to be identified by the feature extraction network to construct aligned text features, aligned audio features and aligned video features includes: Extracting initial text features of text sentiment data based on a text extraction model; After the initial audio features of the audio emotion data are extracted using the audio extraction model, they are input into the Bi-LSTM layer for time series feature extraction to determine the intermediate audio features; The initial video features of the video emotion data are extracted through the video extraction model, and the temporal features are extracted based on the Bi-LSTM layer to generate intermediate video features; The initial text features, the intermediate audio features and the intermediate video features are respectively input into a one-dimensional temporal convolution layer for feature alignment, and aligned text features, aligned audio features and aligned video features are correspondingly output.

3. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The step of inputting the aligned text features, the aligned audio features, and the aligned video features into the transmembrane state interaction network for feature interaction and determining the interactive fusion features comprises: Mapping the aligned audio features into a query matrix and mapping the aligned video features into a key matrix and a value matrix, and then performing a cross attention operation to construct an interactive audio and video feature; The aligned text feature is projected as a query matrix, and the interactive audio and video feature is projected as a key matrix and a value matrix, and then a cross attention operation is performed to determine the interactive text audio and video features; Mapping the aligned video features into a query matrix and mapping the aligned audio features into a key matrix and a value matrix, and then performing a cross attention operation to construct interactive video and audio features; The aligned text feature projection is used as a query matrix, and the interactive video and audio feature projection is used as a key matrix and a value matrix, and then a cross attention operation is performed to determine the interactive text video and audio features; The aligned text features, the interactive text audio and video features, and the interactive text video and audio features are spliced ​​to determine interactive fusion features.

4. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The extracting the interactive fusion features in turn along multiple feature dimensions based on the multi-dimensional perception network to generate mixed multimodal features includes: Extracting the interactive fusion features along the sequence axis using a multi-layer perceptron to determine sequence features; A multi-layer perceptron is used to perform feature transformation on the sequence features along the modal axis to generate modal sequence features; Based on the multi-layer perceptron, feature processing is performed on the modal sequence features along the channel axis to output mixed multi-modal features.

5. The multimodal sentiment analysis method based on multidimensional perception according to claim 1 is characterized in that: The adopting the emotion prediction network to perform emotion scoring on the mixed multimodal features and outputting a target emotion score comprises: Using a flattening layer to flatten the hybrid multimodal feature to determine a flattened feature; Inputting the flattened features into the self-attention layer for feature enhancement to generate enhanced features; The enhanced features are linearly transformed and the target sentiment score is output.

6. The multimodal sentiment analysis method based on multidimensional perception according to claim 1, characterized in that: Also includes: Obtain training multimodal sentiment dataset and true sentiment score set; Inputting the training multimodal sentiment data set into the multimodal sentiment analysis model to be trained, and outputting a predicted sentiment score set; Calculating a loss function value according to the predicted emotion score set and the true emotion score set; If the loss function value has not converged, the multimodal sentiment analysis model to be trained is iteratively optimized according to the loss function value until the loss function value converges, thereby determining a trained multimodal sentiment analysis model.

7. A multimodal sentiment analysis system based on multi-dimensional perception, characterized in that: include: A data acquisition module, used to acquire multimodal emotion data to be identified, and input the multimodal emotion data to be identified into a trained multimodal emotion analysis model; the multimodal emotion analysis model includes a feature extraction network, a transmembrane interaction network, a multidimensional perception network and an emotion prediction network; A feature extraction module, used to extract features of the multimodal emotion data to be identified through the feature extraction network, and construct aligned text features, aligned audio features and aligned video features; A feature interaction module, used for inputting the aligned text features, the aligned audio features and the aligned video features into the transmembrane state interaction network for feature interaction, and determining an interactive fusion feature; A multi-dimensional perception module, used for extracting the interactive fusion features in sequence along multiple feature dimensions based on the multi-dimensional perception network to generate hybrid multi-modal features; The sentiment analysis module is used to use the sentiment prediction network to perform sentiment scoring on the mixed multimodal features and output a target sentiment score.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by the processor, the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the steps of the multimodal sentiment analysis method based on multidimensional perception as described in any one of claims 1 to 6 are implemented.

Images