Multi-modal emotion analysis method and device

Abstract

The invention discloses a multi-modal emotion analysis method and device, and relates to the technical field of emotion analysis. The method comprises the following steps: preprocessing each modal emotion of multi-modal emotion data, performing dimension alignment, and determining a target text feature, an aligned video feature and an aligned audio feature; a single-mode encoder is adopted to encode the aligned video features and the aligned audio features, and target video features and target audio features are correspondingly obtained; performing cross-membrane attention mechanism operation on the basis of the target text features, the target audio features and the target video features through a cross-modal fusion network and a cross-membrane conversion network, inputting the target text features, the target audio features and the target video features into a Transform network, performing self-attention operation, performing feature fusion, and determining emotion fusion features; and performing sentiment classification on the sentiment fusion features by adopting a sentiment classification layer, and outputting a sentiment analysis result. Based on the scheme, information loss during modal interaction is reduced, and the reliability of multi-modal emotion analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method and device. Background Art

[0002] Emotions are a core component of human psychological activities. Through sentiment analysis algorithms, we can determine whether the emotions contained in text or voice data are positive, negative, or neutral, and even further identify specific emotional states. These emotions can be applied to many industries, such as audio and video recommendations and public opinion analysis in the entertainment media industry, and student behavior analysis, classroom evaluation, and mental health monitoring in the smart education industry.

[0003] Traditional sentiment analysis algorithms are mainly focused on the field of text analysis, identifying the emotional tendencies in text by analyzing the lexical, syntactic structure and contextual semantics in the text. With the development of computer vision and speech recognition technology, researchers have begun to expand their attention to other modalities, such as facial expression analysis, speech sentiment analysis, etc. Video data can capture people's facial expressions and body language, which can also be a form of emotional expression, while audio data reflects the sound characteristics of people's emotional changes. For example, a high pitch may indicate that people are angry, and a vibration in the sound may indicate that people are afraid. The study of these multi-modal data reveals the diversity and complexity of human emotional expression, and also shows the limitations of single-modal sentiment analysis when facing complex emotional expressions.

[0004] Existing technologies consider combining multiple modal information (such as text, voice, and video) for sentiment analysis. However, this type of analysis must not only consider the combination of data from different modalities, but also focus on the consistency and differences of information between the modalities. For example, the receivers of audio and video streams may change with changes in the receiving frequency, which often leads to the loss of key local feature information, thus affecting the poor representation of key information in the feature fusion stage. In addition, there is also the phenomenon of important information loss during transmembrane interactions, resulting in low reliability of multimodal sentiment analysis. Summary of the invention

[0005] The present invention provides a multimodal emotion analysis method and device, which are used to improve the technical problem of low reliability in the prior art when performing multimodal emotion analysis.

[0006] A first aspect of the present invention provides a multimodal sentiment analysis method, comprising:

[0007] Acquire multimodal emotion data, preprocess and dimensionally align the emotions in each modality, and determine target text features, aligned video features, and aligned audio features;

[0008] Using a single-mode encoder to encode the aligned video features and the aligned audio features respectively, and correspondingly obtain a target video feature and a target audio feature;

[0009] Through the cross-modal fusion network, a cross-membrane attention mechanism operation between the text modality and the non-text modality is performed with the target text feature, the target video feature and the target audio feature as the center, and the cross-membrane attention mechanism operation is performed between the text modality and the non-text modality, and the cross-membrane attention mechanism operation is performed with the target text feature, the target video feature and the target audio feature as the center ... audio feature and the target audio feature as the center, and the cross-membrane attention mechanism operation is performed with the target audio feature and the target audio feature as the center, and the cross-membrane attention mechanism operation is performed with the target audio feature and the target audio feature as the center

[0010] Based on the transmembrane state conversion network, a pairwise transmembrane state attention mechanism operation is performed according to the target text feature, the target audio feature and the target video feature, and multiple interactive membrane state features are obtained by fusing them according to the modality;

[0011] Inputting the text audio transmembrane state feature, the text video transmembrane state feature and each of the interactive membrane state features into the Transformer network respectively for self-attention operation and then performing feature fusion to determine the emotion fusion feature;

[0012] The sentiment classification layer is used to perform sentiment classification on the sentiment fusion features and output the sentiment analysis results.

[0013] Optionally, the acquiring of multimodal emotion data, preprocessing and dimensionally aligning each of the modal emotions, and determining target text features, target video features, and target audio features include:

[0014] Obtain text modality emotion data, video modality emotion data, and audio modality emotion data;

[0015] Extracting initial text features of text modality sentiment data through a text extraction model;

[0016] An audio extraction model is used to extract initial audio features of audio modality emotion data;

[0017] Extracting initial video features of video modality emotion data based on a video extraction model;

[0018] The initial text features, the initial audio features and the initial video features are dimensionally aligned through a one-dimensional temporal convolution layer to obtain target text features, aligned audio features and aligned video features accordingly.

[0019] Optionally, the cross-modal fusion network includes a cross-modal attention layer, an addition and normalization layer, a feedforward fully connected layer and a fusion layer; the cross-modal fusion network performs a cross-modal attention mechanism operation between the text modality and the non-text modality with the target text feature, the target video feature and the target audio feature as the center, and classifies and fuses them to output text audio cross-modal features and text video cross-modal features, including:

[0020] Input the target text feature and the target audio feature into the transmembrane attention layer, perform transmembrane attention mechanism operation with the target text feature and the target audio feature as the center, and output text audio feature and audio text feature accordingly;

[0021] The target text feature and the target audio feature are respectively added element by element with the text audio feature and the audio text feature through an addition and normalization layer, and then normalized to obtain a standard text audio feature and a standard audio text feature;

[0022] A feed-forward fully connected layer is used to linearly transform the standard text audio features and the standard audio text features, respectively, and map the output transformed text audio features and transformed audio text features;

[0023] Based on the addition and normalization layer, the standard text audio features and the transformed text audio features, and the standard audio text features and the transformed audio text features are respectively added element by element and normalized, and then the text audio transmembrane state features are spliced ​​and output through the fusion layer;

[0024] Input the target text feature and the target video feature into the transmembrane state attention layer, perform transmembrane state attention mechanism operation with the target text feature and the target video feature as the center, and output text video feature and video text feature accordingly;

[0025] The target text feature and the target video feature are respectively added element by element with the text video feature and the video text feature through an addition and normalization layer, and then normalized to obtain a standard text video feature and a standard video text feature;

[0026] A feed-forward fully connected layer is used to linearly transform the standard text video features and the standard video text features, respectively, and map and output the transformed text video features and the transformed video text features;

[0027] Based on the addition and normalization layer, the standard text video features and the transformed text video features, as well as the standard video text features and the transformed video text features are added element by element and normalized, and then the text video transmembrane state features are output through splicing through the fusion layer.

[0028] Optionally, the process of operating the cross-modal attention mechanism in the cross-modal fusion network includes:

[0029] For the first input feature and the second input feature of the association operation, when the first input feature is processed as the center, the first input feature is used as a query matrix and the second input feature is used as a key matrix to perform matrix multiplication, and then normalized using an activation function to obtain a first intermediate feature;

[0030] After performing matrix multiplication operation using the second input feature as a query matrix and the first input feature as a key matrix, an activation function is used for normalization to obtain a second intermediate feature;

[0031] The first input feature is used as a value matrix to perform a matrix multiplication operation with the second intermediate feature, and then multiplied with the first intermediate feature to generate an output feature.

[0032] Optionally, the transmembrane state conversion network performs paired transmembrane state attention mechanism operations according to the target text features, the target audio features and the target video features, and fuses them by modality to obtain multiple interactive film state features, including:

[0033] Using the target text feature, the target audio feature and the target video feature to form a plurality of ordered pairs;

[0034] Based on the transmembrane state conversion network, the first element feature in each of the ordered pairs is used as the query matrix, and the second element feature is used as the key matrix and the value matrix to perform a transmembrane state attention mechanism operation to obtain the modal conversion feature of each of the ordered pairs;

[0035] The modal conversion features with the same first element features are concatenated to obtain text interaction membrane features, audio interaction membrane features and video interaction membrane features accordingly.

[0036] Optionally, when the target text feature is used as the first input feature, and the target video feature or the target audio feature is used as the second input feature, the cross-modal attention mechanism operation in the cross-modal fusion network is specifically as follows:

[0037] ;

[0038] in, ;

[0039] In the formula, For normalization operation, For the dimension of each attention head, is a text-audio feature or a text-video feature, is the target text feature as the query matrix, The target video feature or the target audio feature is used as the key matrix, The target video feature or the target audio feature is the transposed matrix of the key matrix, is a text-audio value matrix or a text-video value matrix, The target video feature or the target audio feature is used as the query matrix, is the target text feature as the key matrix, is the transposed matrix of the target text feature as the key matrix, is the target text feature as a value matrix;

[0040] When the target video feature or the target audio feature is used as the first input feature and the target text feature is used as the second input feature, the cross-modal fusion network cross-modal attention mechanism operation is specifically as follows:

[0041] ;

[0042] in, ;

[0043] In the formula, is an audio text feature or a video text feature, is an audio text value matrix or a video text value matrix, is the target video feature or the target audio feature as a matrix of values.

[0044] A second aspect of the present invention provides a multimodal emotion analysis device, comprising:

[0045] A data preprocessing module, used to obtain multimodal emotion data, preprocess each of the modal emotions and perform dimension alignment, determine target text features, align video features, and align audio features;

[0046] A feature encoding module, used to encode the aligned video features and the aligned audio features respectively using a single-mode encoder to obtain a target video feature and a target audio feature respectively;

[0047] A trans-modal fusion module is used to perform a trans-modal attention mechanism operation between text modality and non-text modality with the target text feature, the target video feature and the target audio feature as the center through a cross-modal fusion network, and classify and fuse them to output text audio trans-modal features and text video trans-modal features;

[0048] A transmembrane state conversion module, used for performing paired transmembrane state attention mechanism operations according to the target text features, the target audio features and the target video features based on the transmembrane state conversion network, and fusing them according to the modality to obtain multiple interactive membrane state features;

[0049] A feature fusion module, for inputting the text audio transmembrane state feature, the text video transmembrane state feature and each of the interactive membrane state features into a Transformer network for self-attention operation and then performing feature fusion to determine an emotion fusion feature;

[0050] The emotion classification module is used to classify the emotion fusion features using the emotion classification layer and output the emotion analysis results.

[0051] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal emotion analysis method as described in any one of the above items.

[0052] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the multimodal sentiment analysis method as described in any one of the above items.

[0053] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method as described in any one of the above items.

[0054] It can be seen from the above technical solutions that the present invention has the following advantages:

[0055] The above scheme of the present invention provides a multimodal emotion analysis method, including: obtaining multimodal emotion data, preprocessing each modal emotion and performing dimension alignment, determining target text features, aligned video features and aligned audio features; using a single-mode encoder to encode the aligned video features and the aligned audio features respectively, and correspondingly obtaining target video features and target audio features; using a cross-modal fusion network to perform a cross-membrane attention mechanism operation between text modality and non-text modality with target text features, target video features and target audio features as the center, and classifying and fusing them to output text audio cross-membrane features and text video cross-membrane features; based on a cross-membrane conversion network, performing a paired cross-membrane attention mechanism operation according to the target text features, target audio features and target video features, and fusing them according to the modality to obtain multiple interactive features; inputting the text audio cross-membrane features, the text video cross-membrane features and each interactive feature into the Transformer network for self-attention operation and then performing feature fusion to determine the emotion fusion feature; using the emotion classification layer to perform emotion classification on the emotion fusion feature, and outputting the emotion analysis result. Based on the above scheme, the quality and consistency of features of different modalities are ensured by preprocessing and dimensional alignment of multimodal emotion data. A single-modal encoder is used to retain the dynamic changes of video features and audio features in the time series, so that the features of each modality can be extracted more effectively. Then, the features of multiple modalities are respectively used for multi-modal information interaction through a cross-modal fusion network and a trans-membrane state conversion network, and the cross-modal interaction features are spliced ​​and output, which helps to reduce information loss during modal interaction and more accurately fuse useful features from each modality. Then, the Transformer self-attention mechanism is used to effectively integrate the information of the three modalities. Finally, the output emotion analysis results are spliced ​​and mapped, which improves the reliability of multimodal emotion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A flowchart of the steps of a multimodal emotion analysis method provided by an embodiment of the present invention;

[0058] Figure 2 A data processing structure diagram of a multimodal emotion analysis method provided by an embodiment of the present invention;

[0059] Figure 3A structural block diagram of a cross-modal fusion network provided by an embodiment of the present invention;

[0060] Figure 4 A structural block diagram of a transmembrane state conversion network provided by an embodiment of the present invention;

[0061] Figure 5 A structural block diagram of a multimodal emotion analysis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention provide a multimodal emotion analysis method and device, which are used to improve the technical problem of low reliability in the prior art when performing multimodal emotion analysis.

[0063] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention.

[0065] The present invention provides a multimodal sentiment analysis method, comprising:

[0066] Step 101: Acquire multimodal emotion data, preprocess each modal emotion and perform dimension alignment, determine target text features, aligned video features, and aligned audio features.

[0067] It should be noted that in the specific implementation, multimodal sentiment data can be obtained from the MOSI dataset and the MOSEI dataset. The MOSI dataset was proposed by A. Zadeh in 2016, and the MOSEI dataset was proposed by Amir Zadeh in 2018. Both datasets were obtained by screening and annotating videos on YouTube, and videos containing too many people were removed. Among them, the MOSEI dataset contains 23,453 text data and 3,228 video data, while the MOSI dataset contains approximately 2,191 text sentences and approximately 2,219 video clips. The sentiment intensity of both datasets is annotated using the [-3, 3] range, and they are commonly used datasets in multimodal sentiment analysis research.

[0068] Step 101 includes the following sub-steps:

[0069] S11. Obtain text modality emotion data, video modality emotion data, and audio modality emotion data.

[0070] It should be noted that multimodal emotion data usually involves three main modalities: text modality L, video modality V and audio modality A. To represent the input feature sequences and their dimensions of the three modalities of text modality emotion data, video modality emotion data, and audio modality emotion data.

[0071] S12. Extracting initial text features of text modality sentiment data through a text extraction model.

[0072] It should be noted that in order to process text data, the text extraction model is used to extract the key text features of the text modality sentiment data to obtain the initial text features. In specific implementation, the text extraction model can adopt the language representation (BERT) model. BERT learns the contextual meaning of words in sentences through a bidirectional Transformer model. When encoding each word, it combines the contextual information on the left and right sides of the word, so that it can better capture the diversity and complexity of the language, and provides a context-sensitive, pre-trained knowledge-rich feature representation, which greatly improves the quality of text feature extraction. Its encoding adopts the pre-trained model Bert-base-uncased. When using BERT for text feature extraction, the input of the model requires adding special tags "[CLS]" and "[SEP]" before and after the text, that is, at the beginning and end of the sequence for processing, to help the model better understand the context structure of the sequence and generate a context vector representation of each sequence. Subsequently, the tokenized text sequence will be input into the BERT model to extract useful word embedding features. Finally, we can obtain a feature sequence with a dimension of 768, which captures rich information in the text data. In the MOSI dataset, the tensor dimension of the text modality L is ,in, , .

[0073] S13. Use an audio extraction model to extract initial audio features of audio modality emotion data.

[0074] It should be noted that the audio feature extraction of the audio modality emotion data is performed through the audio extraction model to obtain the initial audio features. In the specific implementation, the audio extraction model can adopt the wav2vec2.0 model. As a self-supervised learning model, it is specially designed to extract features from audio waveforms and aims to improve the performance of speech recognition and audio processing tasks. Its core idea is to construct a self-built supervised training target through vector quantization (VQ), and use the contrastive learning loss function to train after a large number of masking of the input. The feature extractor based on the convolutional network (CNN) encodes the original audio into a frame feature sequence, and transforms each frame feature into a discrete feature Q through the VQ module, which is used as a self-supervised target. At the same time, the frame feature sequence is masked and enters the Transformer model to obtain the context representation C. Finally, the contrastive learning loss function is used to shorten the distance between the context representation of the mask position and the corresponding discrete feature q, that is, the positive sample pair; after feature extraction, in the MOSI dataset, the tensor dimension of the audio modality A is ,in , .

[0075] S14. Extracting initial video features of video modality emotion data based on a video extraction model.

[0076] It should be noted that the video extraction model is used to extract video features from the video modality emotion data to obtain the initial video features. In the specific implementation, the video extraction model can use the Vision Transformer (ViT) model, in which the video frames are regarded as static images and processed in combination with the time series modeling technology. It can effectively extract features from the video modality emotion data and fully capture the visual information in the video; and the ViT model can use pre-trained models for feature extraction, which is particularly useful for video data, especially when the amount of training data is limited, the pre-trained model can improve the expressiveness of the features; after feature extraction, in the MOSI dataset, the tensor dimension of the video modality V is ,in, , .

[0077] S15. Dimensionally aligning the initial text features, the initial audio features, and the initial video features through a one-dimensional temporal convolutional layer, and obtaining target text features, aligned audio features, and aligned video features accordingly.

[0078] It should be noted that since the direct sequence length and feature dimension of each modality are different after feature extraction, in order to ensure that each element feature of the input feature sequence of each modality has sufficient perception of its neighboring element features, such as Figure 2As shown, we pass the input feature sequence to a one-dimensional temporal convolutional layer (Conv1) to extract local semantic information: ,in, For modal The convolution kernel size is is a common dimension; finally, the tensor dimensions of the three modalities are the target text features , align audio features and align video features ([T, d]); The convolution sequence is expected to contain the local structure of the sequence, which is important because the sequences are collected at different sampling rates. In addition, since the 1D temporal convolution layer projects features from different modalities into the same dimension d, the dot product is admissible in the cross-modal attention mechanism operation.

[0079] Step 102: Use a single-mode encoder to encode the aligned video features and the aligned audio features respectively, and obtain target video features and target audio features accordingly.

[0080] It should be noted that, in specific implementation, the single-mode encoder can use a single-directional long short-term memory network (sLSTM) to align audio features and video features. Unidirectional long short-term memory networks are used for encoding respectively. The hidden state of each unidirectional long short-term memory network obtains the context information between different modalities through the self-attention mechanism (the encoding layer model function of a layer of 8-head Transformer), and finally generates a unimodal representation to obtain the target audio features. and target video features .

[0081] Step 103: Perform a cross-modal attention mechanism operation between the text modality and the non-text modality with the target text feature, the target video feature and the target audio feature as the center through a cross-modal fusion network, classify and fuse them, and output text-audio cross-modal features and text-video cross-modal features.

[0082] It should be noted that, since text data can provide rich semantic information, which is particularly important for sentiment analysis, words and phrases can directly express emotions and feelings, and sentence structures can provide context for emotions. Therefore, this embodiment is based on a cross-modal fusion network, which takes the text modality as the center and combines the two non-text modalities of video and audio to perform cross-modal attention mechanism operations. At the same time, the video modality and audio modality are respectively used as the center to perform cross-modal attention mechanism operations with the text modality, so as to enrich the possible audio and video information loss in the text-centric operation;

[0083] like Figure 3 As shown, the cross-modal fusion network (SL-Attention Transformer) includes at least one trans-membrane state fuser and a fusion layer (CAT), each trans-membrane state fuser includes a trans-membrane state attention layer (SL-Attention), an addition and normalization layer (Add & Norm) and a feed-forward fully connected layer (Feed-Forward); step 103 includes the following sub-steps:

[0084] Input the target text features and target audio features into the transmembrane attention layer, perform transmembrane attention mechanism operations with the target text features and target audio features as the center, and output text audio features and audio text features accordingly;

[0085] The target text features and the target audio features are respectively added element by element with the text audio features and the audio text features through the addition and normalization layer, and then normalized to obtain the standardized text audio features and the standardized audio text features;

[0086] A feed-forward fully connected layer is used to linearly transform the standard text audio features and the standard audio text features, respectively, and map the output transformed text audio features and transformed audio text features;

[0087] Based on the addition and normalization layer, the standard text audio features and the transformed text audio features, as well as the standard audio text features and the transformed audio text features are added element by element and normalized, and then the text audio transmembrane state features are output through splicing through the fusion layer;

[0088] Input the target text features and target video features into the transmembrane attention layer, perform transmembrane attention mechanism operations with the target text features and target video features as the center, and output text video features and video text features accordingly;

[0089] The target text features and the target video features are respectively added element by element with the text video features and the video text features through the addition and normalization layer, and then normalized to obtain the standard text video features and the standard video text features;

[0090] A feed-forward fully connected layer is used to linearly transform the standard text video features and the standard video text features, respectively, and the output is mapped to the transformed text video features and the transformed video text features;

[0091] Based on the addition and normalization layer, the standard text video features and the transformed text video features, as well as the standard video text features and the transformed video text features are added element by element and normalized, and then the text video transmembrane state features are output through splicing through the fusion layer.

[0092] Furthermore, the process of operating the cross-modal attention mechanism in the cross-modal fusion network includes:

[0093] For the first input feature and the second input feature of the association operation, when the first input feature is processed as the center, the first input feature is used as the query matrix and the second input feature is used as the key matrix for matrix multiplication, and then normalized using an activation function to obtain a first intermediate feature;

[0094] After matrix multiplication, the second input feature is used as the query matrix and the first input feature is used as the key matrix, and then normalized using an activation function to obtain a second intermediate feature;

[0095] The first input feature is used as a value matrix to perform matrix multiplication with the second intermediate feature, and then multiplied with the first intermediate feature to generate an output feature.

[0096] It should be noted that if Figure 3 As shown in the text-leading text and audio / video transmembrane attention layer, the target text feature is used as the first input feature in the transmembrane attention mechanism operation, and the target video feature or the target audio feature is used as the second input feature. As query matrix query, key matrix key and value matrix value, extract or Use the query matrix query and the key matrix key to perform cross-modal attention mechanism operations:

[0097] ;

[0098] in, ;

[0099] In the formula, For normalization operation, is the dimension of each attention head, Audio features for text or text video features , is the target text feature as the query matrix, The target video feature or the target audio feature is used as the key matrix, The target video feature or the target audio feature is the transposed matrix of the key matrix, is a text-audio value matrix or a text-video value matrix, The target video feature or the target audio feature is used as the query matrix, is the target text feature as the key matrix, is the transposed matrix of the target text feature as the key matrix, is the target text feature as a value matrix;

[0100] In the audio / video and text transmembrane attention layer with the target video feature or the target audio feature as the center ((A / V)-Lead), the target video feature or the target audio feature is used as the first input feature in the transmembrane attention mechanism operation, and the target text feature is used as the second input feature. By using the extracted or As query matrix query, key matrix key and value matrix value, extract Use the query matrix query and the key matrix key to perform cross-modal attention mechanism operations:

[0101] ;

[0102] in, ;

[0103] In the formula, For normalization operation, is the dimension of each attention head, Audio text features or video text features , The target video feature or the target audio feature is used as the query matrix, is the target text feature as the key matrix, is the transposed matrix of the target text feature as the key matrix, is an audio text value matrix or a video text value matrix, is the target text feature as the query matrix, The target video feature or the target audio feature is used as the key matrix, The target video feature or the target audio feature is the transposed matrix of the key matrix, is the target video feature or the target audio feature as a value matrix;

[0104] Then, after two residual connection and normalization layers and a feed-forward fully connected layer, the text audio features, audio text features, text video features, and video text features are processed respectively, and the corresponding output center text audio features are , center audio text features , Central text video features and central video text features Finally, the central text audio features are concatenated with the central audio text features, the central text video features, and the central video text features through the fusion layer, thereby obtaining the corresponding text audio transmembrane state features. and text-video transmembrane state features :

[0105]

[0106] Step 104: Based on the transmembrane state conversion network, a paired transmembrane state attention mechanism operation is performed according to the target text features, the target audio features and the target video features, and multiple interactive membrane state features are obtained by fusing them according to the modality.

[0107] It should be noted that if Figure 2 and Figure 4 As shown, the role of the cross-modal transformer network is to interactively fuse the target text features, target audio features, and target video features of the three modalities. The specific structure of the Crossmodal Transformer network can refer to the existing technology and will not be repeated here.

[0108] Step 104 includes the following sub-steps:

[0109] Using target text features, target audio features and target video features to form multiple ordered pairs;

[0110] Based on the transmembrane state conversion network, the first element feature in each ordered pair is used as the query matrix, and the second element feature is used as the key matrix and the value matrix to perform the transmembrane state attention mechanism operation to obtain the modal conversion features of each ordered pair;

[0111] The modal conversion features with the same first element features are concatenated to obtain text interaction membrane features, audio interaction membrane features and video interaction membrane features accordingly.

[0112] It should be noted that the target text features are transformed through the transmembrane state conversion network , target audio features and target video features CrossAttention is performed separately. First, the target text features, target audio features, and target video features are arranged and combined to form multiple ordered pairs. An ordered pair refers to a feature pair composed of two element features of the first element feature and the second element feature in a specific order, where the first element feature and the second element characteristics From the modal The target text features in , target audio features Or target video features The first element feature and the second element feature are different; using the extracted , or As the first element feature and as the query matrix query, As the key matrix key and the value matrix value, the cross-modal attention mechanism is operated:

[0113] ;

[0114] In the formula, is the mode conversion feature, For normalization operation, The first element feature is used as the query matrix, The second element feature is used as the key matrix, The second element characteristic is the transposed matrix of the key matrix, For the dimension of each attention head, is the second element feature as the value matrix, is the first element characteristic, is the first learnable parameter, is the second learnable parameter, is the transpose of the second learnable parameter, is the transpose of the second element characteristic, is the second element characteristic, is the third learnable parameter;

[0115] By performing a cross-modal attention mechanism operation on each ordered pair, six cross-membrane modal conversion features are obtained. The modal conversion features with the same first element features are concatenated to obtain the corresponding text interaction membrane state features. , audio interactive membrane characteristics Interactive film characteristics with video :

[0116]

[0117] In the formula, is a modal conversion feature whose first element feature is the target text feature and whose second element feature is the target audio feature, is a modal conversion feature whose first element feature is the target text feature and whose second element feature is the target video feature, is a modal conversion feature whose first element feature is the target audio feature and whose second element feature is the target text feature, is a modal conversion feature whose first element feature is the target audio feature and whose second element feature is the target video feature, is a modal conversion feature whose first element feature is the target video feature and whose second element feature is the target text feature, The first element feature is a modal conversion feature of the target video feature and the second element feature is a modal conversion feature of the target audio feature.

[0118] Step 105: Input the text audio transmembrane state features, the text video transmembrane state features and each interactive membrane state feature into the Transformer network respectively for self-attention operation and then perform feature fusion to determine the emotion fusion feature.

[0119] It should be noted that when processing sequence data, the Self-Attention mechanism of the Transformer network can capture the global dependencies in the input features, so that the information between different time steps or different modes can be fully interacted and integrated. Therefore, in order to further effectively integrate multimodal data, the five features output by the cross-modal fusion network and the trans-membrane state conversion network are used as attention operation input features for the SelfAttention Transformer, and then the output of the last hidden state of the five features in the Transformer network is fused to represent the emotion fusion feature; the SelfAttention layer is defined as:

[0120] ;

[0121] In the formula, Output features for attention operation, The query matrix of input features for the attention operation, The key matrix of input features for the attention operation, The transposed matrix of the key matrix of the input features for the attention operation, is the dimension of each attention head, Matrix of values ​​of the input features to the attention operation.

[0122] Step 106: Use the sentiment classification layer to perform sentiment classification on the sentiment fusion features and output the sentiment analysis results.

[0123] It should be noted that, in specific implementation, the emotion fusion feature can be mapped to a single output node using a linear layer as the emotion classification layer, which generates a score representing the emotion intensity, i.e., the emotion analysis result. , which reflects the prediction of the emotional content of multimodal emotional data; in one implementation, this embodiment can adopt 7-category and 2-category emotional intensity. The 7-category emotional intensity is normalized by the softmax function after the output of the last hidden state, and then Linear outputs a unique value and projects it to the [-3, 3] interval for emotional intensity judgment. For the 2-category task, the output result only needs to be positive or negative.

[0124] In order to verify the effectiveness of the above method, we conducted experiments on the MOSI and MOSEI datasets. The experimental results are shown in Table 1:

[0125] Table 1 Comparison of experimental results of different models on MOSI and MOSEI datasets

[0126]

[0127] In Table 1, the LF-LSTM model, MFN model, TFN model and MULT model are commonly used models in existing multimodal sentiment analysis (MMSA). Our method is improved based on the MULT model. According to the scores measured by indicators such as ACC2, F1 and ACC7 on the MOSI dataset and the MOSEI dataset, the index scores of our method are relatively higher than those of other commonly used models. Therefore, after experimental verification, it can be seen that our method has achieved certain results, which is helpful to improve the reliability of multimodal sentiment analysis and is of great significance to the fields of emotion recognition, user experience evaluation and mental health monitoring.

[0128] In this embodiment, the quality and consistency of features of different modalities are ensured by preprocessing and dimensional alignment of multimodal emotion data, and a single-modal encoder is used to retain the dynamic changes of video features and audio features in the time series, so that the features of each modality can be extracted more effectively. The features of multiple modalities are then used through a cross-modal fusion network and a trans-membrane state conversion network for multi-modal information interaction and spliced ​​to output cross-modal interaction features, which helps to reduce information loss during modal interaction so that useful features from each modality can be more accurately fused. The Transformer self-attention mechanism is then used to effectively integrate the information of the three modalities. Finally, the output emotion analysis results are spliced ​​and mapped, thereby improving the reliability of multimodal emotion analysis.

[0129] See also Figure 5 , Figure 5 A structural block diagram of a multimodal emotion analysis device provided by an embodiment of the present invention.

[0130] The present invention provides a multimodal emotion analysis device, comprising:

[0131] The data preprocessing module 501 is used to obtain multimodal emotion data, preprocess each modal emotion and perform dimension alignment, determine target text features, align video features, and align audio features;

[0132] A feature encoding module 502 is used to encode the aligned video features and the aligned audio features respectively using a single-mode encoder to obtain a target video feature and a target audio feature respectively;

[0133] The trans-modal fusion module 503 is used to perform trans-modal attention mechanism operations between text modalities and non-text modalities with target text features, target video features, and target audio features as the center through a cross-modal fusion network, and classify and fuse them to output text audio trans-modal features and text video trans-modal features;

[0134] A transmembrane state conversion module 504 is used to perform a paired transmembrane state attention mechanism operation according to target text features, target audio features and target video features based on a transmembrane state conversion network, and fuse them according to the modality to obtain multiple interactive membrane state features;

[0135] The feature fusion module 505 is used to input the text audio transmembrane state features, the text video transmembrane state features and each interactive membrane state feature into the Transformer network for self-attention operation and then perform feature fusion to determine the emotion fusion feature;

[0136] The emotion classification module 506 is used to perform emotion classification on the emotion fusion features using the emotion classification layer and output the emotion analysis result.

[0137] Optionally, the data preprocessing module 501 is specifically used for:

[0138] Obtain text modality emotion data, video modality emotion data, and audio modality emotion data;

[0139] Extracting initial text features of text modality sentiment data through a text extraction model;

[0140] An audio extraction model is used to extract initial audio features of audio modality emotion data;

[0141] Extracting initial video features of video modality emotion data based on a video extraction model;

[0142] The initial text features, initial audio features, and initial video features are dimensionally aligned through a one-dimensional temporal convolutional layer to obtain target text features, aligned audio features, and aligned video features respectively.

[0143] Optionally, the cross-modal fusion network includes a cross-modal attention layer, an addition and normalization layer, a feed-forward fully connected layer and a fusion layer; the cross-modal fusion module 503 is specifically used for:

[0144] Input the target text features and target audio features into the transmembrane attention layer, perform transmembrane attention mechanism operations with the target text features and target audio features as the center, and output text audio features and audio text features accordingly;

[0145] The target text features and the target audio features are respectively added element by element with the text audio features and the audio text features through the addition and normalization layer, and then normalized to obtain the standardized text audio features and the standardized audio text features;

[0146] A feed-forward fully connected layer is used to linearly transform the standard text audio features and the standard audio text features, respectively, and map the output transformed text audio features and transformed audio text features;

[0147] Based on the addition and normalization layer, the standard text audio features and the transformed text audio features, the standard audio text features and the transformed audio text features are added element by element and normalized, and then the text audio transmembrane state features are output through splicing through the fusion layer;

[0148] Input the target text features and target video features into the transmembrane attention layer, perform transmembrane attention mechanism operations with the target text features and target video features as the center, and output text video features and video text features accordingly;

[0149] The target text features and the target video features are respectively added element by element with the text video features and the video text features through the addition and normalization layer, and then normalized to obtain the standard text video features and the standard video text features;

[0150] A feed-forward fully connected layer is used to linearly transform the standard text video features and the standard video text features, respectively, and the output is mapped to the transformed text video features and the transformed video text features;

[0151] Based on the addition and normalization layer, the standard text video features and the transformed text video features, as well as the standard video text features and the transformed video text features are added element by element and normalized, and then the text video transmembrane state features are output through splicing through the fusion layer.

[0152] Optionally, the process of operating the cross-modal attention mechanism in the cross-modal fusion network includes:

[0153] For the first input feature and the second input feature of the association operation, when the first input feature is processed as the center, the first input feature is used as the query matrix and the second input feature is used as the key matrix for matrix multiplication, and then normalized using an activation function to obtain a first intermediate feature;

[0154] After matrix multiplication, the second input feature is used as the query matrix and the first input feature is used as the key matrix, and then normalized using an activation function to obtain a second intermediate feature;

[0155] The first input feature is used as a value matrix to perform matrix multiplication with the second intermediate feature, and then multiplied with the first intermediate feature to generate an output feature.

[0156] Furthermore, when the target text feature is used as the first input feature, the target video feature or the target audio feature is used as the second input feature, the cross-modal attention mechanism operation in the cross-modal fusion network is specifically as follows:

[0157] ;

[0158] in, ;

[0159] In the formula, For normalization operation, For the dimension of each attention head, is a text-audio feature or a text-video feature, is the target text feature as the query matrix, The target video feature or the target audio feature is used as the key matrix, The target video feature or the target audio feature is the transposed matrix of the key matrix, is a text-audio value matrix or a text-video value matrix, The target video feature or the target audio feature is used as the query matrix, is the target text feature as the key matrix, is the transposed matrix of the target text feature as the key matrix, is the target text feature as a value matrix;

[0160] When the target video feature or the target audio feature is used as the first input feature and the target text feature is used as the second input feature, the cross-modal fusion network cross-modal attention mechanism operation is specifically as follows:

[0161] ;

[0162] in, ;

[0163] In the formula, is an audio text feature or a video text feature, is an audio text value matrix or a video text value matrix, is the target video feature or the target audio feature as a matrix of values.

[0164] Optionally, the transmembrane state conversion module 504 is specifically used for:

[0165] Using target text features, target audio features and target video features to form multiple ordered pairs;

[0166] Based on the transmembrane state conversion network, the first element feature in each ordered pair is used as the query matrix, and the second element feature is used as the key matrix and the value matrix to perform the transmembrane state attention mechanism operation to obtain the modal conversion features of each ordered pair;

[0167] The modal conversion features with the same first element features are concatenated to obtain text interaction membrane features, audio interaction membrane features and video interaction membrane features accordingly.

[0168] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal emotion analysis method as described in any of the above embodiments.

[0169] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal emotion analysis method as described in any of the above embodiments are implemented.

[0170] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the multimodal emotion analysis method of any of the above embodiments when the computer program / instruction is executed by a processor.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0176] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquire multimodal emotion data, preprocess and dimensionally align the emotions in each modality, and determine target text features, aligned video features, and aligned audio features; Using a single-mode encoder to encode the aligned video features and the aligned audio features respectively, and correspondingly obtain a target video feature and a target audio feature; Through the cross-modal fusion network, a cross-membrane attention mechanism operation between the text modality and the non-text modality is performed with the target text feature, the target video feature and the target audio feature as the center, and the cross-membrane attention mechanism operation is performed between the text modality and the non-text modality, and the cross-membrane attention mechanism operation is performed with the target text feature, the target video feature and the target audio feature as the center, and the cross-membrane attention mechanism operation is performed with the target Based on the transmembrane state conversion network, a pairwise transmembrane state attention mechanism operation is performed according to the target text feature, the target audio feature and the target video feature, and multiple interactive membrane state features are obtained by fusing them according to the modality; Inputting the text audio transmembrane state feature, the text video transmembrane state feature and each of the interactive membrane state features into the Transformer network respectively for self-attention operation and then performing feature fusion to determine the emotion fusion feature; The sentiment classification layer is used to perform sentiment classification on the sentiment fusion features and output the sentiment analysis results.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The step of acquiring multimodal emotion data, preprocessing and dimensionally aligning the emotions in each modality, and determining target text features, target video features, and target audio features includes: Obtain text modality emotion data, video modality emotion data, and audio modality emotion data; Extracting initial text features of text modality sentiment data through a text extraction model; An audio extraction model is used to extract initial audio features of audio modality emotion data; Extracting initial video features of video modality emotion data based on a video extraction model; The initial text features, the initial audio features and the initial video features are dimensionally aligned through a one-dimensional temporal convolution layer to obtain target text features, aligned audio features and aligned video features accordingly.

3. The multimodal sentiment analysis method according to claim 1, characterized in that: The cross-modal fusion network includes a cross-membrane attention layer, an addition and normalization layer, a feedforward fully connected layer and a fusion layer; the cross-modal fusion network performs a cross-membrane attention mechanism operation between the text modality and the non-text modality with the target text feature, the target video feature and the target audio feature as the center, and classifies and fuses them to output text audio cross-membrane features and text video cross-membrane features, including: Input the target text feature and the target audio feature into the transmembrane attention layer, perform transmembrane attention mechanism operation with the target text feature and the target audio feature as the center, and output text audio feature and audio text feature accordingly; The target text feature and the target audio feature are respectively added element by element with the text audio feature and the audio text feature through an addition and normalization layer, and then normalized to obtain a standard text audio feature and a standard audio text feature; A feed-forward fully connected layer is used to linearly transform the standard text audio features and the standard audio text features, respectively, and map the output transformed text audio features and transformed audio text features; Based on the addition and normalization layer, the standard text audio features and the transformed text audio features, and the standard audio text features and the transformed audio text features are respectively added element by element and normalized, and then the text audio transmembrane state features are spliced ​​and output through the fusion layer; Input the target text feature and the target video feature into the transmembrane state attention layer, perform transmembrane state attention mechanism operation with the target text feature and the target video feature as the center, and output text video feature and video text feature accordingly; The target text feature and the target video feature are respectively added element by element with the text video feature and the video text feature through an addition and normalization layer, and then normalized to obtain a standard text video feature and a standard video text feature; A feed-forward fully connected layer is used to linearly transform the standard text video features and the standard video text features, respectively, and map and output the transformed text video features and the transformed video text features; Based on the addition and normalization layer, the standard text video features and the transformed text video features, as well as the standard video text features and the transformed video text features are added element by element and normalized, and then the text video transmembrane state features are output through splicing through the fusion layer.

4. The multimodal sentiment analysis method according to claim 1, characterized in that: The process of operating the cross-modal attention mechanism in the cross-modal fusion network includes: For the first input feature and the second input feature of the association operation, when the first input feature is processed as the center, the first input feature is used as a query matrix and the second input feature is used as a key matrix to perform matrix multiplication, and then normalized using an activation function to obtain a first intermediate feature; After performing matrix multiplication operation using the second input feature as a query matrix and the first input feature as a key matrix, an activation function is used for normalization to obtain a second intermediate feature; The first input feature is used as a value matrix to perform a matrix multiplication operation with the second intermediate feature, and then multiplied with the first intermediate feature to generate an output feature.

5. The multimodal sentiment analysis method according to claim 1, characterized in that: The transmembrane state conversion network performs paired transmembrane state attention mechanism operations according to the target text features, the target audio features, and the target video features, and fuses them according to the modality to obtain multiple interactive membrane state features, including: Using the target text feature, the target audio feature and the target video feature to form a plurality of ordered pairs; Based on the transmembrane state conversion network, the first element feature in each of the ordered pairs is used as the query matrix, and the second element feature is used as the key matrix and the value matrix to perform a transmembrane state attention mechanism operation to obtain the modal conversion feature of each of the ordered pairs; The modal conversion features with the same first element features are concatenated to obtain text interaction membrane features, audio interaction membrane features and video interaction membrane features accordingly.

6. The multimodal sentiment analysis method according to claim 4, characterized in that: When the target text feature is used as the first input feature, and the target video feature or the target audio feature is used as the second input feature, the cross-modal attention mechanism operation in the cross-modal fusion network is specifically as follows: ； in, ; In the formula, For normalization operation, For the dimension of each attention head, is a text-audio feature or a text-video feature, is the target text feature as the query matrix, The target video feature or the target audio feature is used as the key matrix, The target video feature or the target audio feature is the transposed matrix of the key matrix, is a text-audio value matrix or a text-video value matrix, The target video feature or the target audio feature is used as the query matrix, is the target text feature as the key matrix, is the transposed matrix of the target text feature as the key matrix, is the target text feature as a value matrix; When the target video feature or the target audio feature is used as the first input feature and the target text feature is used as the second input feature, the cross-modal fusion network cross-modal attention mechanism operation is specifically as follows: ； in, ; In the formula, is an audio text feature or a video text feature, is an audio text value matrix or a video text value matrix, is the target video feature or the target audio feature as a matrix of values.

7. A multimodal emotion analysis device, characterized in that: include: A data preprocessing module, used to obtain multimodal emotion data, preprocess each of the modal emotions and perform dimension alignment, determine target text features, align video features, and align audio features; A feature encoding module, used to encode the aligned video features and the aligned audio features respectively using a single-mode encoder to obtain a target video feature and a target audio feature respectively; A trans-modal fusion module is used to perform a trans-modal attention mechanism operation between text modality and non-text modality with the target text feature, the target video feature and the target audio feature as the center through a cross-modal fusion network, and classify and fuse them to output text audio trans-modal features and text video trans-modal features; A transmembrane state conversion module, used to perform paired transmembrane state attention mechanism operations according to the target text features, the target audio features and the target video features based on the transmembrane state conversion network, and fuse them according to the modality to obtain multiple interactive membrane state features; A feature fusion module, for inputting the text audio transmembrane state feature, the text video transmembrane state feature and each of the interactive membrane state features into a Transformer network for self-attention operation and then performing feature fusion to determine an emotion fusion feature; The emotion classification module is used to classify the emotion fusion features using the emotion classification layer and output the emotion analysis results.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal emotion analysis method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the multimodal sentiment analysis steps of the method according to any one of claims 1 to 6 are implemented.

Images