Multimodal fusion and sentiment recognition method based on attention mechanism

Abstract

The application relates to the technical field of emotion recognition, and discloses a multi-modal fusion and emotion recognition method based on an attention mechanism, which comprises a preprocessing stage, a first-stage fusion training and a second-stage fusion training; in the first stage, time sequence relationships of audio and video are constructed through a long short-term memory network, then the audio and the video are fused through an attention mechanism, high-dimensional features of a result are extracted, and then high-dimensional features of video and text and text and audio are obtained through the same method. In the first stage, high-dimensional features are obtained by fusing two modes each time, and redundant information is filtered in the training process. In the second stage, the high-dimensional features fused by the two modes each time extracted in the first stage are passed through a gated recurrent network, then the three features are fused through an attention mechanism, then the obtained result is passed through a full connection layer and a multilayer perceptron to obtain a predicted recognition result, and the classification precision is improved through sufficient feature fusion.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, and more specifically to a multimodal fusion and emotion recognition method based on an attention mechanism. Background Technology

[0002] Existing multimodal feature fusion methods are mainly divided into model-independent methods and model-based methods. Model-independent methods are further divided into early fusion, late fusion, and hybrid fusion. These methods do not depend on model structure and are applicable to a wide range of tasks, but they lack specificity and are difficult to adjust according to different task requirements. Model-based multimodal fusion methods can utilize the relationships between different modalities and leverage the advantages of the model to achieve better performance. Model-based methods mainly use neural network methods. Among neural network methods, recurrent neural networks and graph neural networks are often used for multimodal feature fusion. These methods focus on collecting information between discourses but neglect the mining of potential relationships between features within or between modalities.

[0003] The emotion recognition method in this invention uses a recurrent neural network with an attention mechanism to simultaneously focus on the feature fusion of two-way modalities and the feature fusion of three modalities, effectively improving classification accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal fusion and emotion recognition method based on an attention mechanism. The method uses an attention mechanism to classify audio, video, and text features, constructs temporal relationships using a recurrent neural network, and then uses the attention mechanism to fuse the three modalities.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A multimodal fusion and emotion recognition method based on an attention mechanism includes a preprocessing stage, a first stage, and a second stage, specifically including:

[0007] Preprocessing stage: Preprocessing the original video, audio, and text to convert their contents into video features. Audio features Text features

[0008] The first stage involves using a bidirectional long short-term memory network to transmit video and audio features to obtain the temporal relationship of the discourse; embedding speaker information and fusing video and audio features through a multi-head attention mechanism to obtain high-dimensional video and audio features. In this context, video features serve as the keys, while audio features serve as both the query and the value.

[0009] Video and text features are combined using a bidirectional long short-term memory network to obtain the temporal relationship of the discourse; speaker information is embedded, and video and text features are fused through a multi-head attention mechanism to obtain high-dimensional video-text features. In this context, video features serve as keys, while text features serve as both queries and values.

[0010] Text and audio features are combined using a bidirectional long short-term memory network to obtain the temporal relationship of the discourse; speaker information is embedded, and text and audio features are fused through an attention mechanism to obtain high-dimensional features of the audio text. The audio features serve as the keys, while the text features serve as both the query and the value.

[0011] The second stage involves passing the high-dimensional features of video and audio, video and text, and audio and text through a gated recurrent network, then fusing them using an attention mechanism. The resulting data is then passed through a fully connected layer and a multilayer perceptron to obtain the recognition result.

[0012] Furthermore, in the preprocessing stage, when preprocessing the original video, audio, and text, TextCNN is used to extract text features, OpenSmile toolkit with IS10 configuration is used to extract audio features, and a pre-trained dense connection network is used to extract video features.

[0013] Furthermore, the first phase specifically includes:

[0014] Audio features Video features and text features The first audio features were obtained by using a bidirectional long short-term memory network. First video features and first text features Speaker information is embedded into the features to obtain the speaker embedding S. i :

[0015]

[0016] s i This indicates speaker information, where i represents the dialogue sequence number, and W... s Indicates the learnable weight parameters. Indicates single-modal characteristics;

[0017] Speaker embedding adds speaker information to the discourse;

[0018] Each utterance is represented by the following three nodes: Node node node Using h′ i a, h′ i v , h′ i t Initialize the node, where the second audio feature Second video features Second text features

[0019] The high-dimensional video and audio features f are obtained by fusing video features and audio features. i va At this time, video features are used as keys, and audio features are used as both queries and values. A multi-head attention mechanism is used to fuse the features.

[0020]

[0021] d x d k / h, where h represents the number of attention heads, d k Q represents the number of dimensions of the key. a Represents Query, K v Key, K vT In this context, T represents the matrix transpose, and V represents the value;

[0022] The video and audio fusion features f obtained through the attention mechanism i va The classification result is obtained through a fully connected layer.

[0023] The video and audio fusion feature corresponding to the best training classification result is selected as the final video and audio fusion feature. va .

[0024] Furthermore, during the first stage of training, classification cross-entropy and L2 regularization are used as the loss function:

[0025]

[0026] in Here, N represents the number of dialogues, c(i) represents the number of utterances in dialogue i, c(s) represents the number of utterances in dialogue s, and P represents the loss function for the first stage of training. i，j y is the probability distribution of the predicted sentiment label of utterance j in dialogue i. i，j λ is the sentiment label of utterance j in real dialogue i, λ is the weight of L2 regularization term ||·||2, and θ is a trainable parameter.

[0027] Furthermore, the second stage specifically includes: extracting high-dimensional features from video and audio. High-dimensional features of video text High-dimensional features of audio text Construct the timing relationships using gated cyclic networks respectively:

[0028]

[0029]

[0030]

[0031] and These are respectively referred to as the first video / audio high-dimensional feature, the first video / text high-dimensional feature, and the first audio / text high-dimensional feature; speaker information is embedded into these features to obtain the speaker embedding S. i :

[0032]

[0033] s i Indicates speaker information, H′ is obtained after speaker embedding. t va H′ t vt H′ t at H′ t va H′ t vt H′ t at These are respectively referred to as the second video audio high-dimensional features, the second video text high-dimensional features, and the second audio text high-dimensional features;

[0034] Then, an attention mechanism is used for fusion to obtain the fusion result E. i :

[0035] E i =ATT(H′) t va H′ t vt H′ t at );

[0036] ATT stands for multi-head attention mechanism. In the second phase of the multi-head attention mechanism, the key, query, and value are respectively...

[0037] Finally, a fully connected layer and an MLP layer are used for sentiment classification to obtain the final recognition result:

[0038] H i =FC(E i );

[0039] l i=RELU(W l H i +b l );

[0040] P i =Softmax(W smax l i +b smax );

[0041]

[0042] H i E represents i The result after the fully connected layer, E i This represents the result obtained after the attention mechanism is fused, FC represents a fully connected layer, and l i H represents i The result after the activation function is obtained; ReLU represents the activation function, W. l Weight matrix, b l P represents the offset parameter. i W represents the probability distribution calculated by the Softmax function, where Softmax represents the normalization exponential function. smax Let b represent the weight matrix. smax Let argmax represent the offset function, argmax represent the category with the highest probability, and k represent the sentiment category. The sentiment label indicates the prediction.

[0043] Compared with the prior art, the beneficial technical effects of the present invention are:

[0044] The challenge of multimodal emotion recognition lies in effectively utilizing the relationships between different modalities and uncovering the connections between them. Current methods based on graph neural networks or recurrent neural networks typically fuse the three modalities directly. While there are related information between the modalities, there is also some mutually interfering information. Furthermore, training may lead to mutual interference between the modalities, resulting in suboptimal performance. This invention introduces a two-stage attention-based method for feature fusion and training, which reduces interference between modalities during fusion and uncovers the potential connections between different modalities at a deeper level. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the overall model structure of the present invention;

[0046] Figure 2 This is a schematic diagram of the first-stage fusion module of the present invention. Detailed Implementation

[0047] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] The attention-based multimodal fusion and emotion recognition method in this invention designs a two-layer neural network to fuse features for emotion recognition; it can be divided into two stages: the complete model of the first and second stages is shown below. Figure 1 For details of one of the first-stage fusion modules, see [link to relevant documentation]. Figure 2 .

[0049] I. Preprocessing:

[0050] The original video, audio, and text were preprocessed: the original text features were extracted using TextCNN; the original acoustic features were extracted using the OpenSmile toolkit with IS10 configuration; and the visual facial expression features were extracted using DenseNet (DenseNet-Dense Convolutional Network) after pre-training on the Facial Expression Recognition Plus (FER+) corpus.

[0051] II. Integration and Training of Features in the First Stage

[0052] Three features were obtained after preprocessing. These are audio features, video features, and text features, respectively. The audio features, video features, and text features are then processed using an LSTM, and the results are... The speaker information is then embedded into the features, with the original speaker information represented by vector s. i This indicates that the speaker embeds S i The calculation method is as follows:

[0053]

[0054] Speaker embedding adds speaker information to the discourse.

[0055] Then, an attention mechanism is used to fuse them, with each utterance consisting of three nodes. This means using h′ i a , h′ i v , h′ i t Initialize the node, h′ i a , h′ i v , h′ i t Features derived from LSTM, where In the first stage, we use a multi-head attention mechanism to fuse features. For example, when processing audio and video features, we use video features as the key and audio features as both the query and value.

[0056]

[0057] d x d k / h, where h represents the number of attention heads, d k Q represents the number of dimensions of the key. a Represents Query, K v Key, K vT In this formula, T represents the matrix transpose, and V represents the value. For convenience, the formula can be abbreviated as:

[0058] f i va =ATT(h′) i a , h′ i v , h′ i a );

[0059] The video and audio fusion features f obtained through the attention mechanism i va The classification results (predicted labels) are obtained through a fully connected layer.

[0060] Select the video and audio fusion features f that yield better training results. va .

[0061] Then, using the same method, we obtain video features and text features, and text features and audio features to obtain high-dimensional video and text features f. vt High-dimensional features f of audio text at The formula is as follows:

[0062] f i vt =ATT(h′) i t , h′ i v , h′ i t );

[0063] f i at =ATT(h′) i t , h′ ia , h′ i t ).

[0064] The first stage yielded high-dimensional features after pairwise modality fusion, and incorporated prior knowledge through training. During training, classification cross-entropy and L2 regularization were used as the loss functions.

[0065]

[0066] Where N represents the number of dialogues, c(i) represents the number of utterances in dialogue i, and P i，j y is the probability distribution of the predicted sentiment label of utterance j in dialogue i. i，j λ is the sentiment label of utterance j in the real dialogue i, λ is the weight of L2 regularization, and θ is a trainable parameter.

[0067] III. Conduct the second phase of feature fusion and training:

[0068] The three high-dimensional features obtained in the previous step are used to construct temporal relationships using GRU:

[0069]

[0070]

[0071]

[0072] Then, speaker information is embedded into features, with the original speaker information used as vector s. i This indicates that the speaker embeds S i The calculation method is as follows:

[0073]

[0074] H′ is obtained after speaker embedding. t va H′ t vt H′ t at ;

[0075] Then, E is obtained by fusion using an attention mechanism. i :

[0076] E i =ATT(H′) t va H′ t vt H′ t at );

[0077] Where Key, Query, and Value are respectively

[0078] Finally, a fully connected layer and an MLP layer are used for sentiment classification to obtain the final classification result, i.e., the recognition result:

[0079] H i =FC(E i );

[0080] l i =RELU(W l H i +b l );

[0081] P i =Softmax(W smax l i +b smax );

[0082]

[0083] The same loss function is used during training as in the first stage. The innovation of this patent training is to use an attention mechanism for staged fusion and staged training. After the first stage fusion network is trained, high-dimensional features containing contextual information between pairs of modalities are obtained. These high-dimensional features will be used as input to the second stage fusion network. The training of the second stage network is completely independent of the training of the first stage network.

[0084] The key technical point of this invention is to use an attention mechanism to focus on the fusion of pairs of modalities. Through two stages of feature fusion and training, potential connections between and within modalities are discovered, and redundant information is filtered out in the first stage of training.

[0085] This method improves classification accuracy through a two-stage fusion and training process involving audio, video, and text features. In the first stage, audio and video are used to construct a time-series relationship using a Long Short-Term Memory (LSTM) network, and then fused using an attention mechanism. Video features serve as the key, and audio features serve as both the query and value. Training yields good results, and high-dimensional features are extracted from this result. The same method is then used to obtain high-dimensional features for video and text (video as the key, text as the query and value) and text and audio (audio as the key, text as the query and value). The first stage obtains high-dimensional features by fusing pairwise modalities and filters redundant information during training. In the second stage, the high-dimensional features extracted in the first stage are used through a Gate Recurrent Unit (GRU), and the three features are fused using an attention mechanism. The result is then passed through a fully connected layer (FC) and a multilayer perceptron (MLP) to obtain the predicted recognition result. This comprehensive feature fusion improves classification accuracy.

[0086] The accompanying image includes a bilingual (Chinese and English) explanation:

[0087] Audio Feature represents audio features, Vision Feature represents video features, and Text Feature represents text features;

[0088] VA Feature represents high-dimensional features of video and audio, AT Feature represents high-dimensional features of audio and text, and VT Feature represents high-dimensional features of video and text.

[0089] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0090] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal fusion and emotion recognition method based on an attention mechanism, comprising a preprocessing stage, a first stage, and a second stage, specifically including: Preprocessing stage: The original video, audio, and text are preprocessed to convert their contents into video features. Audio features Text features ; The first stage involves using a bidirectional long short-term memory network to transmit video and audio features to obtain the temporal relationship of the discourse; embedding speaker information and fusing video and audio features through a multi-head attention mechanism to obtain high-dimensional video and audio features. The video features serve as the key, while the audio features serve as both the query and the value. Video and text features are combined using a bidirectional long short-term memory network to obtain the temporal relationship of the discourse; speaker information is embedded, and video and text features are fused through a multi-head attention mechanism to obtain high-dimensional video-text features. The video features serve as the keys, while the text features serve as both the query and the value. Text and audio features are combined using a bidirectional long short-term memory network to obtain the temporal relationship of the discourse; speaker information is embedded, and text and audio features are fused through an attention mechanism to obtain high-dimensional features of the audio text. The audio features serve as the keys, while the text features serve as both the query and the value. The first phase specifically includes: Audio features Video features and text features The first audio features were obtained by using a bidirectional long short-term memory network. First video features and first text features Speaker information is embedded into features to obtain speaker embeddings. : ； Indicates speaker information, Indicates the sequence number of the dialogue. Indicates the learnable weight parameters. Indicates single-modal characteristics; Speaker embedding adds speaker information to the discourse; Each utterance is represented by the following three nodes: Node ,node ,node ;use Initialize the node, where the second audio feature Second video features Second text features ; High-dimensional video and audio features are obtained by fusing video and audio features. At this time, video features are used as keys, and audio features are used as both queries and values. A multi-head attention mechanism is used to fuse the features. ； express h represents the number of attention heads. Indicates the number of dimensions of the key. Represents Query, Representative key, In Represents matrix transpose. Representative value; Video and audio fusion features obtained through attention mechanism The classification result is obtained through a fully connected layer; The video and audio fusion feature corresponding to the best training classification result is selected as the final video and audio fusion feature. ; The second stage involves passing the high-dimensional features of video and audio, video and text, and audio and text through a gated recurrent network, fusing them using an attention mechanism, and then passing the results through a fully connected layer and a multilayer perceptron to obtain the recognition result. The second stage specifically includes: extracting high-dimensional features from video and audio. High-dimensional features of video text High-dimensional features of audio text Construct the timing relationships using gated cyclic networks respectively: ； ； ； , and These are respectively referred to as the first video / audio high-dimensional feature, the first video / text high-dimensional feature, and the first audio / text high-dimensional feature; speaker information is embedded into these features to obtain the speaker embedding. : ； Indicates speaker information, After speaker embedding, , These are respectively referred to as the second video audio high-dimensional features, the second video text high-dimensional features, and the second audio text high-dimensional features; Then, an attention mechanism is used to fuse the results, yielding the fused outcome. : ； This represents a multi-head attention mechanism. In the second phase of the multi-head attention mechanism, the key, query, and value are respectively... ; Finally, a fully connected layer and an MLP layer are used for sentiment classification to obtain the final recognition result: ； ； ； ； express The result after passing through the fully connected layer This indicates the result obtained after the attention mechanism is fused. Indicates a fully connected layer. express The result after activation function This represents the activation function. Weight matrix, Indicates the offset parameter. express The probability distribution calculated by the function Represents the normalized exponential function, Represents the weight matrix. Represents the offset function. This represents the category with the highest probability of being obtained. Indicates sentiment classification, The sentiment label indicates the prediction.

2. The multimodal fusion and emotion recognition method based on attention mechanism according to claim 1, characterized in that: During the preprocessing stage, TextCNN is used to extract text features, OpenSmile toolkit with IS10 configuration is used to extract audio features, and pre-trained dense connection network is used to extract video features.

3. The multimodal fusion and emotion recognition method based on attention mechanism according to claim 1, characterized in that: During the first stage of training, classification cross-entropy and L2 regularization were used as the loss functions: ； in The loss function for the first stage of training. Represents the number of dialogues. Representative Dialogue The number of words in the text Indicates dialogue The number of words in the text It is a predictive dialogue words in The probability distribution of sentiment labels, It's a real conversation. words in Emotional tags, It is an L2 regularization term The weight, These are trainable parameters.

Images