A multimodal emotion recognition method, system and storage medium based on text modality-guided attention fusion

Abstract

The present invention provides a multimodal emotion recognition method, system and storage medium based on text modality-guided attention fusion, which relates to the field of emotion recognition technology, and is to solve the problem that the existing methods cannot effectively utilize modal information and lack the deep expression of interactive information between modalities. It includes: step 1, respectively extracting text emotion features, speech emotion features, and visual emotion features, and aligning them according to time series; step 2, using a multimodal feature fusion module to perform feature fusion, and obtain the fusion features of text and speech and the fusion features of text and vision; the multimodal feature fusion module introduces a multi-head cross-modal attention mechanism to make the model pay attention to important emotional information between different modal data, and introduces a multi-head self-attention mechanism to make the model pay attention to different aspects of information at the same time; step 3, establish an emotion recognition model, use the fusion features and unimodal emotion features to perform emotion prediction, and obtain emotion classification.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method, system and storage medium based on text modality-guided attention fusion. Background Art

[0002] As an integral part of human speech and non-verbal communication, emotions are a key factor driving people's behavior and thoughts. At the same time, emotion recognition has important application value in the field of artificial intelligence, which can help us better understand human emotions and behaviors. It can also be applied to customer service, healthcare, education, and automotive driving intelligent systems to improve service quality and user experience.

[0003] Emotion recognition technology can automatically learn features through deep learning, reduce the need for manual feature engineering, and improve the accuracy and generalization ability of the model. Training is performed in an end-to-end manner to simplify the model development process. At the same time, deep learning has great advantages in extracting high-dimensional information, making emotion recognition technology based on deep learning a current hot topic with huge room for development. However, the factors of emotion recognition are diverse. If only visual information or language information is considered to identify emotional states, a large amount of relevant information will be lost, making the accuracy of emotion recognition low. Therefore, many factors need to be considered. However, due to the large differences in data of different categories of factors, it is necessary to integrate features of multiple categories, which greatly increases the difficulty of algorithm design. The huge amount of computation of deep network models also has many limitations on the actual application scenarios of the algorithm, and the diversity of emotional tasks also puts higher requirements on the transferability of the algorithm.

[0004] There are two types of deep learning models currently used in multimodal fusion: (1) Tensor-based methods: Tensor-based multimodal sentiment recognition methods construct tensor representations of multimodal sentences and use tensor decomposition or linear transformation methods to predict sentiment polarity. Tensor Fusion Network (TFN) calculates the Cartesian product of different modal vectors, which can effectively utilize the features of each specific modality and consider the feature fusion between different modalities. However, the Cartesian product will greatly increase the dimension of the multimodal feature vector, causing the model parameters to grow exponentially. (2) Attention-based methods: Attention-based methods aim to mine high-dimensional information within the modality and interactive information between modalities. In the extraction of single modal features, since the modal information at different times is time-dependent, that is, it is necessary to combine the contextual understanding of the sentence, a recurrent neural network such as LSTM is generally used to capture the contextual dependency information of a single modality. In mining the interactive information between modalities, many scholars use the attention mechanism method to mine the interactive information between different modalities. However, existing research methods in multimodal feature fusion mainly focus on feature fusion between different modalities, but cannot effectively utilize modal information and lack the deep expression of interactive information between modalities, thus ignoring its impact on emotion recognition results. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] Existing methods cannot effectively utilize modal information and lack deep expression of interactive information between modalities, ignoring its impact on emotion recognition results.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The present invention provides a multimodal emotion recognition method based on text modality guided attention fusion, comprising the following steps:

[0009] Step 1: Use three independent sub-network feature extraction modules to extract text emotion features from text data, speech emotion features from speech data, and visual emotion features from visual data, and align each feature according to the time series;

[0010] Step 2: According to the aligned features obtained in step 1, text + visual emotion features and text + speech emotion features are formed, and the text + visual emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and visual fusion features; the text + speech emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and speech fusion features; the multimodal feature fusion module introduces a multi-head cross-modal attention mechanism to make the model pay attention to important emotion information between different modal data, and introduces a multi-head self-attention mechanism to make the model pay attention to different aspects of information at the same time;

[0011] Step 3: Establish an emotion recognition model, use the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion features obtained in step 1 to perform emotion prediction and obtain emotion classification.

[0012] Furthermore, step 1 includes the following steps:

[0013] Step 1.1, extracting text sentiment information from text data, specifically includes: preprocessing the text data, building a BERT model and a Transformer model respectively, encoding the preprocessed text data through the BERT model to obtain a word vector representation of each word, using the Transformer model to contextually encode the text to obtain a contextual representation of each word, and obtaining the final text feature representation

[0014] Step 1.2, extracting speech emotion information from speech data, specifically includes: preprocessing the speech data, building a COVAREP framework, extracting sound features through the COVAREP framework, and obtaining speech emotion feature representation

[0015] Step 1.3, extracting visual emotion features from visual data, specifically includes: preprocessing the visual data, building a FACET model, identifying the feature points of facial expressions through the FACET model, encoding the expression of each frame of video, converting the facial expression into a corresponding feature vector, and obtaining a visual emotion feature representation.

[0016] Step 1.4. Finally, align each feature according to the time series to obtain the aligned modal sequence data. Use a bidirectional gated recurrent unit to capture the temporal relationship in the time series data, and convert all hidden vectors into vectors of the same length to obtain text, visual, and speech features with contextual information.

[0017] Furthermore, the preprocessing of the text data includes word segmentation and tokenization operations.

[0018] Furthermore, the preprocessing of the speech data includes denoising and segmentation operations.

[0019] Furthermore, the visual data is preprocessed, including format conversion and frame rate adjustment.

[0020] Furthermore, the multimodal feature fusion module in step 2 includes L layers of stacked attention fusion units AFU, text + speech emotion feature H T+A , text sentiment features H T and text + visual emotion feature H T+V , text sentiment features H T As the input of the first layer attention fusion unit AFU, the global context representation Global is generated through the corresponding AFU operation (1) :

[0021] Global (1) =Concat(AFU(H j , H T ))

[0022] Among them, j∈{T+A,T+V}, Concat(·) represents feature concatenation;

[0023] The global context generated by each layer of AFU is represented as Global (i) And text modal H T As the input of the next layer AFU:

[0024] Global (i) =Concat(AFU(Global (i-1) , H T ))

[0025] Where, i∈(2,…L);

[0026] Finally, the global context deep representation Global generated by multiple layers of AFU operations T+A and Global T+V , used for sentiment prediction.

[0027] Furthermore, the attention fusion unit AFU is a bilaterally symmetrical structure, and its two input features are represented as H m and H n , in order to obtain the mixed contextual representation of the two modalities, a unidirectional GRU is used to encode each modality feature and a Linear layer is added to its output to map the output of the GRU to another space for further processing and conversion representation;

[0028] Obtaining hidden representations at the sentence level through the Linear layer

[0029]

[0030] where i∈{m,n};

[0031] The matrix dot multiplication method is used to generate a fusion matrix containing the context information of the two modalities, and the similarity weight H of the fusion matrix is ​​extracted through the knowledge weight gate. M :

[0032]

[0033] Among them, σ represents the Sigmoid function;

[0034] Input mode H m and H n The feature representation formed by the multi-head cross-modal attention mechanism is CrossMH m and CrossMH n , using the similarity weight matrix to filter out the unique features of the text modality to supplement the interactive information of the cross-modal attention mechanism:

[0035]

[0036] Through the multi-head self-attention mechanism, the temporal relationship of each modality is further explored:

[0037] head i =Attention(QW i Q , KW i K , VW i V )

[0038] MHA=Concat(head1,...,head h )W o

[0039] Among them head i represents the features calculated by the i-th head, W i Q , W i K , W i V , W o represents the training parameters, h represents the number of longs;

[0040] The obtained complementary cross-modal interaction information is subjected to regularization, multi-head self-attention mechanism and feedforward neural network structure to generate feature representation H m→n and H n→m, the formula is as follows:

[0041]

[0042]

[0043] H m→n =FFN(LN(H m，n )+H m，n )

[0044] H n→m =FFN(LN(H n，m )+H n，m )

[0045] Among them, MHA represents the multi-head self-attention mechanism, FFN represents the feedforward neural network, and LN represents the normalization operation.

[0046] Further, step 3 includes the following steps:

[0047] Step 3.1, using the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion obtained in step 1 as input features of the emotion prediction model;

[0048] Step 3.2: Generate a self-attention matrix through the self-attention mechanism;

[0049] Step 3.3, calculate the probability of different sentiment classifications through the Softmax function;

[0050] y * =Softmax(f θ |G * )

[0051] Among them, G * is the generated attention matrix, f θ represents a fully connected layer with parameter θ, y * Label representing the sentiment classification prediction;

[0052] The cross entropy loss function is calculated for the probability distribution of different emotions and the true label vector of the data:

[0053]

[0054] Among them, y represents the true label of the sample;

[0055] At the same time, Adam optimizer is used to optimize the parameters of the network.

[0056] A multimodal emotion recognition system based on text modality-guided attention fusion, the system has a program module corresponding to the steps of any one of the above-mentioned technical solutions, and executes the steps in the above-mentioned multimodal emotion recognition method based on text modality-guided attention fusion when running.

[0057] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of the multimodal emotion recognition method based on text modality-guided attention fusion described in any one of the above technical solutions when called by a processor.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] (1) In order to enhance the emotional representation between text modality and other modalities, the present invention models the interaction between modalities and extracts the associated complementary information between text, visual modalities and text, speech modalities.

[0060] (2) A multimodal attention feature fusion network is proposed to learn the unique features of non-text modalities. At the same time, a multi-head cross-modal attention mechanism and a multi-head self-attention mechanism are introduced to model the differences in emotional contributions of different modalities and the consistency of emotional expression.

[0061] (3) The generated global multimodal high-level sentiment features are combined with the unimodal low-level features to achieve full integration of multimodal features; the sentiment prediction model introduces the self-attention mechanism to better capture the sentiment features in the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a multimodal emotion recognition method based on text modality guided attention fusion in an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of the overall structure of a multimodal emotion recognition model based on text modality-guided attention fusion in an embodiment of the present invention;

[0064] Figure 3 It is a schematic diagram of the structure of the attention fusion unit AFU in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In the description of the present invention, it should be noted that the terms "first", "second", and "third" mentioned in the embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0067] Specific implementation plan 1: Figure 1 As shown, the present invention provides a multimodal emotion recognition method based on text modality guided attention fusion, comprising the following steps:

[0068] Step 1: Use three independent sub-network feature extraction modules to extract text emotion features from text data, speech emotion features from speech data, and visual emotion features from visual data, and align each feature according to the time series;

[0069] Step 2: According to the aligned features obtained in step 1, text + visual emotion features and text + speech emotion features are formed, and the text + visual emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and visual fusion features; the text + speech emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and speech fusion features; the multimodal feature fusion module introduces a multi-head cross-modal attention mechanism to make the model pay attention to important emotion information between different modal data, and introduces a multi-head self-attention mechanism to make the model pay attention to different aspects of information at the same time;

[0070] Step 3: Establish an emotion recognition model, use the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion features obtained in step 1 to perform emotion prediction and obtain emotion classification.

[0071] Specific implementation plan 2: Step 1 includes the following steps:

[0072] Step 1.1, extracting text sentiment information from text data, specifically includes: preprocessing the text data, building a BERT model and a Transformer model respectively, encoding the preprocessed text data through the BERT model to obtain a word vector representation of each word, using the Transformer model to contextually encode the text to obtain a contextual representation of each word, and obtaining the final text feature representation

[0073] Step 1.2, extracting speech emotion information from speech data, specifically includes: preprocessing speech data, building a COVAREP framework, extracting sound features through the COVAREP framework, including sound intensity (sound intensity) and fundamental frequency (basic frequency of sound), etc., using signal processing technology, i.e., short-time Fourier transform to extract these features. Representing the extracted sound features in mathematical form for subsequent information fusion; obtaining speech emotion feature representation

[0074] Step 1.3, extracting visual emotion features from visual data, specifically includes: preprocessing visual data, building a FACET model, identifying feature points of facial expressions, such as the movement and changes of eyebrows, eyes, mouth and other parts through the FACET model; encoding the expression of each frame of video, and converting facial expressions into corresponding feature vectors. The FACET tool will output the facial expression and emotion features in each video. These features include information such as the intensity, frequency, duration of facial movements, and corresponding emotion labels. Finally, the visual emotion feature representation is obtained.

[0075] Step 1.4, finally align each feature according to the time series to obtain the aligned modal sequence data. Considering that the aligned text, vision and speech modalities are all time series data with time dependence, a bidirectional gated recurrent unit (BiGRU) is used in the sequence encoder (Sequence Encoder) to capture the temporal relationship in the time series data, and convert all hidden vectors into vectors of the same length to obtain text, vision and speech features with contextual information. The rest of this implementation is the same as the specific implementation plan one.

[0076] The bidirectional gated recurrent unit is an improved recurrent neural network (RNN) structure that consists of two independent GRU layers, one that processes the sequence in the forward direction and the other in the reverse direction.

[0077] For a feature sequence X = x1, x2, x3…x n , the bidirectional calculation of BiGRU can be expressed as:

[0078]

[0079]

[0080] in and denotes the hidden states from left to right and from right to left, respectively, GRU denotes the GRU unit, x t Represents the t-th element in the input sequence.

[0081] Then concatenate the hidden states in the two directions to get the final hidden state h t :

[0082]

[0083] Where [*;*] represents a concatenation operation.

[0084] Finally, the hidden state h t Pass it to a fully connected layer and get the output y t :

[0085] y t =softmax(Wh t +b)

[0086] Among them, W and b represent the weight and bias of the fully connected layer respectively, and softmax represents the softmax activation function.

[0087] The final text, visual and speech features with contextual information are as follows:

[0088]

[0089] Among them, T stands for text, A stands for voice, and V stands for vision.

[0090] Specific implementation scheme three: the preprocessing of text data includes word segmentation and tokenization operations. The rest of this implementation scheme is the same as the specific implementation scheme two.

[0091] Specific implementation scheme 4: The pre-processing of the speech data includes denoising and segmentation operations. The rest of this implementation scheme is the same as that of the specific implementation scheme 3.

[0092] Specific implementation scheme 5: The visual data is preprocessed, including format conversion and frame rate adjustment. The rest of this implementation scheme is the same as that of specific implementation scheme 4.

[0093] Specific implementation scheme six: The multimodal feature fusion module in step 2 includes L layers of stacked attention fusion units AFU (Attention Fusion Unit), text + speech emotion feature H T+A , text sentiment features H T and text + visual emotion feature H T+V , text sentiment features H T As the input of the first layer attention fusion unit AFU, the global context representation Global is generated through the corresponding AFU operation (1) :

[0094] Global (1) =Concat(AFU(H j , H T ))

[0095] Among them, j∈{T+A,T+V}, Concat(·) represents feature concatenation;

[0096] The global context generated by each layer of AFU is represented as Global (i) And text modal H T As the input of the next layer AFU:

[0097] Global (i) =Concat(AFU(Global (i-1) , H T ))

[0098] Where, i∈(2,…L);

[0099] Finally, the global context deep representation Global generated by multiple layers of AFU operations T+A and Global T+V , used for emotion prediction. The rest of this implementation is the same as the first implementation.

[0100] Specific implementation scheme 7: The attention fusion unit AFU is a bilaterally symmetrical structure, and its two input features are represented as H m and H n , in order to obtain the mixed contextual representation of the two modalities, a unidirectional GRU is used to encode each modality feature and a Linear layer is added to its output to map the output of the GRU to another space for further processing and conversion representation;

[0101] Obtaining hidden representations at the sentence level through the Linear layer

[0102]

[0103] Where i∈{m,n}.

[0104] In order to extract the correlation between different modalities and better utilize multimodal information for emotion recognition, the matrix dot multiplication method is used to generate a fusion matrix containing the contextual information of the two modalities, and the similarity weight H of the fusion matrix is ​​extracted through the knowledge weight gate. M :

[0105]

[0106] Among them, σ represents the Sigmoid function;

[0107] By introducing a multi-head cross-modal attention mechanism, we can better learn the interactive information between different modalities. Input modality H m and H n The feature representation formed by the multi-head cross-modal attention mechanism is CrossMH m and CrossMH n, using the similarity weight matrix to filter out the unique features of the text modality to supplement the interactive information of the cross-modal attention mechanism:

[0108]

[0109] By introducing a multi-head self-attention mechanism, the consistency of emotional expression is modeled and the model's attention to different modal information is dynamically adjusted to better adapt to the characteristics and requirements of different emotion recognition tasks.

[0110] The calculation formula of the self-attention mechanism is:

[0111]

[0112] where Q, K and V are the same mode, Indicates the scaling of features to avoid weights approaching 0 and 1 due to excessive dimensions.

[0113] The multi-head self-attention mechanism calculates a part of the features through multi-head processing, and then concatenates the features obtained by each head to further explore the temporal relationship of each modality:

[0114] head i =Attention(QW i Q , KW i K , VW i V )

[0115] MHA=Concat(head1,...,head h )W o

[0116] Among them head i represents the features calculated by the i-th head, W i Q , W i K , W i V , W o represents the training parameters, h represents the number of longs;

[0117] The obtained complementary cross-modal interaction information is subjected to regularization, multi-head self-attention mechanism and feedforward neural network structure to generate feature representation H m→n and H n→m , the formula is as follows:

[0118]

[0119]

[0120] H m→n =FFN(LN(H m，n )+H m，n )

[0121] H n→m =FFN(LN(H n，m )+H n，m )

[0122] Among them, MHA represents the multi-head self-attention mechanism, FFN represents the feedforward neural network, and LN represents the normalization operation.

[0123] Finally, we will get the output Global of AFU structure (i) As input to the lower AFU structure:

[0124]

[0125] The rest of this implementation plan is the same as the specific implementation plan six.

[0126] In this implementation, AFU aims to use the attention mechanism and weight gating mechanism to establish the connection between the modal data on the basis of multimodal representation learning to improve the contextual semantics. By using the weight gating mechanism, the interactive information of the current bimodal and textual modalities is extracted; through the cross-modal attention mechanism and multi-head self-attention mechanism, the features of the final fusion of the two modalities are obtained.

[0127] Specific implementation scheme eight: Step 3 includes the following steps:

[0128] Step 3.1, using the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion obtained in step 1 as input features of the emotion prediction model;

[0129] Step 3.2: Generate a self-attention matrix through the self-attention mechanism;

[0130] Step 3.3, calculate the probability of different sentiment classifications through the Softmax function;

[0131] y * =Softmax(f θ |G * )

[0132] Among them, G * is the generated attention matrix, f θ represents a fully connected layer with parameter θ, y * Label representing the sentiment classification prediction;

[0133] The cross entropy loss function is calculated for the probability distribution of different emotions and the true label vector of the data:

[0134]

[0135] Among them, y represents the true label of the sample;

[0136] At the same time, the Adam optimizer is used to optimize the parameters of the network. The rest of this implementation plan is the same as the specific implementation plan 1.

[0137] Specific implementation scheme 9: A multimodal emotion recognition system based on text modality guided attention fusion, comprising:

[0138] A feature extraction module is used to use three independent sub-network feature extraction modules to extract text emotion features from text data, extract speech emotion features from speech data, and extract visual emotion features from visual data, and align each feature according to the time series;

[0139] A feature fusion module is used to form text + visual emotion features and text + speech emotion features from the aligned features obtained by the feature extraction module, and input the text + visual emotion features and the text emotion features into the multimodal feature fusion module for feature fusion to obtain text and visual fusion features; input the text + speech emotion features and the text emotion features into the multimodal feature fusion module for feature fusion to obtain text and speech fusion features; the multimodal feature fusion module introduces a multi-head cross-modal attention mechanism to enable the model to focus on important emotion information between different modal data, and introduces a multi-head self-attention mechanism to enable the model to focus on different aspects of information at the same time;

[0140] The emotion prediction module is used to establish an emotion recognition model, and use the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion features obtained in step 1 to perform emotion prediction and obtain emotion classification.

[0141] Specific implementation scheme ten: A computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of the multimodal emotion recognition method based on text modality guided attention fusion described in any one of the above-mentioned specific implementation schemes when called by a processor.

[0142] Example 1

[0143] In order to verify the effect of the method of the present invention, the method of the present invention is described by using the following examples.

[0144] Based on the existing CMU-MOSI dataset and CMU-MOSEI dataset, comparative experiments were carried out using the method of the present invention and multiple models. The results are shown in Table 1.

[0145] Table 1

[0146]

[0147] Table 2

[0148]

[0149] It can be seen from the experimental results in Table 1 and Table 2 that the model of the present invention outperforms the baseline model in both ACC-2 and F1-Score, and can more accurately judge the sentiment polarity. Compared with the classic TFN and LMF models, ACC-2 and F1-Score on the CMU-MOSI dataset increased by 3.6% to 5.3% and 3.8% to 5.5%, respectively. Since the baseline model uses outer products, there is a large amount of redundant information when creating multidimensional tensors, and the single-modal sentiment information and the deep interaction of different modalities are not fully utilized, resulting in low model performance. The MULT, MISA and BBFN models all introduce attention mechanisms. Experiments have shown that the addition of attention mechanisms significantly improves the effect of sentiment classification tasks. This shows that the attention mechanism can help the model focus on information that is more useful for sentiment classification tasks and comprehensively consider the contribution of different modalities to the results.

[0150] Compared with the existing advanced BBFN model, the ACC-2 is significantly improved by 1.8% and the F1-Score value is improved by 1.9%. The results show that the fusion of single-modal low-level sentiment features and the deep and effective fusion of information from different modalities can effectively improve the performance of sentiment classification tasks.

[0151] The method of the present invention outperforms other comparison models in the three evaluation indicators of ACC-2, F1-Score and MAE on the CMU-MOSEI dataset, among which the accuracy of the two-class classification is improved by 0.3% to 4.0%, the F1-Score value is improved by 0.2% to 5.2%, and the MAE index is reduced by 0.03% to 0.067%. This shows that the method of the present invention can extract rich emotional information within the single modality and the dual modality, and focus on the modal information that is beneficial to the emotional classification through the attention mechanism.

[0152] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method based on text modality guided attention fusion, characterized in that: The steps include: Step 1: Use three independent sub-network feature extraction modules to extract text emotion features from text data, speech emotion features from speech data, and visual emotion features from visual data, and align each feature according to the time series; Step 2: According to the aligned features obtained in step 1, text + visual emotion features and text + speech emotion features are formed, and the text + visual emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and visual fusion features; the text + speech emotion features and text emotion features are input into the multimodal feature fusion module for feature fusion to obtain text and speech fusion features; the multimodal feature fusion module introduces a multi-head cross-modal attention mechanism to make the model pay attention to important emotion information between different modal data, and introduces a multi-head self-attention mechanism to make the model pay attention to different aspects of information at the same time; Step 3: Establish an emotion recognition model, use the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion features obtained in step 1 to perform emotion prediction and obtain emotion classification; The multimodal feature fusion module in step 2 includes L layers of stacked attention fusion units AFU, which combines text + speech emotion features H T+A and text sentiment features H T As the input of the first layer attention fusion unit AFU, and the text + visual emotion feature H T+V and text sentiment features H T As the input of the first layer attention fusion unit AFU, the global context representation Global is generated through the corresponding AFU operation (1) : Global (1) =Concat(AFU(H j ,H T )) Among them, j∈{T+A,T+V}, Concat(·) represents feature concatenation; The global context generated by each layer of AFU is represented as Global (i) And text modal H T As the input of the next layer AFU: Global (i) =Concat(AFU(Global (i-1) ,H T )) Where, i∈(2,…L); Finally, the global context deep representation Global generated by multiple layers of AFU operations T+A and Global T+V , for sentiment prediction; The attention fusion unit AFU is a bilaterally symmetrical structure, and its two input features are represented as H m and H n , in order to obtain the mixed contextual representation of the two modalities, a unidirectional GRU is used to encode each modality feature and a Linear layer is added to its output to map the output of the GRU to another space for further processing and conversion representation; Obtaining hidden representations at the sentence level through the Linear layer where i∈{m,n}; The matrix dot multiplication method is used to generate a fusion matrix containing the context information of the two modalities, and the similarity weight H of the fusion matrix is ​​extracted through the knowledge weight gate. M : Among them, σ represents the Sigmoid function; Input mode H m and H n The feature representation formed by the multi-head cross-modal attention mechanism is CrossMH m and CrossMH n , using the similarity weight matrix to filter out the unique features of the text modality to supplement the interactive information of the cross-modal attention mechanism: Through the multi-head self-attention mechanism, the temporal relationship of each modality is further explored: head i =Attention(QW i Q ,KW i K ,VW i V ) MHA=Concat(head1,…,head h )W o Among them head i represents the features calculated by the i-th head, W i Q ,Q i K ,W i V ,W o represents the training parameters, h represents the number of longs; The obtained complementary cross-modal interaction information is subjected to regularization, multi-head self-attention mechanism and feedforward neural network structure to generate feature representation H m→n and H n→m , the formula is as follows: H m→n =FFN(LN(H m,n ) + H m,n ) H n→m =FFN(LN(H n,m ) + H n,m ) Among them, MHA represents the multi-head self-attention mechanism, FFN represents the feedforward neural network, and LN represents the normalization operation.

2. The multimodal emotion recognition method based on text modality guided attention fusion according to claim 1 is characterized in that: Step 1 includes the following steps: Step 1.1, extracting text sentiment information from text data, specifically includes: preprocessing the text data, building a BERT model and a Transformer model respectively, encoding the preprocessed text data through the BERT model to obtain a word vector representation of each word, using the Transformer model to contextually encode the text to obtain a contextual representation of each word, and obtaining the final text feature representation Step 1.2, extracting speech emotion information from speech data, specifically includes: preprocessing the speech data, building a COVAREP framework, extracting sound features through the COVAREP framework, and obtaining speech emotion feature representation Step 1.3, extracting visual emotion features from visual data, specifically includes: preprocessing the visual data, building a FACET model, identifying feature points of facial expressions through the FACET model, encoding the expression of each frame of video, converting the facial expression into a corresponding feature vector, and obtaining a visual emotion feature representation. Step 1.

4. Finally, align each feature according to the time series to obtain the aligned modal sequence data. Use a bidirectional gated recurrent unit to capture the temporal relationship in the time series data, and convert all hidden vectors into vectors of the same length to obtain text, visual, and speech features with contextual information.

3. The multimodal emotion recognition method based on text modality guided attention fusion according to claim 2 is characterized in that: The preprocessing of text data includes word segmentation and tokenization operations.

4. The multimodal emotion recognition method based on text modality guided attention fusion according to claim 3 is characterized in that: The preprocessing of the speech data includes denoising and segmentation operations.

5. The multimodal emotion recognition method based on text modality guided attention fusion according to claim 4 is characterized in that: The preprocessing of visual data includes format conversion and frame rate adjustment.

6. The multimodal emotion recognition method based on text modality guided attention fusion according to claim 1 is characterized in that: Step 3 includes the following steps: Step 3.1, using the fusion features of text and vision, the fusion features of text and speech, and the unimodal emotion obtained in step 1 as input features of the emotion prediction model; Step 3.2: Generate a self-attention matrix through the self-attention mechanism; Step 3.3, calculate the probability of different sentiment classifications through the Softmax function; y * =Softmax(f θ |G * ) Among them, G * is the generated attention matrix, f θ represents a fully connected layer with parameter θ, y * Label representing the sentiment classification prediction; The cross entropy loss function is calculated for the probability distribution of different emotions and the true label vector of the data: Among them, y represents the true label of the sample; At the same time, Adam optimizer is used to optimize the parameters of the network.

7. A multimodal emotion recognition system based on text modality guided attention fusion, characterized in that: The system has a program module corresponding to the steps of any one of claims 1 to 6 above, and executes the steps in the above-mentioned multimodal emotion recognition method based on text modality-guided attention fusion when running.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps in the multimodal emotion recognition method based on text modality guided attention fusion according to any one of claims 1 to 6 when called by a processor.

Images