Text and expression bimodal fused emotion recognition method

Abstract

The invention belongs to the field of emotion recognition, and particularly relates to a text and expression bimodal fused emotion recognition method. Comprising the following steps: extracting a text emotion feature vector by using an ALBERT-BiLSTM model; extracting expression and emotion feature vectors by using a VGG16 model; fusing the text emotion feature vector and the expression feature vector based on a self-attention mechanism and a Transform model; and classifying the fused features by using a classifier to obtain sentiment classification. The emotion recognition accuracy can be improved by combining the text and the expression information. The text and the expression generally contain complementary information, and the emotion state of the elderly can be more comprehensively understood and captured by fusing the data of the two modes, so that the accuracy of emotion recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition, and in particular is an emotion recognition method integrating text and expression bimodalities. Background Art

[0002] Existing emotion recognition technologies for seniors can identify their emotions through their speech, conversational text, and facial expressions. Automatic speech emotion recognition uses computers to simulate the human brain's process of extracting and perceiving emotional information from speech signals. It first perceives and analyzes speech signals to extract acoustic features related to emotional expression. Then, based on certain characteristic criteria, it effectively and reasonably classifies speech emotions, ultimately identifying the speaker's emotional state. However, due to factors such as unclear pronunciation, slow speech rates, and variability in speech characteristics, speech signal recognition accuracy is low, leading to inaccurate emotion recognition. Text emotion recognition, a subfield of natural language processing, focuses on extracting and understanding emotional information from text. It collects and preprocesses data, extracts features into word vectors, and then feeds them into machine learning or deep learning models to output emotion classification. However, in the process of emotion recognition from seniors' conversational text, the possibility of seniors expressing emotions in informal, ambiguous, or complex language increases uncertainty. Some expressions may not accurately capture emotions through text. Facial expression recognition is a computer vision and pattern recognition technology that aims to understand and infer an individual's emotional state by analyzing images or videos of human facial expressions. However, as people age, their facial features may change, including sagging skin and wrinkles, which may affect the accuracy of facial expression recognition.

[0003] Researchers have been studying emotion recognition since the 20th century. However, at the time, research on emotion classification focused primarily on emotion recognition in a single modality (such as speech or text). However, the emotional information contained in a single modality is susceptible to external factors and is incomplete. For example, speech emotion recognition is susceptible to differences in individual voices and interference from environmental noise. Text emotional features are difficult to extract, and text is difficult to normalize. Stop words that lack emotional connotations can also affect emotion determination. Therefore, single-modal emotion recognition suffers from numerous shortcomings, such as poor recognition robustness and low accuracy. Therefore, research on emotion recognition based on multimodal data is necessary. Data from different modalities, such as expressions, text, and speech, contain different representational information. Multimodal emotion recognition requires the fusion of information from different modalities to achieve synergy between the various modalities.

[0004] For multimodal emotion recognition, different fusion methods have varying impacts on the final results. Fusion methods can be categorized into three types: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion involves fusing data from different modalities in some way and then applying the resulting data to emotion recognition. However, due to the difficulty in effectively integrating data from different modalities, the resulting data often contains excessive redundant information, resulting in poor performance for emotion recognition. Feature-level fusion involves extracting features from each modality and fusing them using relevant algorithms to form a new feature. This new, aggregated feature is then input into a classifier for emotion recognition. When fusing features from different modalities using feature-level fusion, information from different modalities can be fully integrated, supplementing any missing features. The most convenient approach is to concatenate the feature vectors from each modality to create a single, high-dimensional feature vector. However, direct concatenation can result in a very high-dimensional emotion feature, significantly increasing the computational complexity. Combining features from different modalities can increase the feature dimensionality, leading to a dimensionality curse. Decision-layer fusion refers to extracting the classification results of each single modality and then processing these classification results. The decision-layer fusion method fully considers the different importance of each modality for emotion recognition classification, but it is also more likely to lose the correlation information between different modalities. Summary of the Invention

[0005] In order to solve the problem of insufficient single-modal data emotion information in emotion recognition of the elderly and improve the accuracy and stability of emotion recognition of the elderly, this paper proposes an emotion recognition method that integrates text and expression. By integrating text and expression to comprehensively identify the emotions of the elderly, the accurate recognition of the emotions of the elderly can be achieved.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A method for emotion recognition integrating text and expression bimodality includes the following steps:

[0008] 1) Use the ALBERT-BiLSTM model to extract text sentiment feature vectors;

[0009] 2) Use the VGG16 model to extract facial expression feature vectors;

[0010] 3) Fusion of text emotion feature vectors and expression feature vectors based on the Transformer model;

[0011] 4) Use the classifier to classify the fused features to obtain sentiment classification.

[0012] The step 1) comprises the following steps:

[0013] 1.1) Use the ALBERT model to dynamically represent the word vector of each word in the text data;

[0014] 1.2) Use the BiLSTM model to capture the contextual information in the word vector and obtain the text sentiment feature vector.

[0015] Described step 2) comprises the following steps:

[0016] 2.1) Preprocessing the expression video to obtain multiple video frames;

[0017] 2.2) Use the VGG16 model for feature extraction to obtain a 256-dimensional feature vector;

[0018] 2.3) Detect facial key points on each video frame, calculate the center point, and calculate the distance between the center point and the key points to obtain a 68-dimensional feature vector;

[0019] 2.4) Combine the 256-dimensional feature vector and the 68-dimensional feature vector to obtain the expression emotion feature vector.

[0020] The step 3) comprises the following steps:

[0021] 3.1) Using text sentiment feature vector V txt As the query vector Q, the expression emotion feature vector V emo As key vector K and value vector V, perform cross query:

[0022] Q=V txt W txt Q

[0023] K=V emo W emo K ,V=V emo W emo V

[0024] Among them, W txt Q , W emo K , W emo V Both are learnable weight matrices;

[0025] 3.2) Calculate the cross attention score Attention and multiply it with the value vector V to obtain the fusion vector Output that combines the attention information of the text to the expression:

[0026]

[0027] Output=Attention·V

[0028] Among them, d k is the dimension of the key vector;

[0029] 3.3) The fused vector is passed into the Transformer model and the multi-head self-attention mechanism is used to classify the sentiment.

[0030] The step 3.3) comprises the following steps:

[0031] 3.3.1) Introduce a position encoding mechanism before transformer encoding to encode the query vector Q;

[0032] 3.3.2) In the Transformer encoding layer, the multi-head self-attention mechanism uses multiple query vectors Q = [q1,…,q m ] Calculate multiple emotional information in the input information in parallel, and then obtain different emotional feature information, splice each self-attention head, and obtain the feature vector through the feedforward neural network.

[0033] The encoding of the query vector Q is specifically as follows:

[0034]

[0035]

[0036] Among them, PE(·) represents position encoding, p represents the position information of the encoding vector, i represents the position information of the input vector, and d model Represents the dimension information of the position vector.

[0037] The self-attention heads are spliced ​​together as follows:

[0038] Multihead(Q,K,V)=Concat(head1,…,head i )W o

[0039] head i =Attention(QW i Q ,KW i K ,VW i V )

[0040] in, They are the weight matrices of the linear transformations Q, K, and V, R represents a set of real numbers, and W oRepresents the weight matrix of the multi-head self-attention mechanism, h represents the number of self-attention heads, and Concat represents the concatenation of h self-attention heads. i Perform splicing.

[0041] The feature vector is obtained by the feedforward neural network, specifically:

[0042] FFN(x)=max(0,ZW1+b1)W2+b2

[0043] Among them, FFN is a feedforward neural network, Z is the weight matrix, W1 and W2 are the weight matrices of the feedforward layer, and b1 and b2 are the biases of the feedforward layer.

[0044] An emotion recognition system that integrates text and expression bimodalities, including:

[0045] The text sentiment feature vector extraction module is used to extract text sentiment feature vectors using the ALBERT-BiLSTM model;

[0046] Expression and emotion feature vector extraction module, used to extract expression and emotion feature vectors using the VGG16 model;

[0047] Vector fusion module, used to fuse text emotion feature vectors and expression feature vectors based on the self-attention mechanism and Transformer model;

[0048] The feature classification module is used to classify the fused features using a classifier to obtain sentiment classification.

[0049] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for emotion recognition that integrates text and expression dual modalities is implemented.

[0050] The present invention has the following beneficial effects and advantages:

[0051] This invention combines text and facial expressions to improve the accuracy of emotion recognition. Text and facial expressions often contain complementary information. By fusing data from these two modalities, we can more comprehensively understand and capture the emotional state of the elderly, thereby improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 ALBERT-BiLSTM text sentiment feature extraction model diagram;

[0053] Figure 2 Flowchart of facial expression and emotion feature extraction;

[0054] Figure 3Flowchart of multimodal emotion recognition based on feature layer fusion of CAT-SA model;

[0055] Figure 4 Transformer module diagram. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] The present invention addresses the problem of inaccurate emotion recognition caused by insufficient emotional information in single-modal data. It fuses data from two modalities, image and text, to extract emotional features. Since the feature-level fusion method can fully fuse information from different modalities and supplement the feature information lacking in different modalities, it can simultaneously play the role of both expression and text modalities, comprehensively analyze the correlation between different modalities, and fuse the two different modalities of expression and text using feature-level fusion, thereby comprehensively identifying the emotions of the elderly.

[0058] The present invention adopts the cross-attention mechanism to fuse data of expression and text modalities to achieve accurate recognition of the emotions of the elderly.

[0059] For text sentiment feature extraction, we use the ALBERT-BiLSTM model. The specific model structure is shown in the attached figure. Figure 1 As shown in the figure, the ALBERT model is first used to represent word vectors, which are then processed into word vectors that can be directly used by the BiLSTM model. The word vector of each word is dynamically represented to solve the problem of polysemy. The word vector is then input into the BiLSTM model. BiLSTM captures the contextual information in the text by simultaneously processing forward and reverse information at each time step, and finally obtains the emotional characteristics of the text.

[0060] The expression feature extraction block diagram is as follows Figure 2 As shown in the figure, after preprocessing operations such as frame cutting on the expression video, the video frame is read, and then the VGG16 pre-trained model is used for feature extraction. The features are expanded and sent to the fully connected layer to obtain a 256-dimensional feature vector. In addition, the facial key points of the framed image are detected, and the center point is calculated to calculate the distance between the center point and the key point to obtain a 68-dimensional feature vector. Finally, the two parts of the feature are combined to form a complete expression emotion feature.

[0061] Finally, feature layer fusion is performed. Concatenating feature vectors involves concatenating the features of each modality dimensionally to form a larger feature vector. Direct concatenation cannot fully integrate the sentiment information of the two modalities. Instead, a cross-query self-attention mechanism is used for fusion. The fused vector is processed by the Transformer, enabling the dynamic allocation of attention weights to each position in the input. Unlike traditional methods that focus only on local regions, the Transformer comprehensively considers information at every position in the input sequence, which is crucial for understanding global image-text relationships in multimodal data. Furthermore, its adaptive learning capabilities enable the model to identify and adjust key information in the input data based on task requirements, effectively addressing the heterogeneity and dynamic changes in multimodal data. The multi-head self-attention mechanism further enhances the model's ability to process multidimensional input features, enabling a deeper understanding of complex information. The parallel processing structure of the Transformer model not only optimizes hardware utilization but also improves training and inference efficiency. Therefore, using the Transformer model for subsequent processing of emotion vectors can provide more accurate weight distribution for the emotion vectors of the fusion of expression and text modalities, effectively solving the instability of emotion classification problems that rely on a single modality, and demonstrating its academic value and application potential in the field of multimodal data processing.

[0062] Example

[0063] like Figure 3 As shown in the figure, the emotion recognition results of text and expression are fused. The feature layer fusion process is as follows:

[0064] Step 1: Based on the multimodal dataset, each modal data is fed into the corresponding feature extraction model to obtain the emotional feature vector of each modality;

[0065] Step 2: We combine the characteristics of the self-attention mechanism to achieve interaction and fusion between text vectors and image vectors. This method allows the model to dynamically focus on the most relevant information in one modality for the other modality, thereby more effectively integrating cross-modal information. The following is a simplified description of the process, showing how to implement self-attention fusion for cross-query of text vectors to expression vectors (or vice versa). In the first step, we extracted the feature vectors of the two modalities and obtained the feature vector V txt and V emo Through cross-query, one modality can "pay attention" to another modality. You can use text vectors as queries (Q), and expression vectors as keys (K) and values ​​(V). The specific calculation process is as follows: 1. Prepare Q, K, V:

[0066] Q=V txt W txt Q(1)

[0067] K=V emo W emo K ,V=V emo W emo V (2)

[0068] Where W txt Q , W emo K , W emo V is a learnable weight matrix 2. Calculate the cross attention score:

[0069] Use Q, K, and V to calculate attention scores and weights to determine how each element in the text should be enhanced by image information:

[0070]

[0071] Output=Attention·V (4)

[0072] The final output is a new vector that combines the attention information of text to expression.

[0073] Step 3: We obtain a feature vector that combines text and expressions through cross-query. We finally pass the vector into the Transformer model and use the multi-head self-attention mechanism to classify emotions.

[0074] The internal diagram of a single Transformer layer is shown in the attached figure. Figure 4 As shown in the figure, the Transformer encoding layer mainly consists of 6 stacked Transformer modules. Each Transformer module mainly consists of four parts: multi-head attention mechanism, residual connection, layer normalization, and feedforward neural network. In order for the model to fully utilize the position information of the sequence, we introduced a position encoding mechanism before the Transformer encoding. Figure 4 As shown in the figure, in the position encoding mechanism, the relative position and absolute position information of the sequence are first added to the model. Since the feature dimensions of the position encoding information and the embedding layer are the same, the position encoding information and the embedding layer of the encoder are directly added. The position encoding is further implemented by using sine and cosine functions of different frequencies. The calculation formula is as follows:

[0075]

[0076]

[0077] In the above formula, p represents the position information of the encoding vector, i represents the position information of the input vector, and d model Represents the dimension information of the position vector.

[0078] In the Transformer encoding layer, the multi-head self-attention mechanism uses multiple query vectors Q = [q1,…,q m ] Calculate multiple emotional information in the input information in parallel. Each self-attention head pays attention to different positions of the input emotional information, thereby obtaining different emotional feature information. Finally, each self-attention head is spliced ​​together. The calculation formula is as follows:

[0079] Multihead(Q,K,V)=Concat(head1,…,head i )W o (7)

[0080] head i =Attention(QW i Q ,KW i K ,VW i V ) (8)

[0081] Where Q, K, and V represent the query vector, key vector, and value vector generated by the encoder’s input vector, respectively. are the weight matrices of the linear transformations of Q, K, and V, respectively, and W o Represents the weight matrix of the multi-head self-attention mechanism, h represents the number of self-attention heads, and the purpose of Concat is to concatenate h self-attention heads and finally obtain the feature vector through the feedforward neural network. The specific formula is:

[0082] FFN(x)=max(0,ZW1+b1)W2+b2 (9)

[0083] The multi-head attention layer inside the Transformer allows the model to focus on important information at different positions in the input sequence, while the feedforward neural network layer performs nonlinear transformation and feature extraction on the hidden representation of each position.

[0084] Step 4: Finally, the output representation is passed to the Softmax layer to output the final sentiment classification.

[0085] The adaptive weighted decision fusion rule introduced in the present invention can analyze the emotion categories of the elderly more objectively and accurately, solving the problem of inaccurate emotion recognition caused by insufficient emotion information in single-modal data.

Claims

1. A dual-modality emotion recognition method integrating text and expression, characterized in that: The following steps are involved: 1) Use the ALBERT-BiLSTM model to extract text sentiment feature vectors; 2) Use the VGG16 model to extract facial expression feature vectors; 3) Fusion of text emotion feature vectors and expression feature vectors based on the Transformer model; 4) Use the classifier to classify the fused features to obtain sentiment classification.

2. The emotion recognition method integrating text and expression according to claim 1, characterized in that: The step 1) comprises the following steps: 1.1) Use the ALBERT model to dynamically represent the word vector of each word in the text data; 1.2) Use the BiLSTM model to capture the contextual information in the word vector and obtain the text sentiment feature vector.

3. The emotion recognition method integrating text and expression according to claim 1, characterized in that: The step 2) comprises the following steps: 2.1) Preprocessing the expression video to obtain multiple video frames; 2.2) Use the VGG16 model for feature extraction to obtain a 256-dimensional feature vector; 2.3) Detect facial key points on each video frame, calculate the center point, and calculate the distance between the center point and the key points to obtain a 68-dimensional feature vector; 2.4) Combine the 256-dimensional feature vector and the 68-dimensional feature vector to obtain the expression emotion feature vector.

4. The emotion recognition method integrating text and expression according to claim 1, characterized in that: The step 3) comprises the following steps: 3.1) Using text sentiment feature vector V txt As the query vector Q, the expression emotion feature vector V emo As key vector K and value vector V, perform cross query: Q=V txt W txt Q K=V emo W emo K ,V=V emo W emo V Among them, W txt Q , W emo K , W emo V Both are learnable weight matrices; 3.2) Calculate the cross attention score Attention and multiply it with the value vector V to obtain the fusion vector Output that combines the attention information of the text to the expression: Output=Attention·V Among them, d k is the dimension of the key vector; 3.3) The fused vector is passed into the Transformer model and the multi-head self-attention mechanism is used to classify the sentiment.

5. The emotion recognition method integrating text and expression according to claim 4, characterized in that: The step 3.3) comprises the following steps: 3.3.1) Introduce a position encoding mechanism before transformer encoding to encode the query vector Q; 3.3.2) In the Transformer encoding layer, the multi-head self-attention mechanism uses multiple query vectors Q = [q1, ..., q m ] Calculate multiple emotional information in the input information in parallel, and then obtain different emotional feature information, splice each self-attention head, and obtain the feature vector through the feedforward neural network.

6. The emotion recognition method integrating text and expression according to claim 5, characterized in that: The encoding of the query vector Q is specifically as follows: Among them, PE(.) represents position encoding, p represents the position information of the encoding vector, i represents the position information of the input vector, d model Represents the dimension information of the position vector.

7. The emotion recognition method integrating text and expression according to claim 5, characterized in that: The above steps of splicing the self-attention heads are as follows: Multihead(Q,K,V)=Concat(head1,...,head i )W o head i =Attention(QW i Q ,KW i K ,VW i V ) in, They are the weight matrices of the linear transformations Q, K, and V, R represents a set of real numbers, and W o Represents the weight matrix of the multi-head self-attention mechanism, h represents the number of self-attention heads, and Concat represents the concatenation of h self-attention heads. i Perform splicing.

8. The emotion recognition method integrating text and expression according to claim 5, characterized in that: The feature vector is obtained by the feedforward neural network, specifically: FFN(x)=max(0,ZW1+b1)W2+b2 Among them, FFN is a feedforward neural network, Z is the weight matrix, W1 and W2 are the weight matrices of the feedforward layer, and b1 and b2 are the biases of the feedforward layer.

9. An emotion recognition system integrating text and expression, characterized in that: include: The text sentiment feature vector extraction module is used to extract text sentiment feature vectors using the ALBERT-BiLSTM model; Expression and emotion feature vector extraction module, used to extract expression and emotion feature vectors using the VGG16 model; Vector fusion module, used to fuse text emotion feature vectors and expression feature vectors based on the self-attention mechanism and Transformer model; The feature classification module is used to classify the fused features using a classifier to obtain sentiment classification.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the emotion recognition method integrating text and expression bimodality as described in any one of claims 1 to 8 is implemented.

Images