A multimodal conversation emotion recognition method

Abstract

The present invention discloses a multimodal dialogue emotion recognition method, which relates to the field of multimodal emotion recognition and human-computer interaction technology, including: using a face recognition model and a posture recognition tool to obtain expression time series features and posture time series features respectively; an attention module performs adaptive weighted fusion on expression and posture features to obtain fused visual modal features; constructs a new expression of contextual information and obtains emotion representation based on the prompt emotion modeling technology and extracts text modal features through a text encoder; uses a data vectorization model to extract the modal features of the speaker's corresponding speech; proposes a jump connection multi-head attention cross-modal fusion method, performs cross-modal alignment and fusion on the modal features of multimodal, and then performs emotion recognition through an emotion classifier module. The present invention effectively solves the problems of insufficient recognition and insufficient fusion of key emotional clues in traditional multimodal emotion recognition, and improves the accuracy and robustness of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal emotion recognition and human-computer interaction, and in particular to a multimodal dialogue emotion recognition method. Background Art

[0002] Emotion recognition can automatically perceive, identify, understand and provide feedback on human emotions, involving the intersection of computer science, neuroscience, psychology and social science. It is essential to achieve natural, human-like and personalized human-computer interaction, and is of great value for the advent of the intelligent and digital era. Emotion recognition has a wide range of applications, including smart assistants, virtual reality, smart healthcare, etc. Traditional emotion recognition methods mainly rely on data sources of a single modality (such as voice, text or expression), which is difficult to fully reflect the complexity of human emotions.

[0003] In practical applications, emotions are often conveyed through a variety of signals, such as facial expressions, body postures, and voice intonation. Each modality complements each other and together conveys a complete emotional state. Existing conversational emotion recognition methods face the following challenges: (1) They rely on a single source of emotional information and ignore the multimodal characteristics of emotional communication, resulting in low recognition accuracy; (2) Many multimodal methods cannot accurately integrate information such as expressions and postures into a unified emotional output; (3) In multi-round conversation scenarios involving multiple people, due to the large number of participants and complex interactions, the importance of different emotional cues at different times varies. Traditional multimodal emotion recognition methods lack a flexible attention mechanism, making it difficult to highlight key emotional cues, which in turn affects the recognition effect. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal dialogue emotion recognition method, which generates visual modal features by combining expression features and posture features, performs cross-modal alignment and fusion of visual modal features, text modal features and speech modal features, and inputs the fused features into an emotion classifier module for emotion recognition. The present invention effectively solves the problems of insufficient recognition and insufficient fusion of key emotional clues in traditional multimodal emotion recognition, and improves the accuracy and robustness of emotion recognition. The present invention is achieved through the following technical solutions.

[0005] The present invention provides a multimodal dialogue emotion recognition method, comprising:

[0006] Construct a multimodal interactive dialogue list U;

[0007] Based on the face recognition model SFace and gesture recognition tool Mediapipe in the multimodal interactive dialogue list A video clip from The expression temporal features are obtained from and posture timing characteristics ;

[0008] Expression temporal features based on face posture attention module FPA and posture timing characteristics Fusion is performed to obtain the final visual modality features ;

[0009] Constructing new expressions of contextual information And the emotional representation is obtained through the suggested emotional modeling technology PEMT , expressing emotions Input to the text encoder SimCSE to obtain text modality features ;

[0010] According to the multimodal interactive dialogue list Every word in Corresponding audio clip Perform equal-interval sampling, input the sampled data into the data vectorization model data2vec to extract speech features, and aggregate the speech features of all frames to obtain speech modal features. ;

[0011] Visual modality features , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output ;

[0012] Cross-modal fusion attention output Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category .

[0013] In practical applications, since the existing conversational emotion recognition methods have a single source of emotion information and cannot accurately integrate information such as expressions and postures into a unified emotion output, the present invention solves this problem by using corresponding models to extract expression timing features and posture timing features and fuse them into visual modal features. In the process of fusing expression timing features and posture timing features, the model is able to focus on more important emotion timing information and enhance feature fusion by using multiple attention mechanisms. This solves the problem that traditional multimodal emotion recognition methods lack flexible attention mechanisms, making it difficult to highlight key emotion clues, thereby affecting recognition effects.

[0014] The present invention realizes the multi-dimensional expression and in-depth understanding of emotional information by cross-modal alignment and fusion of visual modal features, textual modal features and speech modal features. Textual modal features can capture emotional clues in language content, such as word selection, sentence structure and implicit semantics, providing rich contextual information for emotion recognition; speech modal features reflect the dynamic changes of emotions through acoustic parameters such as intonation, speaking speed and volume, making up for the details that are difficult to capture with simple text analysis. The integration of the three not only enhances the comprehensiveness of emotion recognition, but also significantly improves the robustness and accuracy of the system.

[0015] In addition, the predicted emotion probability distribution finally generated by the present invention can improve the interpretability and transparency of the emotion recognition system. Users can observe the judgment confidence of the emotion classification model in each category, which helps to provide support for decision-making. In practical applications, especially in the fields of medical care, education and psychological counseling, this probability information can help professionals better understand the emotional state of the person being analyzed and support more targeted interventions or suggestions.

[0016] Optionally, the multimodal interactive dialogue list is constructed It includes: collecting multimodal dialogue data of multiple rounds of dialogue involving multiple participants, preprocessing the multimodal dialogue data, and finally constructing a multimodal interactive dialogue list. ,

[0017] The multimodal interactive dialogue list Includes multiple words , where each utterance Contains a text record , a video clip and a voice clip ,

[0018] The multimodal interactive dialogue list The expression is: ,

[0019] Each of the above words The expression is: ,

[0020] in, Indicates the sequence number of the speech, also known as the Session units, the value range is - , Represents the entire multimodal interactive dialogue list The total number of utterances in The corresponding speaker is .

[0021] Optionally, the face recognition model SFace and the gesture recognition tool Mediapipe are used in the multimodal interactive dialogue list. A video clip from The expression temporal features are obtained from and posture timing characteristics include:

[0022] A video clip is detected by face recognition model SFace Recognize the face in the image, extract the face area, and form a sequence containing the face image , ,in Represents a sequence of face images A face image in one frame;

[0023] In the same video clip, the gesture recognition tool Mediapipe Human body posture recognition is performed in the image processing, and the human body area is extracted to form a sequence containing posture images. ,in Representing posture image sequence The posture image of one frame;

[0024] The face image sequence and pose image sequences Input into the spatiotemporal feature extraction model Timesformer to extract the expression temporal features and posture timing characteristics , used to capture facial expression dynamics and posture changes, the expression timing features and posture timing characteristics The expressions are as follows:

[0025] ,

[0026] .

[0027] Optionally, the facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics The integration includes:

[0028] Temporal features of facial expressions and posture timing characteristics Encoding is performed to obtain rich temporal information. The encoded expression temporal features and posture temporal features are obtained through MLP nonlinear transformation:

[0029] ,

[0030] ,

[0031] in, and Respectively represent the encoded expression temporal features and posture temporal features;

[0032] The encoded expression temporal features and posture timing characteristics The expression attention coefficient is obtained by adaptive weighting through the following formula: and posture attention coefficient :

[0033] ,

[0034] ,

[0035] in, is the encoded expression temporal feature The weight matrix of is the encoded posture temporal feature The weight matrix of is the encoded expression temporal feature The bias vector of is the encoded posture temporal feature The bias vector of

[0036] In order to improve the emotional representation ability of expressions and gestures, a multi-head attention mechanism is used to respectively and posture timing characteristics Enhance the emotional representation and the encoded expression temporal features and posture timing characteristics The multi-head attention calculation is obtained by the following formulas: and :

[0037] ,

[0038] ,

[0039] in, and They represent the weight coefficients of the expression timing features and posture timing features obtained after calculation by the multi-head attention mechanism, , , , and are all learnable parameters, , , , and are all learnable parameters. and posture timing characteristics By performing multi-head attention calculations separately, in practical applications, the multimodal dialogue emotion recognition model can adaptively focus on more important temporal information about emotions.

[0040] Optionally, the facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics Integration also includes:

[0041] The encoded posture temporal features As query Q, the encoded expression temporal features As key K and value V, the enhanced expression timing feature is obtained by the following formula :

[0042] ,

[0043] The encoded expression temporal features As query Q, the encoded posture temporal features As the key K and value V, the enhanced posture timing feature is obtained by the following formula :

[0044] .

[0045] The above uses the cross attention mechanism to realize the encoded expression temporal features and posture timing characteristics The information interaction between them can further enhance feature fusion.

[0046] The present invention uses the face posture attention module FPA to fuse the expression timing features and the posture timing features. The face posture attention module FPA includes a feature encoding layer, an attention generation layer, an attention calibration layer, a cross attention layer and a fusion layer connected in sequence. First, the expression timing features are combined Posture timing characteristics Input feature encoding layer to get encoded expression temporal features and posture timing characteristics Secondly, the encoded expression temporal features and posture timing characteristics Input the attention generation layer to get the expression attention coefficient and posture attention coefficient Next, the encoded expression temporal features and posture timing characteristics Input the attention calibration layer to obtain the expression timing features and posture timing characteristics The weight coefficient and Then, the weight coefficient and Input the cross attention layer to obtain enhanced expression temporal features and enhanced posture timing features Finally, the expression attention coefficient , Posture Attention Coefficient , Enhanced expression timing features and enhanced posture timing features At the same time, the fusion layer is input to fuse the expression and posture features to obtain the final visual modality features. The specific steps are as follows:

[0047] Optionally, the facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics Fusion is performed to obtain the final visual modality features include:

[0048] Enhanced expression timing features and posture timing characteristics The final visual modality feature is obtained by fusion in a weighted manner through the following formula: :

[0049] .

[0050] Optionally, the constructing of a new expression of context information And the emotional representation is obtained through the suggested emotional modeling technology PEMT , expressing emotions Input to the text encoder SimCSE to obtain text modality features include:

[0051] Context modeling: extracting the current utterance The text record of the three previous utterances aims to enhance the text record of the current utterance by taking into account the historical utterance information The semantic understanding of emotion recognition provides richer context information and constructs new expressions containing context information. , the new expression containing contextual information The expression is as follows:

[0052] ,

[0053] Text feature extraction: new expressions using contextual information , Speaker and text records , proposed a prompt-based emotion modeling technique PEMT to capture the speaker Long-distance dependencies between the utterances and the emotions expressed , the emotion expressed The expression is as follows:

[0054] ,

[0055] The sentiment representation is obtained by using the text encoder SimCSE Special Tokens <mask>Encode the representation to obtain the text modality feature , providing key information of text modality for multimodal emotion recognition. The text modality feature The expression is as follows:

[0056] .

[0057] Optionally, the multimodal interactive dialogue list Every word in Corresponding audio clip Perform equal-interval sampling, input the sampled data into the data vectorization model data2vec to extract speech features, and aggregate the speech features of all frames to obtain speech modal features. include:

[0058] Speech signal segmentation: Given each utterance Corresponding audio clip , continuous speech segments Split into The frame is used as voice data, and the voice data expression is as follows:

[0059] ,

[0060] in, Indicates Voice Clips No. Frame of voice data, The value range is 1- , is the total number of frames,

[0061] Speech feature extraction: Each speech data Input into the data vectorization model data2vec to obtain speech feature representation , the speech feature representation The expression is as follows:

[0062] ,

[0063] Aggregate all frame features: Use average pooling to represent all speech features Aggregation into discourse The speech modality characteristics of , the speech modality feature Calculated by the following formula:

[0064] .

[0065] Optionally, the visual modality feature , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output include:

[0066] Modal feature concatenation: concatenate visual modal features and speech modality features Perform splicing to obtain the spliced ​​feature vector , the concatenated feature vector The expression is as follows:

[0067] ,

[0068] Cross-modal attention calculation: text modality features As a query Sum , the concatenated features As a key , calculate the weight score of each attention head by the following formula, and get the The output representation of an attention head :

[0069] ,

[0070] in, Indicates the sequence number of the attention head, The value range is , , , , For the The weight score corresponding to the attention head, is the key V vector dimension, used to scale the attention score;

[0071] Concatenate the weight scores of all attention heads through a linear transformation matrix Get the final cross-modal fusion attention output , the cross-modal fusion attention output Calculated by the following formula:

[0072] .

[0073] The skip connection multi-head attention module SMA proposed above fuses multimodal information. On the one hand, adaptive weights assign importance to each modality. On the other hand, the skip connection can prevent the model from ignoring key emotional information in different modalities, and help the model capture and integrate subtle and important emotional clues from different modalities.

[0074] Optionally, the cross-modal fusion attention output Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category include:

[0075] Nonlinear transformation of cross-modal fusion attention output: Cross-modal fusion attention output is realized through the fully connected layer and activation function ReLU in the sentiment classifier The nonlinear transformation is performed to obtain the features after nonlinear transformation. , in order to enhance the model's ability to capture complex relationships and high-order interactions between features, the features after the nonlinear transformation Calculated by the following formula:

[0076] ,

[0077] in, is the weight matrix of the fully connected layer, is the bias vector, represents the activation function ReLU,

[0078] Sentiment classifier output: features after nonlinear transformation Input into the sentiment classifier and calculate the predicted probability distribution vector of the sentiment category using the following formula :

[0079] ,

[0080] in, is the weight matrix of the classifier, is the bias term. The above sentiment classifier uses a fully connected layer and generates the predicted probability distribution of the sentiment category through the softmax function.

[0081] Beneficial Effects

[0082] (1) The present invention achieves efficient fusion of multimodal information by combining the attention mechanism of expression and posture. This method can perform weighted processing according to the importance of different emotional clues, so that the emotion recognition model can dynamically capture the core emotional features, thereby significantly improving the accuracy of emotion recognition. At the same time, the introduction of the attention mechanism in practical use can enable the multimodal dialogue emotion recognition model to automatically pay attention to the expression and posture information with emotional significance in the dialogue, filter out irrelevant or noise information, and thus improve the accuracy and robustness of emotion recognition.

[0083] (2) The predicted emotion probability distribution finally generated by the present invention not only provides the final judgment of the emotion classification model on a certain emotion category, but also reveals the possibility of other emotion categories. By viewing the probability values ​​of each category, the ambiguity of the input emotion data and the potential emotion mixing can be understood. For example, when the probability of a data point in the two categories of anger and sadness is high, it can assist users to conduct more detailed emotion analysis, thereby more accurately reflecting the real emotional state. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Shown is a flow chart of a multimodal dialogue emotion recognition method in one embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following is a further description in conjunction with the accompanying drawings and specific embodiments. In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features.

[0086] Example 1

[0087] This embodiment provides a multimodal dialogue emotion recognition method, including:

[0088] Construct a multimodal interactive dialogue list U;

[0089] Based on the face recognition model SFace and gesture recognition tool Mediapipe in the multimodal interactive dialogue list A video clip from The expression temporal features are obtained from and posture timing characteristics ;

[0090] Expression temporal features based on face posture attention module FPA and posture timing characteristics Fusion is performed to obtain the final visual modality features ;

[0091] Constructing new expressions of contextual information And based on the emotion modeling technology PEMT, the emotion representation is obtained , expressing emotions Input to the text encoder SimCSE to obtain text modality features ;

[0092] According to the multimodal interactive dialogue list Every word in Corresponding audio clip Perform equal-interval sampling, input the sampled data into the data vectorization model data2vec to extract speech features, and aggregate the speech features of all frames to obtain speech modal features. ;

[0093] Visual modality features , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output ;

[0094] Cross-modal fusion attention output Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category .

[0095] In practical applications, since the existing conversational emotion recognition methods have a single source of emotion information and cannot accurately integrate information such as expressions and postures into a unified emotion output, the present invention solves this problem by using corresponding models to extract expression timing features and posture timing features and fuse them into visual modal features. In the process of fusing expression timing features and posture timing features, the model is able to focus on more important emotion timing information and enhance feature fusion by using multiple attention mechanisms. This solves the problem that traditional multimodal emotion recognition methods lack flexible attention mechanisms, making it difficult to highlight key emotion clues, thereby affecting recognition effects.

[0096] The present invention realizes multi-dimensional expression and in-depth understanding of emotional information by cross-modal alignment and fusion of visual modal features, textual modal features and speech modal features. In addition, the predicted emotional probability distribution finally generated by the present invention can improve the interpretability and transparency of the emotion recognition system. Users can observe the judgment confidence of the emotion classification model in each category, which helps to provide support for decision-making. In practical applications, especially in the fields of medical care, education and psychological counseling, this probability information can help professionals better understand the emotional state of the person being analyzed and support more targeted interventions or suggestions.

[0097] Example 2

[0098] Based on Example 1, this example introduces a specific implementation process of a multimodal dialogue emotion recognition method, such as Figure 1 As shown, specifically including the following:

[0099] 1. Data Preprocessing

[0100] Step 1: First, collect multimodal conversation data of multiple rounds of conversations involving multiple people, preprocess the data, and finally build a multimodal interactive conversation list ;

[0101] Step 1.1 collects all multimodal conversation data involving multiple participants, including text modality data, voice modality data, video modality data and speaker names corresponding to all utterances;

[0102] Step 1.2: Standardize and preprocess the text modal data of each utterance, including removing stop words, correcting spelling errors and punctuation, to ensure the coherence and accuracy of the text content, so as to obtain the text record corresponding to each utterance. ;

[0103] Step 1.3: Clean and standardize the speech modal data of each utterance, including noise reduction and equalization, to improve the clarity and consistency of the speech, so as to obtain the speech segment corresponding to each utterance. ;

[0104] Step 1.4 pre-processes the video modality data corresponding to each utterance, extracts 15 key frames by sampling at equal intervals to construct a streamlined video segment, thereby reducing data redundancy and improving processing efficiency, and finally obtains a video segment corresponding to each utterance ;

[0105] Building a multimodal interactive dialogue list It includes: collecting multimodal dialogue data of multiple rounds of dialogue involving multiple participants, preprocessing the multimodal dialogue data, and finally constructing a multimodal interactive dialogue list. ,

[0106] Multimodal interactive dialogue list Includes multiple words , where each utterance Contains a text record , a video clip and a voice clip ,

[0107] Multimodal interactive dialogue list The expression is: ,

[0108] Each word The expression is: ,

[0109] in, Indicates the sequence number of the speech, also known as the Session units, the value range is - , Represents the entire multimodal interactive dialogue list The total number of utterances in The corresponding speaker is .

[0110] 2. Feature Multimodal Fusion

[0111] Step 2: First, use the face recognition model SFace and the gesture recognition tool Mediapipe in a video clip. Perform face recognition and body recognition on the image sequence of face area and body area and ; Then, the spatiotemporal feature extraction model Timesformer is used to extract and The temporal features in the expression are obtained respectively. and posture timing characteristics Finally, the face posture attention module FPA is proposed to fuse the expression temporal features and the posture temporal features to obtain the final visual modal features ;

[0112] Step 2.1: Extract expression temporal features and posture temporal features;

[0113] Step 2.1.1, face sequence representation: a video clip is represented by the face recognition model SFace Recognize the face in the image, extract the face area, and form a sequence containing the face image , ,in Represents a sequence of face images A face image in one frame;

[0114] Step 2.1.2, gesture sequence representation: Use the gesture recognition tool Mediapipe to represent the same video clip. Human body posture recognition is performed in the image processing, and the human body area is extracted to form a sequence containing posture images. ,in Representing posture image sequence The posture image of one frame;

[0115] Step 2.1.3, temporal feature extraction: extract the face image sequence and pose image sequences Input into the spatiotemporal feature extraction model Timesformer to extract the expression temporal features and posture timing characteristics , used to capture facial expression dynamics and posture changes, the expression timing features and posture timing characteristics The expressions are as follows:

[0116] ,

[0117] ,

[0118] Step 2.2: Expression temporal features and posture timing characteristics Encoding is performed to obtain rich temporal information. The encoded expression temporal features and posture temporal features are obtained through MLP nonlinear transformation:

[0119] ,

[0120] ,

[0121] in, and Respectively represent the encoded expression temporal features and posture temporal features;

[0122] Step 2.3: Encoded expression temporal features and posture timing characteristics The expression attention coefficient is obtained by adaptive weighting through the following formula: and posture attention coefficient :

[0123] ,

[0124] ,

[0125] in, is the encoded expression temporal feature The weight matrix of is the encoded posture temporal feature The weight matrix of is the encoded expression temporal feature The bias vector of is the encoded posture temporal feature The bias vector of

[0126] Step 2.4: In order to improve the emotional representation ability of expressions and gestures, a multi-head attention mechanism is used to separately and posture timing characteristics Enhance the emotional representation and the encoded expression temporal features and posture timing characteristics The multi-head attention calculation is obtained by the following formulas: and :

[0127] ,

[0128] ,

[0129] in, and They represent the weight coefficients of the expression timing features and posture timing features obtained after calculation by the multi-head attention mechanism, , , , and are all learnable parameters, , , , and All are learnable parameters. The encoded expression temporal features and posture timing characteristics Performing multi-head attention calculations separately can enable the multimodal dialogue emotion recognition model to adaptively focus on more important temporal information about emotions in practical applications;

[0130] Step 2.5: Encoded posture temporal features As query Q, the encoded expression temporal features As key K and value V, the enhanced expression timing feature is obtained by the following formula :

[0131] ,

[0132] The encoded expression temporal features As query Q, the encoded posture temporal features As the key K and value V, the enhanced posture timing feature is obtained by the following formula :

[0133] .

[0134] The above uses the cross attention mechanism to realize the encoded expression temporal features and posture timing characteristics The information interaction between them can further enhance feature fusion;

[0135] Step 2.6: Enhanced expression temporal features and posture timing characteristics The final visual modality feature is obtained by fusion in a weighted manner through the following formula: :

[0136] The posture temporal features are fused to obtain the final visual modality features .

[0137] Step 3: Construct a new representation of the contextual information And the emotional representation is obtained through the suggested emotional modeling technology PEMT , expressing emotions Input to the text encoder SimCSE to obtain text modality features ;

[0138] Step 3.1, context modeling: extracting the current discourse The text record of the three previous utterances aims to enhance the text record of the current utterance by taking into account the historical utterance information The semantic understanding of emotion recognition provides richer context information and constructs new expressions containing context information. , the new expression containing contextual information The expression is as follows:

[0139] ,

[0140] Step 3.2, text feature extraction: new expressions using contextual information , Speaker and text records , proposed a prompt-based emotion modeling technique PEMT to capture the speaker Long-distance dependencies between the utterances and the emotions expressed , the emotion expressed The expression is as follows:

[0141] ,

[0142] The sentiment representation is obtained by using the text encoder SimCSE Special Tokens <mask>Encode the representation to obtain the text modality feature , providing key information of text modality for multimodal emotion recognition. The text modality feature The expression is as follows:

[0143] .

[0144] Step 4: First, according to the discourse Corresponding audio clip Perform equal-interval sampling. Then, input the sampled data into the data vectorization model data2vec to extract speech features. Finally, aggregate the speech features of all frames to obtain speech modal features. ;

[0145] Step 4.1, speech signal segmentation: Given each utterance Corresponding audio clip , continuous speech segments Split into The frame is used as voice data, and the voice data expression is as follows:

[0146] ,

[0147] in, Indicates Voice Clips No. Frame of voice data, The value range is 1- , is the total number of frames. is set to 8; in other possible embodiments of the present invention, different numbers may be set;

[0148] Step 4.2, speech feature extraction: extract each speech data Input into the data vectorization model data2vec to obtain speech feature representation , the speech feature representation The expression is as follows:

[0149] ,

[0150] Step 4.3, aggregate all frame features: use average pooling to represent all speech features Aggregation into discourse The speech modality characteristics of , the speech modality feature Calculated by the following formula:

[0151] .

[0152] Step 5: Visual modality features , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output ;

[0153] Step 5.1, modal feature concatenation: concatenate the visual modal features and speech modality features Perform splicing to obtain the spliced ​​feature vector , the concatenated feature vector The expression is as follows:

[0154] ,

[0155] Step 5.2, cross-modal attention calculation: Text modality features As a query Sum , the concatenated features As a key , calculate the weight score of each attention head by the following formula, and get the The output representation of an attention head :

[0156] ,

[0157] in, Indicates the sequence number of the attention head, The value range is , , , , For the The weight score corresponding to the attention head, is the key V vector dimension, used to scale the attention score;

[0158] Concatenate the weight scores of all attention heads through a linear transformation matrix Get the final cross-modal fusion attention output , the cross-modal fusion attention output Calculated by the following formula:

[0159] ,

[0160] The skip connection multi-head attention module SMA proposed above fuses multimodal information. On the one hand, adaptive weights are used to assign importance to each modality. On the other hand, the skip connection can prevent the model from ignoring key emotional information in different modalities, and helps the model capture and integrate subtle and important emotional clues from different modalities. In this embodiment, It is set to 3. In other possible embodiments of the present invention, different numbers may be set.

[0161] Step 6: Fusion of attention outputs across modalities Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category ;

[0162] Step 6.1, nonlinear transformation of cross-modal fusion attention output: cross-modal fusion attention output is realized through the fully connected layer and activation function ReLU in the sentiment classifier The nonlinear transformation is performed to obtain the features after nonlinear transformation. , in order to enhance the model's ability to capture complex relationships and high-order interactions between features, the features after the nonlinear transformation Calculated by the following formula:

[0163] ,

[0164] in, is the weight matrix of the fully connected layer, is the bias vector, Represents the activation function ReLU;

[0165] Step 6.2, sentiment classifier output: transform the features after nonlinear transformation Input into the sentiment classifier and calculate the predicted probability distribution vector of the sentiment category using the following formula :

[0166] ,

[0167] in, is the weight matrix of the classifier, is the bias term. The above sentiment classifier uses a fully connected layer and generates the predicted probability distribution of the sentiment category through the softmax function.

[0168] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.< / mask> < / mask>

Claims

1. A multimodal dialogue emotion recognition method, characterized by comprising: Construct a multimodal interactive dialogue list U; Based on the face recognition model SFace and gesture recognition tool Mediapipe in the multimodal interactive dialogue list A video clip from The expression temporal features are obtained from and posture timing characteristics ; Expression temporal features based on face posture attention module FPA and posture timing characteristics Fusion is performed to obtain the final visual modality features ; Constructing new expressions of contextual information And the emotional representation is obtained through the suggested emotional modeling technology PEMT , expressing emotions Input to the text encoder SimCSE to obtain text modality features ; According to the multimodal interactive dialogue list Every word in Corresponding audio clip Perform equal-interval sampling, input the sampled data into the data vectorization model data2vec to extract speech features, and aggregate the speech features of all frames to obtain speech modal features. ; Visual modality features , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output ; Cross-modal fusion attention output Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category ; The facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics The integration includes: Temporal features of facial expressions Posture timing characteristics Encoding is performed, and the encoded expression time series features and posture time series features are obtained through MLP nonlinear transformation: ， ， in, and Respectively represent the encoded expression temporal features and posture temporal features; The encoded expression temporal features and posture timing characteristics The expression attention coefficient is obtained by adaptive weighting through the following formula: and posture attention coefficient : ， ， in, is the encoded expression temporal feature The weight matrix of is the encoded posture temporal feature The weight matrix of is the encoded expression temporal feature The bias vector of is the encoded posture temporal feature The bias vector of The encoded expression temporal features and posture timing characteristics The multi-head attention calculation is obtained by the following formulas: and : ， ， in, and They represent the weight coefficients of the expression timing features and posture timing features obtained after calculation by the multi-head attention mechanism, , , , and are all learnable parameters, , , , and All are learnable parameters; The facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics Integration also includes: The encoded posture temporal features As query Q, the encoded expression temporal features As key K and value V, the enhanced expression timing feature is obtained by the following formula : ， The encoded expression temporal features As query Q, the encoded posture temporal features As the key K and value V, the enhanced posture timing feature is obtained by the following formula : ； The facial posture attention module FPA is used to analyze the expression temporal features and posture timing characteristics Fusion is performed to obtain the final visual modality features include: Enhanced expression timing features and posture timing characteristics The final visual modality feature is obtained by fusion in a weighted manner through the following formula: : 。 2. The multimodal dialogue emotion recognition method according to claim 1 is characterized in that: The multimodal interactive dialogue list is constructed It includes: collecting multimodal dialogue data of multiple rounds of dialogue involving multiple participants, preprocessing the multimodal dialogue data, and finally constructing a multimodal interactive dialogue list. , The multimodal interactive dialogue list Includes multiple words , where each utterance Contains a text record , a video clip and a voice clip , The multimodal interactive dialogue list The expression is: , Each of the above words The expression is: , in, Indicates the sequence number of the speech, the value range is - , Represents the entire multimodal interactive dialogue list The total number of utterances in The corresponding speaker is .

3. The multimodal dialogue emotion recognition method according to claim 1 is characterized in that: The face recognition model SFace and gesture recognition tool Mediapipe are used in the multimodal interactive dialogue list. A video clip from The expression temporal features are obtained from and posture timing characteristics include: A video clip is detected by face recognition model SFace Recognize the face in the image, extract the face area, and form a sequence containing the face image , ,in Represents a sequence of face images A face image in one frame; In the same video clip, the gesture recognition tool Mediapipe Human body posture recognition is performed in the image processing, and the human body area is extracted to form a sequence containing posture images. ,in Representing posture image sequence A pose image of a frame in the image; The face image sequence and pose image sequences Input into the spatiotemporal feature extraction model Timesformer to extract the expression temporal features and posture timing characteristics , the expression temporal features and posture timing characteristics The expressions are as follows: ， 。 4. The multimodal dialogue emotion recognition method according to claim 1, characterized in that: The new expression of constructing contextual information And the emotional representation is obtained through the suggested emotional modeling technology PEMT , expressing emotions Input to the text encoder SimCSE to obtain text modality features include: Extract current utterance The text record of the previous three utterances constructs a new utterance containing contextual information , the new expression containing context information The expression is as follows: ， New expressions using contextual information , Speaker and text records , proposed a prompt-based emotion modeling technique PEMT to capture the speaker Long-distance dependencies between the utterances and the emotions expressed , the emotion expressed The expression is as follows: ， The sentiment representation is obtained by using the text encoder SimCSE Special Tokens <mask>Encode the representation to obtain the text modality feature , the text modality feature The expression is as follows:< / mask> 。 5. The multimodal dialogue emotion recognition method according to claim 1 is characterized in that: The multimodal interactive dialogue list Every word in Corresponding audio clip Perform equal-interval sampling, input the sampled data into the data vectorization model data2vec to extract speech features, and aggregate the speech features of all frames to obtain speech modal features. include: Given each utterance Corresponding audio clip , continuous speech segments Split into The frame is used as voice data, and the voice data expression is as follows: ， in, Indicates Voice Clips No. Frame of voice data, The value range is 1- , is the total number of frames, Each voice data Input into the data vectorization model data2vec to obtain speech feature representation , the speech feature representation The expression is as follows: ， Use average pooling to represent all speech features Aggregation into discourse The speech modality characteristics of , the speech modality feature Calculated by the following formula: 。 6. The multimodal dialogue emotion recognition method according to claim 1 is characterized in that: The visual modality features , text modality features and speech modality features Input to the skip connection multi-head attention module SMA to fuse multimodal information and obtain cross-modal fusion attention output include: Visual modality features and speech modality features Perform splicing to obtain the spliced ​​feature vector , the concatenated feature vector The expression is as follows: ， Text modal features As a query Sum , the concatenated features As a key , calculate the weight score of each attention head by the following formula, and get the The output representation of an attention head : ， in, Indicates the sequence number of the attention head, The value range is , , , , For the The weight score corresponding to the attention head, is the key V vector dimension, used to scale the attention score; Concatenate the weight scores of all attention heads through a linear transformation matrix Get the final cross-modal fusion attention output , the cross-modal fusion attention output Calculated by the following formula: 。 7. The multimodal dialogue emotion recognition method according to claim 6 is characterized in that: The cross-modal fusion attention output Perform nonlinear transformation and transform the features after nonlinear transformation Input to the sentiment classifier to generate the predicted probability distribution vector of the sentiment category include: Cross-modal fusion attention output is achieved through the fully connected layer and activation function ReLU in the sentiment classifier The nonlinear transformation is performed to obtain the features after nonlinear transformation. , the feature after the nonlinear transformation Calculated by the following formula: ， in, is the weight matrix of the fully connected layer, is the bias vector, represents the activation function ReLU, The features after nonlinear transformation Input into the sentiment classifier and calculate the predicted probability distribution vector of the sentiment category using the following formula : ， in, is the weight matrix of the classifier, is the bias term.

Images