Scene context assisted multi-modal emotion recognition method

Abstract

The invention provides a scene context-assisted multi-modal emotion recognition method, which comprises the following steps: constructing an emotion data set, each group of data Si comprising video frame data Ii, voice data Ai and text data Ti; extracting a figure eye movement sequence signal Ei in the data Ii, and extracting a feature DIi1 of the sequence signal Ei under a dynamic space-time view by using a dynamic space-time extraction network GI1; extracting a feature DIi2 of the data Ii under a static local view and a feature DIi3 of the data Ii under a global context view by using a static local extraction network GI2 and a global context extraction network GI3; extracting a voice modal feature DAi of the data Ai by using a voice feature extraction network GA; performing word segmentation processing on the data Ti to obtain a word sequence vector DTi; and splicing the features DTi, DIi1, DIi2, DIi3 and DAi, and inputting the spliced features into a trained large language network for emotion classification to obtain a data Si emotion classification result. The method effectively promotes the improvement of a multi-modal emotion recognition theory system which takes a person as a center and is assisted by a scene context, and provides a more reliable solution for emotion analysis in a complex real scene.

Description

Technical Field

[0001] The present invention relates to the fields of emotion recognition technology and multimodal learning, and in particular to a situational context-assisted multimodal emotion recognition method. Background Art

[0002] With the rapid development of artificial intelligence (AI), emotion recognition has become a crucial component in areas such as personalized services, mental health monitoring, and human-computer interaction. Multimodal emotion recognition, by combining multiple information sources like vision, speech, and text, can effectively capture the diversity and complexity of emotions, providing more accurate emotion analysis tools for related applications.

[0003] Situational context, a composite of situational semantics, social interactions, and environmental cues, provides a crucial semantic background for emotion recognition. However, existing emotion recognition research lacks relevant exploration of the joint modeling of situational context. Related emotion recognition technologies still face numerous challenges, limiting their practical application. These challenges are specifically reflected in the following aspects:

[0004] The difficulty of labeling and modeling situational context: In current emotion recognition research, the labeling process of situational context is complex and costly, often relying on manual labeling and unable to achieve large-scale data processing.

[0005] Stability issues of data-driven context modeling: Many emotion recognition methods are mainly based on data-driven global modeling techniques. However, in practical applications, data is often affected by external noise and environmental changes. In complex or dynamic environments, the lack of stability of context information affects the recognition accuracy and model adaptability.

[0006] Insufficient collaborative utilization of cross-modal complementary information: Existing methods fail to effectively coordinate the relationship between situational context information and character subjects in multimodal fusion, which limits the overall performance of the emotion recognition system, such as overly one-sided judgment of emotions.

[0007] Therefore, new fusion strategies are needed to fully explore the information correlation between different modalities and form a more comprehensive and consistent emotion representation. Summary of the Invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a situational context-assisted multimodal emotion recognition method, which aims to integrate information from three modalities: vision, speech, and text through a large language model to improve the accuracy and effectiveness of emotion classification.

[0009] Technical solution: To achieve the above-mentioned purpose, the context-assisted multimodal emotion recognition method of the present invention comprises the following steps:

[0010] S1. Construct an emotion dataset, where each set of data S i Including video frame data I i , Voice Data A i and text data T i , i represents the sentiment category of the data;

[0011] S2. Extract video frame data I i Eye movement sequence signal E i , and use the dynamic spatiotemporal extraction network G I 1 Extract the eye movement sequence signal E i Features D under the dynamic spatiotemporal view Ii 1;

[0012] S3, using static local extraction network G I 2. Global context extraction network G I 3. Extract video frame data I i Feature D under the static local view Ii 2 and feature D under the global context view Ii 3;

[0013] S4, using speech feature extraction network G A Extract voice data A i The speech modality feature D Ai ;

[0014] S5. Text data T i Perform word segmentation to obtain the word sequence vector D Ti ;

[0015] S6, feature D Ti 、D Ii 1. D Ii 2. D Ii 3. D Ai Perform concat splicing to obtain feature D i (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai );

[0016] S7, use the trained large language network model to analyze feature D i (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai ) to perform emotion classification and obtain data S i Emotion classification results.

[0017] Preferably, the character's eye movement sequence signal E i The extraction method is: use the tool RetinaFace library to extract the video frame data I i Perform face detection, locate the face bounding box and facial key points, extract eye movement frame-level features from the facial key points, and align the eye movement frame-level features in time to form a timing signal E i .

[0018] Preferably, the feature D under the dynamic spatiotemporal view Ii 1 Extraction method: Sequence signal E i Input dynamic spatiotemporal extraction network G I 1. Network G I 1. The spatiotemporal block embedding layer first obtains the sequence signal E i Position information, the output feature vector is processed by masking, and then enters the encoding layer, MLP layer, decoding layer, and MLP layer in sequence. The output is the original spatiotemporal visual feature containing effective spatiotemporal information. The original spatiotemporal visual feature is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the dynamic spatiotemporal view. Ii 1.

[0019] Preferably, the feature D in the static local view Ii 2 Extraction method is: video frame data I i Input static local extraction network G I 2. In the network G I 2 is processed in sequence by the convolutional mapping layer, the encoding layer, and the MLP layer, and the output is the original static local view feature containing effective static local information. The original static local view feature is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the static local view Ii 2.

[0020] Preferably, the feature D under the global context view Ii 3 Extraction method is: video frame data I i As visual data input global context extraction network G I The visual coding module in 3 is processed in sequence by the convolutional mapping layer, the coding layer, and the MLP layer in the visual coding module, and the output is the original global context view feature; the video frame data I i The classification label data Y is used as text data to input the global context extraction network G I 3. In network G IAfter being processed by the MLP layer in step 3, the output feature vector enters the pre-trained text encoder module of the Clip network to obtain the text encoding, calculate the cosine similarity between the text encoding and the original global context view feature, and use the cosine similarity as guidance information to guide the training of the visual encoding module; finally, the output feature of the trained visual encoding module is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the global context view. Ii 3.

[0021] Preferably, the speech modality feature D Ai The extraction method is: voice data A i Input speech modality feature extraction network G A , in the network G A The spatiotemporal position information of the speech is extracted through the position encoding layer, and the output feature vector enters the ViT encoding layer and the MLP layer for processing, and the output is the original speech modal feature. The original speech modal feature inputs a parameter-trainable one-dimensional linear mapping layer to obtain the speech feature D Ai .

[0022] Preferably, the word sequence vector D Ti The acquisition method is: use the pre-trained word segmentation tool BERT tokenizer pre-defined rules and vocabulary to train the text data T i Perform basic segmentation and use the bidirectional maximum matching algorithm to correct the ambiguous words that may exist after basic segmentation. After correction, filter out stop words and low-frequency words to obtain a standardized word sequence vector D Ti .

[0023] Preferably, the training process of the large language network model is: the training data is feature D i During the training process, the other model parameters are frozen, the last layer is fine-tuned, and the parameters of the last layer are iteratively updated until the model converges.

[0024] The network G I 1~G I 3, and network G A The training data is data S i And labels, the network parameters are continuously adjusted during the training process until the model converges.

[0025] Preferably, the network G I 1~G I 3. Network G A The iterative update method for the parameters of the last layer of the large language network is to use the ε-Greedy strategy to make decisions and update them. The expression is as follows:

[0026]

[0027] Among them, θ G G I 1~G I 2. Network G A The parameters of the network trained in , For network G I 3 parameters, θ B is the parameter of the large language network, α is the learning rate, G I 1~G I 2. Network G A The network gradients trained in is the gradient of the large language network, is the gradient of the large language network, Loss B is the output logistic cross entropy loss of network B, For network G I Similarity loss between visual encoding features and text encoding features in 3.

[0028] Preferably, the network G I 1~G I 3. Network G A The iterative update method of the last layer parameters of the large language network is: by minimizing the cross entropy Loss between the output value of the fully connected layer and the true category value B , and use the gradient descent optimization algorithm to update, the expression is as follows:

[0029]

[0030] Where X includes the network G I 1~G I 2. Network G A and the large language network, is the output probability of the emotion category of the i-th sample by the large language network, y i is the true emotion category of the i-th sample, f image (I) represents the data I input into the visual encoding module f image The visual features obtained after text (T) is the label data T input text encoding module f text The text features obtained after , || || represents the regularization operation.

[0031] Beneficial effects: The present invention has the following advantages: 1. By decomposing the emotion recognition process into three hierarchical context-aware modules: visual modality, speech modality, and text modality, and introducing a deep learning feature extraction and dynamic fusion mechanism, the present invention effectively solves the common problems of feature redundancy and semantic loss in multimodal emotion recognition;

[0032] 2. Innovatively constructing a dynamic representation space for context-person interactions, it can adaptively capture the correlation between scene semantics and individual emotional expressions, significantly improving the ability to discern complex emotional states;

[0033] 3. Through the context-aware fusion of multimodal features, a more accurate understanding of emotional semantics is achieved, which significantly improves the system's sensitivity to subtle emotional changes while maintaining high robustness, thus providing an effective solution for building a more humane intelligent emotion recognition system.

[0034] 4. It solves the problems faced by existing research in multimodal emotion recognition, such as the lack of stability of data-driven global modeling and the inefficient integration of the complementary relationship between situational representation and character subject. It effectively promotes the improvement of the "character-centered, situational context-assisted" multimodal emotion recognition theoretical system, and provides a more reliable solution for emotion analysis in complex real-world scenarios. In particular, it shows important practical value in fields such as medical monitoring and intelligent education that have strict requirements on the robustness of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of the present invention;

[0036] Figure 2 It is the structural diagram of the multi-view multimodal network G;

[0037] Figure 3 Dynamic spatiotemporal extraction network G I 1 structure diagram;

[0038] Figure 4 Extract the global context network G I 2 structure diagram;

[0039] Figure 5 For the static local extraction network G I 3. Structural diagram;

[0040] Figure 6 G is the speech modality feature extraction network A Structural diagram of

[0041] Figure 7 Flowchart for network training and fine-tuning. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the multimodal emotion recognition method of the present invention includes the following contents:

[0044] First, collect video data and classify it into seven sub-datasets S1, S2, S3, S4, S5, S6, and S7 according to emotion categories: anger, disgust, fear, joy, neutral, sadness, and surprise;

[0045] Each sub-dataset S i Including video frame data I i , Voice Data A i And the text data T of audio to text i , i=1~7, indicating the emotion category.

[0046] Among them, the video frame data I i It is obtained by segmenting and extracting frames from video data. Voice data A i It is extracted from the video data using FFmpeg, and the voice data A is converted using the Vosk method. i Convert the audio to text and get text data T i .

[0047] 2. Text Data T i Perform word segmentation processing to convert Chinese words into word vectors D that can be recognized by the large language model network B T , used for subsequent model training.

[0048] The word segmentation process is as follows: First, use the pre-trained word segmentation tool BERT tokenizer pre-defined rules and vocabulary to segment the text data T i Perform basic segmentation, then use the bidirectional maximum matching algorithm to correct the ambiguous words that may exist after basic segmentation, and finally filter out stop words and low-frequency words to obtain the normalized word sequence vector D Ti .

[0049] 3. Extracting Video Frame Data I i The sequence signal E of the eye movement (including gaze vector and facial key points) of the character in the i ;

[0050] The RetinaFace library is used to extract eye movement sequence signals. The extraction process is as follows:

[0051] Locating facial bounding boxes and key points: RetinaFace uses a lightweight MobileNet-0.25 backbone network for real-time face detection, leveraging pre-trained geometry-sensitive feature pyramids to accurately locate facial bounding boxes and 108-dimensional facial key points in each frame.

[0052] Extract eye movement features from facial key points: Calculate the binocular ROI (Region of Interest) based on the key point coordinates, and extract the gaze direction vector through a cascaded deformable convolutional network. The gaze direction vector is output in the form of pitch and yaw angles in a 3D spherical coordinate system.

[0053] Construct sequence signal E i :All frame-level features are aligned in time to form a timing signal E i , which contains the normalized keypoint coordinates and sight angle parameters of each frame.

[0054] 4. Using dynamic spatiotemporal extraction network G I 1 Extract the eye movement sequence signal E i Features D under the dynamic spatiotemporal view Ii 1.

[0055] like Figure 3 As shown, the dynamic spatiotemporal extraction network G I 1 is a Transformer network, and the feature extraction process is:

[0056] Character eye movement sequence signal E i As data input, it first passes through a spatiotemporal block embedding layer to obtain position information. The output feature vector is masked and enters a 6-layer encoding layer module. It is processed by the MLP layer and input to the 6-layer decoding layer module. Then it is processed by the MLP layer and output as the original spatiotemporal visual feature containing effective spatiotemporal information. Finally, it passes through a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the dynamic spatiotemporal view. Ii 1.

[0057] 5. Using static local extraction network G I 2. Global context extraction network G I 3. Extract video frame data I i Feature D under the static local view Ii 2 and feature D under the global context view Ii 3;

[0058] like Figure 4 As shown, the static local extraction network G I 2 is a MAE network based on ViT, and the feature extraction process is:

[0059] Video frame data I i As image data input, it first passes through a convolutional mapping layer, and the output feature vector enters a 6-layer encoding layer module. After being processed by the MLP layer, the output is the original static local view feature containing effective static local information. Finally, it passes through a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the static local view.Ii 2.

[0060] like Figure 5 As shown, the global context extraction network G I 3 is an EVA structure network based on CLIP. The feature extraction process is:

[0061] Video frame data I i As the image data input, it first passes through a convolutional mapping layer, and the output feature vector enters a 6-layer encoding layer module, is processed by the MLP layer, and the output is the original global context view feature; while the video frame data I i The classification label data Y is used as text data input, first passes through an MLP layer, and the output feature vector enters the text encoder module of the Clip network, and the output is the text encoding, which serves as guidance information to guide the encoding of the MLP layer. Finally, the original global context view feature passes through a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the global context view Ii 3.

[0062] 6. Using speech feature extraction network G A Extract voice data A i The speech modal features D Ai ;

[0063] like Figure 6 As shown, the speech modality feature extraction network G A It is a VGGish network for processing speech signals. The feature extraction process is:

[0064] Voice data A i As data input, it first passes through a position encoding layer to extract the spatiotemporal position information of the speech, and the output feature vector enters a 6-layer ViT encoding layer module, is processed by the MLP layer, and the output is the original speech modal feature. Finally, it passes through a parameter-trainable one-dimensional linear mapping layer to obtain the speech feature D Ai .

[0065] 7. Acquired Features D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai Perform concat splicing to obtain feature D i (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai ).

[0066] 8. If Figure 7 As shown, the large language network model B is used to analyze the feature Di (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai ) to classify emotions.

[0067] The training process of the large language network model B is as follows: freeze the other parameters of the model, fine-tune the last layer, and iteratively update the parameters of the last layer (fully connected layer) of network B until the model converges. The training data is feature D i and their emotional labels.

[0068] 9. The above multi-view multimodal feature network G(G I 1, G I 2, G I 3, G A ) is initialized using the maximum likelihood estimation method, and the multi-view multimodal feature network G structure is as follows Figure 2 As shown, the initialization process is expressed as:

[0069] θ MLE =argmax θ L(θ)

[0070]

[0071] Where θ is the model parameter, L(θ) is the log-likelihood function of the training data, N is the number of training samples, and x i is the i-th observation data, P d (x i ) is the sample x under the data distribution i The probability, P m (x i ; θ) is the model generating sample x under parameter θ i probability.

[0072] The multi-view multi-modal feature network G continuously adjusts the parameters of G during the training process until the network converges. The training data is the sub-dataset S i and its labels.

[0073] 10. The iterative update method for the last layer parameters of the multi-view multimodal feature network G and the large language network model B is to use the ε-Greedy strategy to make decisions and update them. The expression is as follows:

[0074]

[0075] Among them, θ G ,θ B is the parameter of the last layer of the multi-view multimodal feature network and the large language network model, α is the learning rate, is a gradient operation.

[0076] 11. The iterative update method of the parameters of the last layer of the multi-view multimodal feature network G and the large language network model B is: by minimizing the cross entropy Loss between the output value of the fully connected layer and the true category value B , and use the gradient descent optimization algorithm to update, the expression is as follows:

[0077]

[0078] Among them, X includes G and B, which represents the multi-view multimodal feature network G and the large language model network B. is the output probability of the emotion category of the i-th sample by the large language network model B, y i

[0079] is the true emotion category of the i-th sample, θ X is the parameter of the corresponding model network X, α is the learning rate, is the loss function with respect to θ X gradient.

Claims

1. A context-assisted multimodal emotion recognition method, characterized in that: The following steps are involved: S1. Construct an emotion dataset, where each set of data S i Including video frame data I i , Voice Data A i and text data T i , i represents the sentiment category of the data; S2. Extract video frame data I i Eye movement sequence signal E of the character in the figure i , and use the dynamic spatiotemporal extraction network G I 1 Extract the eye movement sequence signal E i Features D under the dynamic spatiotemporal view Ii 1; S3, using static local extraction network G I 2. Global context extraction network G I 3. Extract video frame data I i Feature D under the static local view Ii 2 and feature D under the global context view Ii 3; S4, using speech feature extraction network G A Extract voice data A i The speech modality feature D Ai ; S5. Text data T i Perform word segmentation to obtain the word sequence vector D Ti ; S6, feature D Ti 、D Ii 1. D Ii 2. D Ii 3. D Ai Perform concat splicing to obtain feature D i (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai ); S7, use the trained large language network model to analyze feature D i (D Ti , D Ii 1, D Ii 2, D Ii 3, D Ai ) to perform emotion classification and obtain data S i Emotion classification results.

2. The multimodal emotion recognition method according to claim 1, characterized in that The character's eye movement sequence signal E i The extraction method is: use the tool RetinaFace library to extract the video frame data I i Perform face detection, locate the face bounding box and facial key points, extract eye movement frame-level features from the facial key points, and align the eye movement frame-level features in time to form a timing signal E i .

3. The multimodal emotion recognition method according to claim 1, wherein: The feature D under the dynamic spatiotemporal view Ii 1 Extraction method: Sequence signal E i Input dynamic spatiotemporal extraction network G I 1. Network G I 1. The spatiotemporal block embedding layer first obtains the sequence signal E i Position information, the output feature vector is processed by masking, and then enters the encoding layer, MLP layer, decoding layer, and MLP layer in sequence. The output is the original spatiotemporal visual feature containing effective spatiotemporal information. The original spatiotemporal visual feature is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the dynamic spatiotemporal view. Ii 1.

4. The multimodal emotion recognition method according to claim 1, wherein: The feature D in the static local view Ii 2 Extraction method is: video frame data I i Input static local extraction network G I 2. In the network G I 2 is processed in sequence by the convolutional mapping layer, the encoding layer, and the MLP layer, and the output is the original static local view feature containing effective static local information. The original static local view feature is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the static local view Ii 2.

5. The multimodal emotion recognition method according to claim 1, wherein: The feature D under the global context view Ii 3 Extraction method is: video frame data I i As visual data input global context extraction network G I The visual encoding module in 3 is processed by the convolutional mapping layer, encoding layer, and MLP layer in sequence, and the output is the original global context view feature; The video frame data I i The classification label data Y is used as text data to input the global context extraction network G I 3. In network G I After being processed by the MLP layer in step 3, the output feature vector enters the pre-trained text encoder module of the Clip network to obtain the text encoding, calculate the cosine similarity between the text encoding and the original global context view feature, and use the cosine similarity as guidance information to guide the training of the visual encoding module; finally, the output feature of the trained visual encoding module is input into a parameter-trainable one-dimensional linear mapping layer to obtain the feature D under the global context view. Ii 3.

6. The multimodal emotion recognition method according to claim 1, wherein: The speech modality feature D Ai The extraction method is: voice data A i Input speech modality feature extraction network G A , in the network G A The spatiotemporal position information of the speech is extracted through the position encoding layer, and the output feature vector enters the ViT encoding layer and the MLP layer for processing, and the output is the original speech modal feature. The original speech modal feature inputs a parameter-trainable one-dimensional linear mapping layer to obtain the speech feature D Ai .

7. The multimodal emotion recognition method according to claim 1, characterized in that: The word sequence vector D Ti The acquisition method is: use the pre-trained word segmentation tool BERT tokenizer pre-defined rules and vocabulary to train the text data T i Perform basic segmentation and use the bidirectional maximum matching algorithm to correct the ambiguous words that may exist after basic segmentation. After correction, filter out stop words and low-frequency words to obtain a standardized word sequence vector D Ti .

8. The multimodal emotion recognition method according to claim 1, wherein: The training process of the large language network model is as follows: the training data is feature D i During the training process, the other model parameters are frozen, the last layer is fine-tuned, and the parameters of the last layer are iteratively updated until the model converges. The network G I 1~G I 3, and network G A The training data is data S i And labels, the network parameters are continuously adjusted during the training process until the model converges.

9. The multimodal emotion recognition method according to claim 8, characterized in that: Network G I 1~G I 3. Network G A The iterative update method for the parameters of the last layer of the large language network is to use the ε-Greedy strategy to make decisions and update them. The expression is as follows: Among them, θ G G I 1~G I 2. Network G A The parameters of the network trained in , For network G I 3 parameters, θ B is the parameter of the large language network, α is the learning rate, G I 1~G I 2. Network G A The network gradients trained in is the gradient of the large language network, is the gradient of the large language network, Loss B is the output logistic cross entropy loss of network B, For network G I Similarity loss between visual encoding features and text encoding features in 3.

10. The multimodal emotion recognition method according to claim 8, characterized in that: Network G I 1~G I 3. Network G A The iterative update method of the last layer parameters of the large language network is: by minimizing the cross entropy Loss between the output value of the fully connected layer and the true category value B , and use the gradient descent optimization algorithm to update, the expression is as follows: Where X includes the network G I 1~G I 2. Network G A and the large language network, is the output probability of the emotion category of the i-th sample by the large language network, y i is the true emotion category of the i-th sample, f image (I) represents the data I input into the visual encoding module f image The visual features obtained after text (T) is the label data T input text encoding module f text The text features obtained after ,|||| represents the regularization operation.

Images