A multi-modal based video emotion recognition method

Abstract

The application discloses a video emotion recognition method based on multi-modal, relates to the technical field of computer vision, deep learning and emotion computing, and comprises the following steps: constructing a micro-motion label system, performing secondary labeling on a CAER data set based on the micro-motion label system, and labeling micro-motion labels; constructing a dual-modal feature extraction network, including a face macro-emotion encoder for extracting macro facial features and a time-series micro-motion feature encoder for extracting time-series micro-motion features; the face macro-emotion encoder outputs a Logits vector for face emotion classification for the face in the video; the time-series micro-motion feature encoder outputs a Logits vector for micro-motion emotion classification for the body in the video; and the dual-modal posterior Logits are fused, and the final emotion category is output based on the fused Logits vector. The application significantly improves the accuracy and robustness of video emotion recognition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and emotion computing, and in particular to a multimodal video emotion recognition method. Background Technology

[0002] Human emotion recognition is a core technology in human-computer interaction, affective computing, and behavioral analysis. By analyzing human behavioral characteristics and facial expressions, it achieves accurate judgment of inner emotional states, and has significant application value in scenarios such as intelligent customer service, mental health monitoring, and intelligent education. Human emotional expression often contains multimodal information, with facial expressions being the most direct carrier of emotion, while micro-gestures can supplement and convey richer emotional details. Combining the two can significantly improve the comprehensiveness and accuracy of emotion recognition.

[0003] Existing video emotion recognition methods mainly suffer from two types of problems: First, traditional emotion recognition methods often rely on single-modal information, such as judging emotions solely through facial expressions (e.g., DeepFace focuses only on facial emotion recognition). While these models perform well in static images or macroscopic expression recognition, they neglect key emotional cues contained in micro-movements, resulting in poor adaptability to recognition scenarios and susceptibility to factors such as facial occlusion. Second, in existing emotion recognition schemes that include micro-movements, the micro-movement category labeling system is not perfect, lacking a standardized label set that comprehensively covers emotion-related micro-movements, and failing to fully utilize the complementarity of multimodal information, making it difficult to meet the recognition accuracy requirements of practical applications.

[0004] In terms of datasets, the CAER dataset is a large-scale real-world visual dataset for context-aware emotion recognition. As a commonly used video emotion recognition dataset, its original annotations mainly focus on facial emotion categories and lack fine-grained annotations of micro-movements, failing to provide sufficient training data support for multimodal emotion recognition. Meanwhile, existing multimodal fusion methods often employ simple feature splicing or early fusion strategies, failing to fully explore the deep correlations between different modalities, resulting in poor fusion performance and hindering the realization of the synergistic advantages of multimodal data. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this invention provides a multimodal video emotion recognition method. By constructing a comprehensive micro-motion labeling system and optimizing dataset annotation, this method introduces non-facial, non-verbal micro-motion cues to compensate for the deficiencies of macro-facial recognition in describing the dynamic process of emotions. Furthermore, by employing an efficient posterior fusion strategy, this method fully combines the modal advantages of facial expressions and body micro-movements, significantly improving the accuracy and robustness of video emotion recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution, including: A multimodal video emotion recognition method includes the following steps: S1. Construct a micro-action labeling system. Based on the micro-action labeling system, perform secondary annotation on the existing video datasets used for emotion recognition to label micro-actions. S2, construct a dual-modal feature extraction network, including a facial macro-emotion encoder for extracting macro-facial features and a temporal micro-motion feature encoder for extracting temporal micro-motion features; The facial macro-emotion encoder takes a face from a video and outputs a Logits vector for facial emotion classification. ;in, Number of emotion categories; The temporal micro-motion feature encoder takes the body in the video and outputs Logits vectors for micro-motion emotion classification. ; Among them, the Logits vectors output by the facial macro-emotion encoder and the temporal micro-motion feature encoder are for the same video time window; S3, Construct a classifier for... and Perform bimodal posterior logits fusion to obtain the final fused logits vector. Based on fused Logits vector Output the final emotion category; S4. The model is trained based on the augmented dataset after secondary annotation. The trained model is used for multimodal video emotion recognition. The model includes a bimodal feature extraction network and a classifier.

[0007] Preferred, ;in, These are the weighting coefficients.

[0008] Preferably, in step S1, the micro-motion tagging system is constructed as follows: first, the body parts are divided to obtain N coarse-grained body part tags; then, based on each body part, the micro-motions are further divided into M fine-grained micro-motion tags; N>M.

[0009] Preferably, the temporal micro-motion feature encoder is based on a network architecture of graph convolution temporal shift, including a temporal shift module and a graph convolution module. The temporal shift module realizes information exchange between adjacent frames by shifting part of the channel features in the time dimension, and the graph convolution module is used to process the body context of micro-motions and capture the spatial structural dependencies of micro-motions between different body parts.

[0010] Preferably, the facial macro-emotion encoder is based on the DeepFace network architecture, which extracts facial features through a sequence of convolutional layers, local connected layers, and fully connected layers.

[0011] Preferably, the video samples are preprocessed by performing face detection and cropping, face alignment, image size adjustment and format conversion, and then used as the input to the facial macro-emotion encoder; the RGB video frame sequence is extracted after video sampling and used as the input to the temporal micro-motion feature encoder.

[0012] Preferably, secondary annotation is performed on existing video datasets used for emotion recognition: Identify the start and end frames of micro-movements in a video, and label each micro-movement instance in the video frame sequence with a micro-movement tag; Micro-action labels are recorded synchronously with the existing emotion labels in the dataset to form a multi-level annotation.

[0013] The present invention also provides a readable storage medium having a computer program stored thereon, which, when executed, implements the aforementioned multimodal video emotion recognition method.

[0014] The present invention also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned multimodal video emotion recognition method.

[0015] The present invention also provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the aforementioned multimodal video emotion recognition method.

[0016] The advantages of this invention are: (1) This invention proposes a video emotion recognition method based on multimodality. By constructing a complete micro-action labeling system and optimizing dataset annotation, it introduces these non-facial and non-verbal micro-action cues to make up for the shortcomings of macro-facial recognition in describing the dynamic process of emotion. It also adopts an efficient posterior fusion strategy to fully combine the modal advantages of facial expressions and body micro-actions, thereby significantly improving the accuracy and robustness of video emotion recognition.

[0017] (2) This invention relates to an efficient and high-precision video emotion recognition method. This method combines macroscopic facial expression features with spatiotemporal micro-motion features based on temporal graph convolution. Through multimodal posterior fusion technology, it aims to overcome the limitations of traditional methods in capturing fine-grained micro-motions of human subconscious expression, thereby achieving more realistic and robust emotion analysis. It can be widely applied to scenarios with extremely high requirements for real-time performance and recognition accuracy, such as human-computer interaction (HCI), mental health assessment, behavior analysis, and intelligent security.

[0018] (3) By establishing and utilizing a micro-action tagging system, this invention achieves comprehensive capture of emotional cues from macroscopic facial features to microscopic body behavior, and the fine-grained behavioral representation it provides greatly enhances the authenticity of emotion recognition.

[0019] (4) The present invention adopts a graph convolutional temporal shift (GC-TSM) network architecture and utilizes the zero-parameter temporal modeling capability of the temporal shift module (TSM) to ensure that the system can maintain low latency when processing high-density temporal information (micro-actions), support real-time online recognition tasks, and have high-efficiency real-time processing capabilities.

[0020] (5) This invention introduces and optimizes the analysis of subconscious micro-movements, so that the recognition results can more realistically reflect the inner emotional state of humans, reduce the influence of individual subjective control factors, and have high authenticity of emotional perception.

[0021] (6) This invention cleverly combines the spatial stability of the facial macro-emotion encoder and the temporal dynamics of the temporal micro-motion feature encoder by directly summing and fusing the Logits posterior. This linear fusion strategy, which does not require complex training, achieves direct complementarity of the prediction results of the two modalities and improves the robustness of emotion recognition. Attached Figure Description

[0022] Figure 1 This is a flowchart of a video emotion recognition method based on multimodal fusion according to the present invention.

[0023] Figure 2 This is a model architecture diagram of a video emotion recognition method based on multimodal fusion according to the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention significantly improves the accuracy of emotion recognition, the ability to describe emotional dynamics, and the real-time processing efficiency of the system by deeply mining fine-grained micro-motion information in human nonverbal behavior and optimizing its fusion with traditional macro-facial expression information. Through innovations at the data layer, model layer, and fusion layer, this invention proposes a video emotion recognition method based on multimodal fusion, the core innovations of which include: Data Layer Innovation: Ultra-fine granular labeling system. A micro-action labeling system containing 82 categories of behavior was proposed and summarized. This system significantly improves the dimensionality and granularity of the representation of human behavior.

[0026] Data benchmark enhancement: Secondary annotation, based on 82 micro-action labels, high-precision secondary annotation of timestamps and micro-action categories was performed on the existing CAER dataset, constructing a multi-level, multi-granularity sentiment analysis benchmark.

[0027] Model layer innovation: A complementary dual-stream architecture was designed, which is a dual-modal feature extraction network based on a macro-facial emotion encoder (DeepFace Stream, used to extract macro-facial features) and a temporal micro-motion feature encoder (GC-TSM Stream, used to extract temporal micro-motion features).

[0028] Innovation in the fusion layer: Optimize Logits fusion by adopting a posterior Logits fusion strategy. The linear weighted fusion mechanism is used to balance the stability of the facial macro-emotion encoder and the temporal sensitivity of the temporal micro-motion feature encoder, thereby maximizing the complementary benefits of the two modalities.

[0029] Depend on Figure 1 and Figure 2 As shown, the video emotion recognition method based on multimodal fusion of the present invention specifically includes the following: S1, Construction of the micro-action tagging system and dataset processing.

[0030] (1) Definition and coding specifications of 82 micro-action tags

[0031] To overcome the limitations of traditional emotion recognition methods that rely solely on macroscopic facial expressions, this invention establishes a multi-level micro-motion tagging system covering multiple parts of the body. This system significantly surpasses traditional Facial Action Coding Systems (FACS) in terms of the dimensionality and granularity of behavioral representation, enabling the structured capture of extremely brief, unconscious human behaviors. The system is designed to provide high-dimensional behavioral representations that better reflect genuine human emotions. It divides body parts into seven coarse-grained tags, and based on these seven body part tags, further divides micro-movements into 82 fine-grained micro-motion tags.

[0032] The seven coarse-grained body part labels are shown below: A: Body; B: Head; C: Upper limb; D: Lower limb; E: Body-Hand; F: Head-Hand. G: Leg-Hand.

[0033] The 82 fine-grained micro-motion tags are shown in Table 1 below.

[0034] Table 1. Complete Micro-Motion Tagging System

[0035] The introduction of these micro-motion tags provides the model with the fine-grained temporal signals needed to describe the occurrence, development, and decay of emotions, thus fundamentally solving the inherent defects of macro-facial recognition in capturing emotional dynamics.

[0036] Table 2 below shows some examples of how micro-action tags are interpreted.

[0037] Table 2 Examples of interpreting micro-action tags

[0038] (2) Secondary labeling of the CAER dataset To train and validate this ultra-fine-grained model, this invention selected the CAER (Context-Aware Emotion Recognition) dataset for secondary annotation of timestamps and micro-motion categories, resulting in the CAER augmented dataset. The CAER dataset is widely popular due to its emphasis on "context-aware" emotion recognition; it not only includes faces but also provides rich background and body context information. This characteristic makes the CAER dataset an ideal basis for annotating the 82 micro-motion labels covering the entire body in this invention.

[0039] Labeling process: S1, Time Boundary Definition: Using the professional annotation tool Labelu, rigorously trained annotators accurately identify the start and end frames of each micro-motion instance in the video frame sequence. This high-precision temporal positioning is crucial for solving the problem of short duration micro-motions.

[0040] S2, Category Mapping: The 82 micro-action labels are synchronously recorded with the original macro-emotion labels in the CAER dataset to form a multi-level annotation. This includes 7 macro-emotion labels: anger, disgust, fear, happy, neutral, sad, and surprise.

[0041] S3, Data Preprocessing: For the input data of the facial macro-emotion encoder, after video sampling, preprocessing is performed such as face detection and cropping, face alignment, and image size adjustment and format conversion; for the input data of the temporal micro-motion feature encoder, after video sampling, RGB frame sequences are extracted and used as input for graph convolution.

[0042] By performing secondary annotation on the CAER dataset, this invention constructs a unique multi-granularity emotion benchmark, successfully linking fine-grained behavioral representation with macro-emotion prediction, and solving the problem of traditional datasets lacking body context and dynamic information.

[0043] S2, construct a dual-modal feature extraction network.

[0044] This invention employs a dual-stream architecture to ensure that the system can simultaneously capture both stable (macro) and dynamic (micro) characteristics of emotions, thereby achieving optimal complementarity. The overall structure is as follows: Figure 1 As shown.

[0045] (1) Facial macro-emotion encoder (DeepFace Stream, used to extract macro-facial features) The Facial Macro Emotion Encoder aims to provide a stable baseline for emotion recognition. Based on the DeepFace network architecture, it leverages the powerful spatial feature extraction capabilities of DeepFace.

[0046] The details are as follows: The facial macro-emotion encoder extracts high-dimensional facial features through a sequence of convolutional layers, locally connected layers, and fully connected layers.

[0047] Furthermore, detection and cropping alignment techniques are employed to ensure that the extracted facial features are highly robust to head pose changes and partial occlusion. Specifically, before feature extraction, facial detection algorithms are used to extract face bounding boxes and locate facial key points (such as eyes, nose, and mouth). Subsequently, affine transformations or 3D alignment techniques based on facial key points are used for face cropping and spatial alignment. This standardized preprocessing mechanism ensures that the spatial features input to the DeepFace network have high consistency, thereby significantly enhancing the model's robustness to head pose changes and partial facial occlusion in videos.

[0048] The final layer output of the facial macro-emotion encoder is the raw Logits vector for facial emotion classification. ( Here, the number of emotion categories is... ).because It relies on macroscopic, slowly changing facial expressions, and may be relatively sluggish during periods of emotional outburst.

[0049] (2) Temporal micro-motion feature encoder (GC-TSM Stream, used to extract temporal micro-motion features) The temporal micro-motion encoder is specifically designed to efficiently and accurately capture the temporal dynamics of micro-motions. Based on a graph convolutional temporal shifter (GC-TSM) network architecture, it includes a temporal shifting module and a graph convolution module to meet the stringent requirements of high temporal resolution and low computational latency for micro-motions.

[0050] The details are as follows: Temporal Modeling: The Temporal Shift Module (TSM) achieves information exchange between adjacent frames by shifting a portion of the channel features along the temporal dimension. Specifically, for the input video frame feature map, the TSM performs misalignment alignment of feature channels along the temporal dimension according to a preset ratio (e.g., shifting 1 / 4 of the channels to past frames, shifting another 1 / 4 to future frames, and keeping the remaining 1 / 2 channels unchanged). The misaligned feature map is then input into a standard 2D convolutional layer for spatial feature extraction. The key advantage of this mechanism is that it enables 2D convolutional layers to capture temporal dynamic changes without increasing computational overhead or the number of parameters. This allows the temporal micro-motion feature encoder to maintain extremely low latency (e.g., up to 13ms on embedded devices), a fundamental guarantee for achieving online emotion recognition.

[0051] Spatial Modeling: The Graph Convolution Module (GCM) processes the body context of micro-movements, capturing the spatial structural dependencies between body parts. First, key points of the human skeleton in video frames are extracted using human pose estimation techniques, serving as nodes in the spatiotemporal graph. Then, an adjacency matrix is ​​constructed based on the body's natural physical connectivity (or non-physical dependencies learned through self-attention mechanisms), forming edges in the spatiotemporal graph. The GCM performs information aggregation on the constructed skeleton topology graph, propagating and updating node features along the body's physical structure. Since the 82 micro-movement labels involve the torso and limbs, this topology-based feature aggregation ensures the model can accurately focus on and understand the fine-grained motion patterns of the coordinated operation of various body parts.

[0052] The temporal micro-motion feature encoder outputs the raw Logits vector for micro-motion emotion classification. ( Here, the number of emotion categories is... ).

[0053] The introduction of Graph Convolutional Temporal Shifting (GC-TSM) transforms micro-motion modeling into spatiotemporal graph sequence analysis, greatly improving the model's ability to capture the dynamics and accuracy of micro-motions, while avoiding the high computational cost of traditional 3D CNNs.

[0054] (3) Feature synchronization For effective fusion, the Logits outputs of the two modalities must be synchronized and aligned in the temporal dimension. The facial macro-emotion encoder processes the face, while the temporal micro-motion feature encoder processes the whole body (or a specific region). Their outputs... and The data needs to be aggregated and mapped within the same video time window in order to proceed to the next stage of fusion processing.

[0055] S3, construct a classifier and perform multimodal posterior logits fusion.

[0056] The key to this invention lies in using a late fusion strategy for Logits fusion, so as to fully utilize the complementarity of macroscopic facial modalities and temporal micro-motion modalities, while avoiding noise interference that may be caused by fusion at the feature level.

[0057] (1) Necessity of fusion strategy: modal complementarity Logits fusion occurs before the decision layer, allowing both modalities to independently optimize their internal feature representations. Since the micro-motion features captured by the temporal micro-motion feature encoder are noisy and sparsity, fusing them too early with the stable macro-features extracted by the facial macro-emotion encoder can lead to feature contamination and performance degradation. Posterior fusion strategies, on the other hand, allow the system to learn how to dynamically weigh the contributions from the two independent predictions.

[0058] (2) Mathematical model of Logits fusion This invention employs a direct linear addition fusion mechanism. Both modal encoders are trained to the output... The emotion Logits vector of dimension can be directly summed by adding the corresponding elements.

[0059] Final fused Logits vector for: ; in, represents the weighting coefficient. In this embodiment, In this fusion mechanism, macro-level facial emotional information and micro-action timing information The contributions to the final decision are considered equal. This direct summation fusion strategy aims to leverage the predictions from two independent modalities to directly enhance the final classification performance in a complementary manner.

[0060] (3) Based on fused Logits vector Output the final emotion category.

[0061] S4. The model is trained based on the CAER augmented dataset after secondary annotation. The trained model is used for multimodal video emotion recognition. The model includes a bimodal feature extraction network and a classifier.

[0062] Example 1: Experimental Verification and Effect Analysis (1) Experimental setup and evaluation indicators

[0063] Dataset: CAER augmented dataset, containing macro sentiment labels and micro-motion labels with precise time boundaries.

[0064] Evaluation metrics: accuracy of emotion classification and weighted F1 score.

[0065] (2) Ablation experiment and performance comparison By comparing different modal combinations and fusion strategies, the advantages of this invention are quantified. The experimental results are shown in Table 3 below: Table 3 Performance Comparison of Dual-Modal Fusion Systems

[0066] According to Table 3: Advantages of micro-motion features: Under the experimental setup and dataset, the GC-TSM baseline model based on micro-motions significantly outperforms the DeepFace baseline model based solely on macro-facial features. This result strongly demonstrates that fine-grained micro-motion information is a key, high-value cue for emotion recognition, effectively compensating for the limitations of macro-facial expression recognition.

[0067] Effectiveness of Logits fusion: Although the GC-TSM stream has achieved leading performance, the final fusion model achieves optimal performance by directly adding and fusing the Logits vectors of the two modalities. The performance of the fused model is slightly higher than that of the standalone GC-TSM stream and significantly better than that of the standalone DeepFace stream. This improvement verifies that although macroscopic facial features have weak individual recognition capabilities, they can still serve as stable and complementary signals at the decision-making level, providing calibration for time-sensitive micro-motion features, thereby maximizing overall recognition robustness.

[0068] (3) Robustness and application advantages analysis This invention demonstrates exceptional robustness in complex application scenarios. Because the GC-TSM stream focuses on body posture and micro-movements, its feature extraction is far less sensitive to changes in ambient lighting and facial occlusion (such as wearing a mask) than pure facial recognition methods. Even when facial features are unstable or damaged, the system can still utilize micro-movement information to ensure stable system performance.

[0069] Meanwhile, since the temporal shifting module of the temporal micro-motion feature encoder achieves efficient temporal modeling, it avoids the huge computational overhead brought by traditional 3D convolution or complex recurrent neural networks (RNN). While maintaining high accuracy, this invention can still be applied to real-time application scenarios that require low latency response.

[0070] In summary, this invention successfully constructs an efficient, accurate, and robust multimodal video emotion recognition framework through collaborative innovation at the data and model levels. Experimental results show that the posterior Logits fusion strategy can significantly improve emotion classification performance, verifying a strong complementary relationship between macroscopic facial features and fine-grained micro-motion features.

[0071] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0072] The facial recognition technology involved in this patent application strictly complies with the provisions of the *Personal Information Protection Law of the People's Republic of China*, the *Cybersecurity Law of the People's Republic of China*, the *Data Security Law of the People's Republic of China*, and relevant national standards. All technical implementations are based on the principles of legality, legitimacy, and necessity. In the data collection stage, we obtain facial information through explicit notification and user authorization, ensuring that collection does not exceed the permitted scope. In the data processing stage, we employ de-identification and encrypted storage technologies to protect the original biometric data, ensuring data security during transmission, storage, and use. We promise not to use the relevant data for any purpose unrelated to identifying natural persons, nor will we share or publicly disclose it to any third party, except as otherwise provided by law. Furthermore, the system design follows the principle of minimum necessity to avoid undue infringement on public privacy rights. All algorithms have undergone ethical review, eliminating discriminatory judgments based on race, religion, gender, or other characteristics. We have established a comprehensive data security management system, including access control, operation log auditing, and data leakage emergency plans, to fully protect user rights and social morality.

Claims

1. A video emotion recognition method based on multimodal methods, characterized in that, include: S1. Construct a micro-action labeling system. Based on the micro-action labeling system, perform secondary annotation on the existing video datasets used for emotion recognition to label micro-actions. S2, construct a dual-modal feature extraction network, including a facial macro-emotion encoder for extracting macro-facial features and a temporal micro-motion feature encoder for extracting temporal micro-motion features; The facial macro-emotion encoder takes a face from a video and outputs a Logits vector for facial emotion classification. ;in, Number of emotion categories; The temporal micro-motion feature encoder takes the body in the video and outputs Logits vectors for micro-motion emotion classification. ; Among them, the Logits vectors output by the facial macro-emotion encoder and the temporal micro-motion feature encoder are for the same video time window; S3, Construct a classifier for... and Perform bimodal posterior logits fusion to obtain the final fused logits vector. Based on fused Logits vector Output the final emotion category; S4. The model is trained based on the augmented dataset after secondary annotation. The trained model is used for multimodal video emotion recognition. The model includes a bimodal feature extraction network and a classifier.

2. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, ;in, These are the weighting coefficients.

3. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, In step S1, the micro-motion tagging system is constructed as follows: first, the body parts are divided to obtain N coarse-grained body part tags; then, based on each body part, the micro-motions are further divided into M fine-grained micro-motion tags; N>M.

4. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, The temporal micro-motion feature encoder is based on a network architecture of graph convolution temporal shift, which includes a temporal shift module and a graph convolution module. The temporal shift module realizes information exchange between adjacent frames by shifting part of the channel features in the time dimension, and the graph convolution module is used to process the body context of micro-motions and capture the spatial structural dependencies of micro-motions between different body parts.

5. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, The facial macro-emotion encoder is based on the DeepFace network architecture and extracts facial features through a sequence of convolutional layers, local connected layers, and fully connected layers.

6. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, After preprocessing the video sampling, including face detection and cropping, face alignment, image size adjustment, and format conversion, the results are used as input to the facial macro-emotion encoder. The RGB video frame sequence extracted after video sampling is used as input to the temporal micro-motion feature encoder.

7. The video emotion recognition method based on multimodal processing according to claim 1, characterized in that, Secondary annotation of existing video datasets used for emotion recognition: Identify the start and end frames of micro-movements in a video, and label each micro-movement instance in the video frame sequence with a micro-movement tag; Micro-action labels are recorded synchronously with the existing emotion labels in the dataset to form a multi-level annotation.

8. A readable storage medium, characterized in that, It stores a computer program, which, when executed, implements a multimodal video emotion recognition method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a multimodal video emotion recognition method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements a multimodal video emotion recognition method as described in any one of claims 1 to 7.

Images