Facial micro-expression analysis method based on space-time multi-mode large language model
By integrating micro-expression localization and recognition tasks using a spatiotemporal multimodal large language model, the problem of modeling framework separation in existing technologies is solved, improving the robustness and flexibility of micro-expression analysis and supporting multi-task expansion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGZHI UNIV
- Filing Date
- 2025-11-19
- Publication Date
- 2026-05-12
AI Technical Summary
Existing micro-expression analysis methods lack a unified modeling framework, resulting in the separation of micro-expression localization and recognition tasks. This makes it difficult to solve the problem efficiently within a unified framework. Furthermore, they are easily affected by changes in ambient lighting, occlusion, and individual differences, resulting in insufficient robustness and difficulty in extending to multimodal tasks.
We adopt a spatiotemporal multimodal large language model approach, which fuses visual features and optical flow information through a multimodal input layer, extracts cross-frame emotion changes using a spatiotemporal Transformer and a multimodal fusion module, and achieves joint modeling of localization and recognition through a large language model interaction layer, sharing underlying features to improve robustness.
It achieves unified modeling for micro-expression localization and recognition tasks, improves the ability to capture micro-expressions, reduces the dependence on accurate peak frame detection, supports multi-task expansion, and has good application flexibility.
Smart Images
Figure CN122024293A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of facial micro-expression analysis technology, specifically to a facial micro-expression analysis method based on a spatiotemporal multimodal large language model. Background Technology
[0002] With the rapid development of artificial intelligence and human-computer interaction, society is gradually entering a highly intelligent era. Intelligent human-computer interaction not only requires robots to complete predetermined tasks, but also to possess human-like emotional cognition, expression, and feedback capabilities during the interaction process. Psychological research shows that over 50% of the information in human emotional expression comes from facial expressions.
[0003] Facial expressions are categorized into macro-expressions and micro-expressions. Compared to macro-expressions, micro-expressions are characterized by their short duration and small muscle movement amplitude, often revealing an individual's true emotions. However, existing micro-expression analysis methods typically decompose the task into two stages: micro-expression localization and micro-expression recognition. First, segments in a video that may contain micro-expressions are located, and then keyframes within those segments are identified. This staged approach has limitations: micro-expression muscle movements are highly coupled with individual identity, and the motion signals are weak, leading to a separation between the localization and recognition tasks, making it difficult to efficiently solve within a unified framework.
[0004] Existing micro-expression analysis methods primarily employ staged strategies based on optical flow, feature point matching, or convolutional neural networks, separating micro-expression localization from recognition. For example, in the baseline schemes of MEGC 2024 and 2025, localization and recognition must be performed independently, lacking a unified modeling framework; feature extraction heavily relies on accurate temporal frame annotation and peak frame detection, making it susceptible to changes in ambient lighting, occlusion, and individual differences. Therefore, these methods have significant limitations: on the one hand, task separation leads to low information utilization, with a lack of sharing mechanisms between localization and recognition; on the other hand, due to the small amplitude of micro-expression muscle movements, staged methods lack robustness and are difficult to extend to more complex multimodal tasks such as visual question answering, limiting overall recognition accuracy and application flexibility. Summary of the Invention
[0005] The purpose of this invention is to provide a facial micro-expression analysis method based on a spatiotemporal multimodal large language model, which solves the problems of lacking a unified modeling framework and being easily affected by changes in ambient lighting, occlusion, and individual differences.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A facial micro-expression analysis method based on a spatiotemporal multimodal large language model includes the following steps: model construction, localization and recognition, motion feature extraction, and output. S1 Model Construction: The model structure consists of a multimodal input layer, a spatiotemporal coding layer, and a large language model interaction layer. The multimodal input layer integrates visual features and optical flow information. The spatiotemporal coding layer uses a spatiotemporal Transformer and a multimodal fusion module to extract local and global sentiment changes across frames. The large language model interaction layer transforms features into linguistic semantic embeddings to achieve joint modeling of localization and recognition. S2 Localization and Recognition: Input micro-expression video sequences or keyframe pairs to complete the task of micro-expression localization and recognition. In the localization stage, optical flow or emotional changes of two frames in the video sequence are used as input, and the inter-frame differences are used as the audio-visual model output. S3 extracts motion features: It extracts micro-expression video sequences or key frame pairs from the output audio-visual model, and extracts motion features between two frames through optical flow calculation, while performing face detection and alignment. S4 Output: The model extracts motion features based on S3 and runs a multi-task output layer that provides both localization and recognition task outputs. The localization task output refers to the inter-frame difference representation, and the recognition task output refers to the micro-expression emotion category. During training, a multi-task joint strategy is adopted to share the underlying features.
[0007] The beneficial effects of this invention are as follows: This application integrates the localization and recognition tasks of micro-expressions through a unified multimodal large language model framework, avoiding the separation problem of traditional staged methods; at the same time, by utilizing spatiotemporal optical flow features and multimodal inputs, it improves the ability to capture micro-expressions and reduces the dependence on accurate peak frame detection; in addition, this method supports multi-task extensions such as visual question answering, and has good application scalability and flexibility.
[0008] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description
[0009] Figure 1 This is a flowchart illustrating the overall process of an embodiment of the present invention. Detailed Implementation
[0010] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0011] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0012] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0013] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0014] Please see Figure 1 A preferred embodiment of this application illustrates a facial micro-expression analysis method based on a spatiotemporal multimodal large language model, comprising the following steps: model construction, localization and recognition, motion feature extraction, and output. S1 Model Construction: The model structure consists of a multimodal input layer, a spatiotemporal coding layer, and a large language model interaction layer. The multimodal input layer integrates visual features and optical flow information. The spatiotemporal coding layer uses a spatiotemporal Transformer and a multimodal fusion module to extract local and global sentiment changes across frames. The large language model interaction layer transforms features into linguistic semantic embeddings to achieve joint modeling of localization and recognition. S2 Localization and Recognition: Input micro-expression video sequences or keyframe pairs to complete the task of micro-expression localization and recognition. In the localization stage, optical flow or emotional changes of two frames in the video sequence are used as input, and the inter-frame differences are used as the audio-visual model output. S3 extracts motion features: It extracts micro-expression video sequences or key frame pairs from the output audio-visual model, and extracts motion features between two frames through optical flow calculation, while performing face detection and alignment. S4 Output: The model extracts motion features based on S3 and runs a multi-task output layer that provides both localization and recognition task outputs. The localization task output refers to the inter-frame difference representation, and the recognition task output refers to the micro-expression emotion category. During training, a multi-task joint strategy is adopted to share the underlying features.
[0015] In summary, this invention provides a facial micro-expression analysis method based on a spatiotemporal multimodal large language model. This method integrates the localization and recognition tasks of micro-expressions through a unified multimodal large language model framework, avoiding the separation problem of traditional staged methods. At the same time, by utilizing spatiotemporal optical flow features and multimodal inputs, it improves the ability to capture micro-expressions and reduces the dependence on accurate peak frame detection. In addition, this method supports multi-task extensions such as visual question answering, and has good application scalability and flexibility.
[0016] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0017] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.
Claims
1. A facial micro-expression analysis method based on a spatiotemporal multimodal large language model, characterized in that, The process includes the following steps: model construction, localization and recognition, motion feature extraction, and output. S1 Model Construction: The model structure consists of a multimodal input layer, a spatiotemporal coding layer, and a large language model interaction layer. The multimodal input layer integrates visual features and optical flow information. The spatiotemporal coding layer uses a spatiotemporal Transformer and a multimodal fusion module to extract local and global sentiment changes across frames. The large language model interaction layer transforms features into linguistic semantic embeddings to achieve joint modeling of localization and recognition. S2 Localization and Recognition: Input micro-expression video sequences or keyframe pairs to complete the task of micro-expression localization and recognition. In the localization stage, optical flow or emotional changes of two frames in the video sequence are used as input, and the inter-frame differences are used as the audio-visual model output. S3 extracts motion features: It extracts micro-expression video sequences or key frame pairs from the output audio-visual model, and extracts motion features between two frames through optical flow calculation, while performing face detection and alignment. S4 Output: The model extracts motion features based on S3 and runs a multi-task output layer that provides both localization and recognition task outputs. The localization task output refers to the inter-frame difference representation, and the recognition task output refers to the micro-expression emotion category. During training, a multi-task joint strategy is adopted to share the underlying features.