Micro-expression recognition method and device based on local-global time relationship and medium

Abstract

The application discloses a micro-expression recognition method and device based on local-global time relationship and a medium, and relates to the technical field of video processing. The method comprises the following steps: decomposing a micro-expression video into a micro-expression image sequence; preprocessing the micro-expression image sequence; constructing a multi-scale space-time feature extraction module for extracting local spatial features and local time features; constructing an improved double-flow global time sequence learning module for extracting global space-time features of the micro-expression; constructing a cross-modal global time relationship learning module for fusing the global space-time features and modeling the relationship between the time features and the spatial features by using a multi-head self-attention mechanism; constructing and training a local-global time relationship network; preprocessing a micro-expression video to be recognized and inputting the trained local-global time relationship network to recognize the micro-expression category. The application has high automation, strong robustness and can fully learn the local-global time relationship.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, more particularly, to a micro-expression recognition method and device based on local-global temporal relationship and a medium. BACKGROUND

[0002] In the field of micro-expression recognition, existing methods can be roughly divided into two categories: key frame based on manual annotation and video sequence based. These methods have different input forms, leading to differences in dynamic feature extraction and model design.

[0003] Based on manually annotated key frame images, researchers mainly use the frame with the strongest expression, i.e. the peak frame, to identify micro-expression categories. In order to ensure the accuracy of classification, these methods usually require researchers to manually annotate each video, marking key frames such as the starting frame, peak frame and ending frame. This process not only consumes a lot of time and manpower, but also depends on the quality and consistency of the annotation. In addition, since only the peak frame is focused on and other frames are ignored, the subtle changes of micro-expression in the time dimension may be lost, limiting the accuracy of image-based micro-expression recognition methods.

[0004] In recent years, with the development of video understanding, researchers have gradually turned to video sequence based methods, trying to model the temporal relationship of micro-expression videos. These methods usually use deep learning frameworks such as convolutional neural networks (CNN) or long short-term memory networks (LSTM) to extract spatio-temporal features in videos to enhance the performance of micro-expression recognition. However, these methods usually have weak dependence on long sequence relationship modeling, and are easily affected by sequence length and frame rate changes.

[0005] Therefore, it has become a core problem to be solved to develop a micro-expression recognition method with high automation, strong robustness and sufficient learning of local and global temporal relationship. SUMMARY

[0006] To solve the above technical problems, the present application provides a micro-expression recognition method based on local-global temporal relationship, device and medium to overcome the shortcomings of traditional methods in low intensity frames and global temporal relationship representation.

[0007] In a first aspect, the present application provides a micro-expression recognition method based on local-global temporal relationship, which comprises:

[0008] decomposing a micro-expression video into a sequence of micro-expression images;

[0009] preprocessing the sequence of micro-expression images to obtain the effective face area of each frame image;

[0010] The multiscale spatiotemporal feature extraction module is configured to: extract spatial features of each frame in the image sequence as local spatial features by using a macro-expression pre-training model; segment the micro-expression image sequence into multiple time segments with overlap through a time sliding window; and extract dynamic change features of the time segments in a local time range as local time features by using a multi-phase optical flow method;

[0011] The improved dual-flow global temporal learning module is configured to extract global features of the local spatial features and the local time features on a time axis, so as to obtain global spatiotemporal features of the micro-expression;

[0012] The cross-modal global temporal relationship learning module is configured to fuse the global spatiotemporal features and model the relationship between the time features and the spatial features by using a multi-head self-attention mechanism, so as to comprehensively improve the recognition accuracy of the micro-expression video;

[0013] The multiscale spatiotemporal feature extraction module, the improved dual-flow global temporal learning module, and the cross-modal global temporal relationship learning module are combined to form a local-global temporal relationship network, the local-global temporal relationship network is trained, and the best weight model is saved as a trained local-global temporal relationship network;

[0014] The micro-expression video to be recognized is preprocessed and input into the trained local-global temporal relationship network, and a micro-expression category is recognized.

[0015] Further, the micro-expression image sequence is preprocessed to obtain an effective face region of each frame of image, including:

[0016] Face detection and feature point detection are performed on each frame of image in the micro-expression image sequence;

[0017] The detected face features are aligned by using an affine transformation;

[0018] Based on the inner canthus of the left eye of each frame of face, an effective face region with consistent size is cropped.

[0019] Further, the macro-expression pre-training model is used to extract spatial features of each frame in the image sequence as local spatial features; the micro-expression image sequence is segmented into multiple time segments with overlap through a time sliding window; and the multi-phase optical flow method is used to extract dynamic change features of the time segments in a local time range as local time features, including:

[0020] Each frame of image is input frame by frame, the macro-expression pre-training DAN model is used to capture facial local details related to expressions, and the face pre-training ArcFace model is used to capture face appearance and posture encoding information as local spatial features;

[0021] A sliding window with length t and stride s is designed to slide along the time axis, and the micro-expression image sequence is divided into overlapping time segments with a fixed number of frames,

[0022] The dynamic features of each micro-expression segment are calculated using the multi-phase optical flow method. The horizontal optical flow, vertical optical flow, and optical flow strain between each frame and the first frame of each segment are calculated to obtain the multi-phase optical flow changes from short to long as the local temporal features.

[0023] Further, the total number of segmented time segments is represented as:

[0024]

[0025] where T is the total number of frames of the micro-expression video, and τ is the total number of segments.

[0026] Further, the global features of the local spatial features and the local temporal features on the time axis are extracted to obtain the global spatio-temporal features of the micro-expression, including:

[0027] The multi-phase horizontal optical flow, multi-phase vertical optical flow, and multi-phase optical flow strain are stacked into a three-channel optical flow map, and the macro-expression features and face appearance features are stacked into two-channel spatial features.

[0028] TCN channels are designed for the optical flow map and spatial features, and the optical flow map and spatial features are input into the corresponding TCN channels to capture their global temporal features.

[0029] Further, the global spatio-temporal features are fused, and a multi-head self-attention mechanism is used to model the relationship between the temporal features and the spatial features, including:

[0030] A micro-expression video encoder based on visual Transformer is constructed, and the global temporal features of each frame of the micro-expression segment are regarded as sequence elements to extract the feature sequence.

[0031] The feature sequence is input into the Transformer encoder of the multi-head self-attention mechanism, which simultaneously captures the information of different positions within the frame sequence, learns the cross-frame dependency relationship, and the relationship between the local frame and the global micro-expression video.

[0032] The output features of the Transformer encoder are input into a classification network composed of two fully connected layers and one batch normalization layer, and the Softmax function is used to complete the final classification of the micro-expression categories.

[0033] In a second aspect, the present application provides a micro-expression recognition device based on local-global temporal relationship, the device comprising:

[0034] a video decomposition module configured to decompose a micro-expression video into a micro-expression image sequence;

[0035] a preprocessing module configured to preprocess the micro-expression image sequence to obtain an effective face region of each frame image;

[0036] a local feature extraction module configured to construct a multi-scale spatio-temporal feature extraction module, wherein the multi-scale spatio-temporal feature extraction module is configured to extract spatial features of each frame in the image sequence as local spatial features by using a macro-expression pre-training model; segment the micro-expression image sequence into multiple time segments with overlap by using a time sliding window; and extract dynamic change features of the time segments in a local time range as local temporal features by using a multi-phase optical flow method;

[0037] a global feature extraction module configured to construct an improved dual-flow global temporal learning module for extracting global features of the local spatial features and the local temporal features on a time axis to obtain global spatio-temporal features of the micro-expression;

[0038] a feature fusion module configured to construct a cross-modal global temporal relationship learning module for fusing the global spatio-temporal features and modeling the relationship between the temporal features and the spatial features by using a multi-head self-attention mechanism, so as to comprehensively improve the recognition accuracy of the micro-expression video;

[0039] a network training module configured to combine the multi-scale spatio-temporal feature extraction module, the improved dual-flow global temporal learning module, and the cross-modal global temporal relationship learning module to form a local-global temporal relationship network, train the local-global temporal relationship network, and save a weight model with the best performance as a trained local-global temporal relationship network;

[0040] a micro-expression recognition module configured to preprocess a micro-expression video to be recognized, input the trained local-global temporal relationship network, and recognize a micro-expression category.

[0041] Further, the preprocessing module is further configured to:

[0042] perform face detection and feature point detection on each frame image in the micro-expression image sequence;

[0043] align the detected face features by using an affine transformation;

[0044] crop an effective face region with a consistent size based on an inner corner of a left eye of each frame face.

[0045] Further, the local feature extraction module is further configured to:

[0046] Each frame image is input frame by frame, a macro-expression pre-training DAN model is used to capture facial local details related to expression, and a face pre-training ArcFace model is used to capture face appearance and posture encoding information as local spatial features;

[0047] A sliding window with a length of t and a step of s is designed, the micro-expression image sequence is segmented into overlapping time segments with a fixed number of frames along the time axis,

[0048] A multi-phase optical flow method is used to calculate the dynamic features of each micro-expression segment, the horizontal optical flow, the vertical optical flow and the optical flow strain between each frame image and the first frame of each segment are calculated, and the multi-phase optical flow changes from short to long are obtained as local time features.

[0049] In a third aspect, the present application provides a readable storage medium, the readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the method as described above.

[0050] The present application has at least the following beneficial effects:

[0051] The present application aims at the problems of low recognition accuracy and dependence on a large number of labeled images in the image-based micro-expression recognition method. First, through the synergistic effect of the multi-scale spatio-temporal feature extraction module and the dual-flow independent time sequence modeling, an average F1 value of 91.55% is achieved on the CASMEII dataset, which is 21.29% higher than the traditional LBP-TOP method. Secondly, the multi-phase optical flow calculation method effectively obtains stable facial change trend. Finally, the cross-modal global time relationship learning module can learn the video time relationship, avoiding the problem of a large number of manually labeled key frames. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of a micro-expression recognition method based on local-global time relationship according to an embodiment of the present application is shown;

[0053] Figure 2 A flowchart of pre-processing of a micro-expression image sequence according to an embodiment of the present application is shown;

[0054] Figure 3 A multi-scale spatio-temporal feature extraction flowchart according to an embodiment of the present application is shown;

[0055] Figure 4 A schematic diagram of a time sliding window method according to an embodiment of the present application is shown;

[0056] Figure 5 A multi-phase optical flow method calculation schematic diagram in a micro-expression recognition method based on local-global time relationship according to an embodiment of the present application is shown;

[0057] Figure 6 A global spatio-temporal feature extraction flowchart according to an embodiment of the present application is shown;

[0058] Figure 7 A global spatio-temporal feature fusion flowchart according to an embodiment of the present application is shown;

[0059] Figure 8 A structural diagram of a local-global temporal relationship network according to an embodiment of the present application is shown;

[0060] Figure 9 A structural diagram of a local-global temporal relationship-based micro-expression recognition device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0061] To enable a person skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. The embodiments of the present application will be further described in detail below in combination with the drawings and specific embodiments, but not as a limitation on the present application. The order in which each step is described herein as an example should not be considered as a limitation, and a person skilled in the art should know that the order can be adjusted, as long as the logic between them is not destroyed and the entire process cannot be implemented.

[0062] The embodiments of the present application provide a local-global temporal relationship-based micro-expression recognition method, as shown in Figure 1 The local-global temporal relationship-based micro-expression recognition method includes the following steps S100-S700.

[0063] S100, decompose the micro-expression video into a micro-expression image sequence.

[0064] In this embodiment, the micro-expression video refers to the video data containing a face collected by a video collection device, which can be real-time collected video data or historically collected video data. The video collection device includes but is not limited to a video camera or a screen recording software device. As an example, the way of decomposing the micro-expression video into a micro-expression image sequence includes but is not limited to using video software processing and / or using a Python script. The purpose of decomposition is to extract images frame by frame and save them as an image sequence for subsequent step processing.

[0065] S200, pre-process the micro-expression image sequence to obtain the effective face area of each frame image.

[0066] In some embodiments, as Figure 2As shown, the micro-expression image sequence is preprocessed to obtain an effective face region of each frame image, including the following steps:

[0067] S201, face detection and feature point detection are performed on each frame image in the micro-expression image sequence.

[0068] The purpose of step S201 is to locate the face in each frame image and detect the feature points such as eyes, nose and corners of the mouth. Step S201 can be processed by using existing tools, such as Dlib (classic HOG+SVM model, 68-point detection), MediaPipe (lightweight, 468 / 478-point detection, strong real-time performance) and MTCNN (deep learning, suitable for complex angles / lighting).

[0069] S202, affine transformation is used to align the detected facial features.

[0070] The purpose of step S202 is to align the face by key points and eliminate pose changes (such as rotation, translation, etc.). An exemplary step of using affine transformation to align the detected facial features is as follows:

[0071] S2021, selecting a reference point: selecting a target position as a reference; wherein the target position can be one or a combination of the positions of the eyes, the tip of the nose and the corners of the mouth;

[0072] S2022, calculating an affine matrix: mapping the current facial key points to a standard position;

[0073] S2023, applying transformation: rotating / translation of the image to align the features.

[0074] S203, based on the inner corner of the left eye of each frame face, an effective face region with consistent size is cropped.

[0075] Through step S203, a feature image containing an effective face region can be obtained. These feature images can be combined in the original time sequence to obtain a preprocessed micro-expression image sequence, which will serve as the basis for subsequent step S300.

[0076] Exemplarily, based on the inner corner of the left eye of each frame face, an effective face region with consistent size is cropped as follows:

[0077] S2031, locating the inner corner of the left eye: determining the base point according to the left eye key point in the feature point, and taking the distance between the centers of the two eyes as the reference distance.

[0078] S2032, calculating the cropping region: taking the left eye as the center, extending twice the eye distance in the horizontal positive direction, extending one time the eye distance in the horizontal negative direction; extending one time the eye distance in the vertical positive direction, and extending twice the eye distance in the vertical negative direction.

[0079] S2033, boundary check: ensure that the clipping area does not exceed the image range.

[0080] S300, construct a multi-scale spatio-temporal feature extraction module, wherein the multi-scale spatio-temporal feature extraction module is configured to: extract spatial features of each frame in the image sequence as local spatial features by using a macro-expression pre-training model; segment the micro-expression image sequence into a plurality of time segments with overlap by using a time sliding window; and extract dynamic change features of the time segments in a local time range as local time features by using a multi-phase optical flow method.

[0081] For example, the multi-scale spatio-temporal feature extraction module takes the micro-expression image sequence preprocessed in step S200 as input, adopts the DAN model and the ArcFace model pre-trained on the macro-expression dataset as a spatial feature extractor based on a transfer learning strategy, and outputs a 512-dimensional spatial feature vector; a time sliding window is designed to divide the sequence into N local time segments by customizing the overlap rate (0%-100%); a TV-L1 optical flow is used to extract multi-phase optical flow features to extract dynamic change features of the micro-expression segments in a local time range.

[0082] In some embodiments, as shown in Figure 3 S300 includes:

[0083] S301, input each frame of image frame by frame, capture expression-related facial local details by using a macro-expression pre-trained DAN model, and capture facial appearance and posture encoding information by using a face pre-trained ArcFace model, as local spatial features.

[0084] S302, design a sliding window with a length of t and a step of s, slide along the time axis, and segment the micro-expression image sequence into overlapping time segments with a fixed number of frames. A schematic diagram of the time sliding window method is shown in Figure 4 .

[0085] In some embodiments, the total number of segmented time segments is represented as:

[0086]

[0087] wherein T is the total number of frames of the micro-expression video, and τ is the total number of segments.

[0088] S303, calculate the dynamic features of each micro-expression segment by using a multi-phase optical flow method, take the first frame of each segment as a reference frame, calculate the horizontal optical flow, the vertical optical flow and the optical flow strain between each frame of image and the first frame, obtain the multi-phase optical flow changes from short to long as local time features, and the specific calculation method is shown in Figure 5 .

[0089] S400, construct an improved double-flow global timing learning module for extracting global features of local spatial features and local time features on a time axis to obtain global space-time features of micro expressions.

[0090] In some embodiments, as shown in Figure 6 extracting global features of local spatial features and local time features on a time axis to obtain global space-time features of micro expressions includes:

[0091] S401, stack multi-period horizontal optical flow, multi-period vertical optical flow and multi-period optical flow strain into a three-channel optical flow graph, and stack macro-expression features and face appearance features into two-channel spatial features.

[0092] S402, design a TCN channel for the optical flow graph and the spatial features, and input the optical flow graph and the spatial features into the corresponding TCN channel to capture global timing features thereof.

[0093] S500, construct a cross-modal global timing relationship learning module for fusing global space-time features and modeling the relationship between time features and spatial features by using a multi-head self-attention mechanism, thereby comprehensively improving the recognition accuracy of micro-expression videos.

[0094] In some embodiments, as shown in Figure 7 fuse global space-time features and model the relationship between time features and spatial features by using a multi-head self-attention mechanism, including:

[0095] S501, construct a micro-expression video encoder based on a visual Transformer, treat global timing features of each frame of a micro-expression segment as sequence elements, and extract feature sequences;

[0096] S502, input the feature sequences into a Transformer encoder of the multi-head self-attention mechanism, simultaneously capture information at different positions within the frame sequences by using the multi-head self-attention mechanism, learn cross-frame dependency relationships and the relationship between local frames and global micro-expression videos, and

[0097] S503, input output features of the Transformer encoder into a classification network composed of two fully connected layers and one batch normalization layer, and use a Softmax function to complete final classification of micro-expression categories.

[0098] S600, combine the multi-scale space-time feature extraction module, the improved double-flow global timing learning module and the cross-modal global timing relationship learning module to form a local-global timing relationship network, train the local-global timing relationship network, and save the best weight model as a trained local-global timing relationship network.

[0099] As shown in the embodiment, as shown in Figure 8 The structure diagram of the local-global temporal relationship network, which is composed of three modules: a multi-scale spatio-temporal feature extraction module, a dual-flow independent time sequence modeling module, and a cross-modal relationship learning module.

[0100] S700, preprocessing the micro-expression video to be identified, and inputting the trained local-global temporal relationship network to identify the micro-expression category.

[0101] It should be noted that the way of preprocessing the micro-expression video to be identified is the operation process of steps S100 and S200 described in the foregoing, that is, the micro-expression video to be identified is decomposed, and the effective face area of the decomposed micro-expression image sequence is extracted.

[0102] The embodiment of the application also provides a micro-expression recognition device based on local-global temporal relationship, as shown in Figure 9 The device comprises:

[0103] A video decomposition module 901 configured to decompose the micro-expression video into a micro-expression image sequence;

[0104] A preprocessing module 902 configured to preprocess the micro-expression image sequence to obtain the effective face area of each frame of image;

[0105] A local feature extraction module 903 configured to construct a multi-scale spatio-temporal feature extraction module, wherein the multi-scale spatio-temporal feature extraction module is used to: extract the spatial feature of each frame in the image sequence as the local spatial feature by using a macro-expression pre-training model; segment the micro-expression image sequence into multiple time segments with overlap by using a time sliding window; and extract the dynamic change feature of the time segment in the local time range as the local time feature by using a multi-period optical flow method;

[0106] A global feature extraction module 904 configured to construct an improved dual-flow global time sequence learning module for extracting the global feature of the local spatial feature and the local time feature on the time axis to obtain the global spatio-temporal feature of the micro-expression;

[0107] A feature fusion module 905 configured to construct a cross-modal global temporal relationship learning module for fusing the global spatio-temporal feature and modeling the relationship between the time feature and the spatial feature by using a multi-head self-attention mechanism, so as to comprehensively improve the recognition accuracy of the micro-expression video;

[0108] The network training module 906 is configured to combine the multi-scale spatio-temporal feature extraction module, the improved dual-stream global temporal learning module and the cross-modal global temporal relationship learning module to form a local-global temporal relationship network, train the local-global temporal relationship network, and save the best weight model as the trained local-global temporal relationship network.

[0109] The micro-expression recognition module 907 is configured to pre-process a micro-expression video to be recognized, and input the trained local-global temporal relationship network to recognize a micro-expression category.

[0110] In some embodiments, the preprocessing module is further configured to:

[0111] perform face detection and facial feature point detection on each frame of the micro-expression image sequence;

[0112] align the detected facial features by using an affine transformation;

[0113] crop an effective facial region with a consistent size based on the inner corner of the left eye of each frame of the face.

[0114] In some embodiments, the local feature extraction module is further configured to:

[0115] input each frame of image frame by frame, capture facial local details related to expression by using a macro-expression pre-trained DAN model, and capture facial appearance and posture encoding information by using a face pre-trained ArcFace model as local spatial features;

[0116] design a sliding window with a length of t and a step of s, slide along the time axis, and divide the micro-expression image sequence into overlapping time segments with a fixed number of frames,

[0117] calculate the dynamic features of each micro-expression segment by using a multi-period optical flow method, take the first frame of each segment as a reference frame, calculate the horizontal optical flow, vertical optical flow and optical flow strain between each frame of image and the first frame, obtain multi-period optical flow changes from short to long as local temporal features.

[0118] In some embodiments, the total number of segmented time segments is represented as:

[0119]

[0120] wherein T is the total number of frames of the micro-expression video, and τ is the total number of segments.

[0121] In some embodiments, the global feature extraction module is further configured to:

[0122] The stacked multi-phase horizontal optical flow, multi-phase vertical optical flow and multi-phase optical flow strain into a three-channel optical flow map, and the stacked macro-expression features and face appearance features into two-channel spatial features.

[0123] The TCN channels are designed for the optical flow map and the spatial features respectively, and the optical flow map and the spatial features are input into the corresponding TCN channels to capture the global time sequence features thereof.

[0124] In some embodiments, the feature fusion module is further configured to:

[0125] A micro-expression video encoder based on the visual Transformer is constructed, the global time sequence features of each frame of the micro-expression segment are regarded as sequence elements, and the feature sequence is extracted;

[0126] The feature sequence is input into the Transformer encoder of the multi-head self-attention mechanism, the information of different positions in the frame sequence is captured simultaneously through the multi-head self-attention mechanism, the cross-frame dependency relationship and the relationship between the local frame and the global micro-expression video are learned, and the output features of the Transformer encoder are input into the classification network composed of two full connection layers and one batch normalization layer.

[0127] The output features of the Transformer encoder are input into the classification network composed of two full connection layers and one batch normalization layer, and the final classification of the micro-expression category is completed by using the Softmax function.

[0128] It should be noted that the structures of the various micro-expression recognition devices based on the local-global time relationship described in the embodiments belong to the same technical concept as the micro-expression recognition method based on the local-global time relationship described above, and the same beneficial effects are achieved through the same principles, and thus will not be described here.

[0129] The embodiment of the application further provides a readable storage medium, the readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in any one of the above embodiments.

[0130] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. Other embodiments can be used, which will be apparent to those of ordinary skill in the art upon reviewing the above description. Additionally, the various features described above can be grouped together or divided into separate features for simplicity. This should not be interpreted as requiring that the subject matter of any one claim require a particular feature, as aspects of the subject matter can be claimed in all possible combinations. The scope of the subject matter should be determined with reference to the claims and the full scope of equivalents to which such claims are entitled.

Claims

1. A micro-expression recognition method based on local-global temporal relationships, characterized in that, The method includes: Decompose micro-expression videos into micro-expression image sequences; The micro-expression image sequence is preprocessed to obtain the effective face region of each frame; A multi-scale spatiotemporal feature extraction module is constructed, wherein the multi-scale spatiotemporal feature extraction module is used to: extract the spatial features of each frame in the image sequence as local spatial features using a macro-expression pre-trained model; segment the micro-expression image sequence into multiple overlapping time segments through a time sliding window; and extract the dynamic change features of the time segments within the local time range as local time features using a multi-period optical flow method. An improved dual-stream global temporal learning module is constructed to extract global features of local spatial and temporal features on the time axis, thereby obtaining global spatiotemporal features of micro-expressions; A cross-modal global temporal relationship learning module is constructed to fuse global spatiotemporal features, and a multi-head self-attention mechanism is used to model the relationship between temporal and spatial features, thereby comprehensively improving the recognition accuracy of micro-expression videos. A local-global temporal relationship network is formed by combining a multi-scale spatiotemporal feature extraction module, an improved dual-stream global temporal learning module, and a cross-modal global temporal relationship learning module. The local-global temporal relationship network is trained, and the weight model with the best performance is saved as the trained local-global temporal relationship network. The micro-expression video to be identified is preprocessed and then input into a trained local-global temporal relationship network to identify the micro-expression category; By fusing global spatiotemporal features and employing a multi-head self-attention mechanism, the relationship between temporal and spatial features is modeled, including: Construct a micro-expression video encoder based on visual Transformer, treat the global temporal features of each frame of the micro-expression segment as sequence elements, and extract the feature sequence; The feature sequence is input into the Transformer encoder with a multi-head self-attention mechanism. The multi-head self-attention mechanism simultaneously captures information from different positions within the frame sequence, learns cross-frame dependencies and the relationship between local frames and global micro-expression videos. The output features of the Transformer encoder are input into a classification network consisting of two fully connected layers and one batch normalization layer. The Softmax function is then used to complete the final classification of micro-expression categories.

2. The micro-expression recognition method based on local-global temporal relationship according to claim 1, characterized in that, Preprocessing the micro-expression image sequence to obtain the effective face region of each frame includes: Face detection and facial feature key point detection are performed on each frame of the micro-expression image sequence; Affine transformation is used to align the detected facial features; Based on the inner corner of the left eye of each frame, a valid face area of ​​consistent size is cropped out.

3. The micro-expression recognition method based on local-global temporal relationship according to claim 1, characterized in that, The spatial features of each frame in the image sequence are extracted as local spatial features using a macro-expression pre-trained model. The micro-expression image sequence was segmented into multiple overlapping time segments using a time-sliding window method. A multi-phase optical flow method was employed to extract the dynamic change features of each time segment within a local time range as local temporal features, including: Each frame of image is input frame by frame. A macro-expression pre-trained DAN model is used to capture local facial details related to expressions, and a face pre-trained ArcFace model is used to capture facial appearance and pose encoding information as local spatial features. The design length is and stride length A sliding window, moving along the time axis, segments the micro-expression image sequence into overlapping time segments with a fixed number of frames. The dynamic features of each micro-expression segment are calculated using a multi-phase optical flow method. The first frame of each segment is used as the reference frame. The horizontal optical flow, vertical optical flow, and optical flow strain between each frame and the first frame are calculated to obtain the multi-phase optical flow changes from short to long as local temporal features.

4. The micro-expression recognition method based on local-global temporal relationship according to claim 3, characterized in that, The total number of time segments is expressed as: ； Where T represents the total number of frames in the micro-expression video. This represents the total number of segments.

5. The micro-expression recognition method based on local-global temporal relationship according to claim 1, characterized in that, Extracting local spatial and temporal features onto global features along the time axis yields global spatiotemporal features of micro-expressions, including: Stacked multi-phase horizontal optical flow, multi-phase vertical optical flow, and multi-phase optical flow strain are used to form a three-channel optical flow map, and stacked macro-expression features and facial appearance features are used to form a two-channel spatial feature map. TCN channels were designed for optical flow maps and spatial features respectively. The optical flow maps and spatial features were input into the corresponding TCN channels to capture their global temporal features.

6. A micro-expression recognition device based on local-global temporal relationships, characterized in that, The device includes: The video decomposition module is configured to decompose micro-expression videos into micro-expression image sequences; The preprocessing module is configured to preprocess the micro-expression image sequence to obtain the effective face region of each frame image; The local feature extraction module is configured to construct a multi-scale spatiotemporal feature extraction module, wherein the multi-scale spatiotemporal feature extraction module is used to: extract the spatial features of each frame in the image sequence as local spatial features using a macro-expression pre-trained model; segment the micro-expression image sequence into multiple overlapping time segments through a time sliding window; and extract the dynamic change features of the time segments within the local time range as local time features using a multi-period optical flow method. The global feature extraction module is configured to build an improved dual-stream global temporal learning module, which is used to extract global features of local spatial features and local temporal features on the time axis to obtain global spatiotemporal features of micro-expressions; The feature fusion module is configured to build a cross-modal global temporal relationship learning module, which is used to fuse global spatiotemporal features and adopt a multi-head self-attention mechanism to model the relationship between temporal and spatial features, thereby comprehensively improving the recognition accuracy of micro-expression videos. The network training module is configured to combine a multi-scale spatiotemporal feature extraction module, an improved dual-stream global temporal learning module, and a cross-modal global temporal relationship learning module to form a local-global temporal relationship network, train the local-global temporal relationship network, and save the best-performing weight model as the trained local-global temporal relationship network. The micro-expression recognition module is configured to preprocess the micro-expression video to be recognized and input it into a trained local-global temporal relationship network to identify the micro-expression category; The feature fusion module is further configured as follows: Construct a micro-expression video encoder based on visual Transformer, treat the global temporal features of each frame of the micro-expression segment as sequence elements, and extract the feature sequence; The feature sequence is input into the Transformer encoder with a multi-head self-attention mechanism. The multi-head self-attention mechanism simultaneously captures information from different positions within the frame sequence, learns cross-frame dependencies and the relationship between local frames and global micro-expression videos. The output features of the Transformer encoder are input into a classification network consisting of two fully connected layers and one batch normalization layer. The Softmax function is then used to complete the final classification of micro-expression categories.

7. The micro-expression recognition device based on local-global temporal relationship according to claim 6, characterized in that, The preprocessing module is further configured to: Face detection and facial feature key point detection are performed on each frame of the micro-expression image sequence; Affine transformation is used to align the detected facial features; Based on the inner corner of the left eye of each frame, a valid face area of ​​consistent size is cropped out.

8. The micro-expression recognition device based on local-global temporal relationship according to claim 6, characterized in that, The local feature extraction module is further configured as follows: Each frame of image is input frame by frame. A macro-expression pre-trained DAN model is used to capture local facial details related to expressions, and a face pre-trained ArcFace model is used to capture facial appearance and pose encoding information as local spatial features. The design length is and stride length A sliding window, moving along the time axis, segments the micro-expression image sequence into overlapping time segments with a fixed number of frames. The dynamic features of each micro-expression segment are calculated using a multi-phase optical flow method. The first frame of each segment is used as the reference frame. The horizontal optical flow, vertical optical flow, and optical flow strain between each frame and the first frame are calculated to obtain the multi-phase optical flow changes from short to long as local temporal features.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, perform the method according to any one of claims 1 to 5.

Images