A high-precision micro-expression recognition method and device, electronic equipment and storage medium

Abstract

The application discloses a high-precision micro-expression recognition method and device, electronic equipment and storage medium. The method comprises the following steps: S1, acquiring the RGB original image sequence corresponding to the micro-expression video frame and the RGB feature and the optical flow feature corresponding to the optical flow sequence between adjacent frames, and obtaining the adaptive bimodal fusion feature through weighted fusion; S2, simultaneously feeding the adaptive bimodal fusion feature into a multi-stage time convolution network branch and a state space model branch to obtain local time sequence output features and global time sequence features, and performing weighted summation on the local time sequence output features and the global time sequence features based on dynamic gating weights to obtain final gated time sequence fusion features; and S3, after compressing and aggregating the final gated time sequence fusion features, completing the emotion category prediction of the micro-expression through linear transformation and function mapping. The micro-expression recognition is based on motion saliency perception and local-global time sequence cooperation, and the micro-expression recognition precision is greatly improved.

Description

Technical Field

[0001] This application relates to the field of facial recognition technology, and in particular, to a high-precision micro-expression recognition method, device, electronic device, and storage medium. Background Technology

[0002] Microexpressions are spontaneous, extremely short-lived, and subtle facial expressions that can reveal an individual's deliberately concealed true psychological state. They have broad application prospects in highly sensitive fields such as medical auxiliary diagnosis, public safety, psychological assessment, and business negotiations. Due to their short duration, small changes, and high degree of concealment, high-precision microexpression recognition has become an important research direction in the fields of computer vision and affective computing.

[0003] Existing micro-expression analysis techniques are mainly divided into methods based on manually annotated keyframes and methods based on video temporal sequences. The first type of method uses keyframe images as the core, typically selecting the peak frame with the most significant expression changes as the criterion to distinguish micro-expression types. However, to ensure recognition accuracy, this type of method relies on manually annotating video segments to precisely locate the start, peak, and end times of expressions. This process is not only cumbersome and labor-intensive, but the results also highly depend on the annotator's experience and consistency. Furthermore, for extremely subtle muscle movements, dynamic temporal information is far more discriminative than single-frame static information. Relying on only a few keyframes can easily lose details of how expressions evolve over time, thus significantly affecting overall recognition accuracy and robustness.

[0004] The second category of methods targets continuous video sequences, aiming to achieve automated, high-precision micro-expression recognition. These methods typically employ deep learning models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformers, or Graph Convolutions to extract spatiotemporal features from the video and learn temporal variation patterns to achieve end-to-end micro-expression classification. Despite significant improvements in automation, existing technologies still have obvious limitations: firstly, micro-expressions are short-lived and involve subtle movements, making it difficult for traditional networks to accurately characterize the fine-grained dynamic features corresponding to subtle facial muscle movements; secondly, existing models generally suffer from insufficient modeling ability and limited receptive fields when balancing local short-term dynamic details with global long-term temporal dependencies, making it difficult to simultaneously and efficiently capture both the instantaneous changes and global evolutionary patterns of micro-expressions.

[0005] Overall, existing high-precision micro-expression recognition methods still generally suffer from problems such as difficulty in effectively representing weak facial motion signals, susceptibility to noise interference from static textures and identity information, and difficulty in co-modeling local short-term dynamics and global long-term temporal dependencies. Therefore, developing a highly automated, high-precision micro-expression recognition method capable of accurately perceiving weak motion information has become an important technological direction that urgently needs breakthroughs in this field. Summary of the Invention

[0006] This application provides a high-precision micro-expression recognition method to solve the technical problems of existing high-precision micro-expression recognition methods, such as difficulty in accurately perceiving micro-expressions with weak motion information, low automation, low recognition accuracy, and low robustness.

[0007] This application is achieved through the following solution: A high-precision micro-expression recognition method includes the following steps: S1. For the input micro-expression video segment, obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames, and extract the corresponding RGB features and optical flow features. Use fusion weights to perform weighted fusion of the enhanced RGB features and optical flow features to obtain adaptive dual-modal fusion features. S2. The adaptive dual-modal fusion features are simultaneously fed into the multi-stage temporal convolutional network (MS-TCN) branch and the state space model (Mamba) branch to extract short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. S3. After compressing and aggregating the final gated temporal fusion features and high-dimensional video-level semantic representations, the probability distribution of each frame for the final emotion classification is adaptively evaluated through linear transformation and function mapping to complete the emotion category prediction of micro-expressions.

[0008] Further, step S1 specifically includes the following steps: S11. Initial Spatial Feature Extraction of Two Streams: For the input micro-expression video segment, obtain the RGB original image sequence corresponding to the video frame, and use the TV-L1 algorithm to calculate the optical flow sequence between adjacent frames. Input the RGB original image sequence and the corresponding optical flow sequence into their respective residual network branches to extract deep high-dimensional spatial semantic features, denoted as RGB features. and optical flow characteristics ,in, For batch size, The number of feature channels, and These are the height and width of the feature map, respectively. S12. Optical flow-guided spatial attention map generation: A spatial attention map is generated by constructing joint motion response features using optical flow features, based on the initial RGB features. Residual enhancement is performed to obtain the enhanced RGB features. ; S13, Adaptive dual-modal gating fusion, enhances RGB features With optical flow characteristics The adaptive fusion weights generated by concatenating features along the feature dimension are used to weight and fuse the enhanced RGB features and optical flow features to obtain the adaptive dual-modal fusion features. .

[0009] Furthermore, in step S11, before obtaining the RGB original image sequence corresponding to the video frame and calculating the optical flow sequence between adjacent frames using the TV-L1 algorithm, preprocessing operations such as face key point detection and alignment cropping are first performed to unify the spatial resolution.

[0010] Further, step S12 specifically includes the following steps: S121. Extract the horizontal optical flow component from the original optical flow field calculated based on the preprocessing stage. With vertical optical flow component Calculate their absolute values ​​to obtain the horizontal motion intensity diagram. and vertical motion intensity diagram Simultaneously, calculate the optical flow amplitude map: ; in, Used to characterize the overall motion intensity at each spatial location; S122, will , and The joint motion response features are constructed by concatenating them along the channel dimension, and a spatial attention map is generated by using a two-dimensional convolutional layer and a sigmoid activation function. : ; in, This indicates a channel splicing operation. This represents the Sigmoid activation function; S123, Based on the generated spatial attention map , initial features of RGB Perform residual enhancement to obtain the enhanced features. : ; in, This represents element-wise multiplication. This is the learnable scaling factor.

[0011] Furthermore, step S13 specifically includes the following steps: S131, Enhanced RGB features With optical flow characteristics By concatenating the features along the feature dimension, we obtain the joint features: ; Generate adaptive fusion weights based on joint features: ; in, Indicates global average pooling. Indicates a fully connected mapping; S132. Finally, the enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain adaptive dual-modal fusion features. : ; in, This indicates element-wise multiplication.

[0012] Furthermore, step S2 specifically includes the following steps: S21. The features output by the adaptive dual-modal fusion module... Simultaneously, the input features are fed into a multi-stage temporal convolutional network (MS-TCN) branch and a state-space model (Mamba) branch for parallel modeling. The multi-stage temporal convolutional network (MS-TCN) branch will input features... Short-term dynamic information is extracted using several dilated convolutional blocks containing residual connections, thereby obtaining local temporal output features: ; MS-TCN consists of three stages, each of which deploys several dilated residual blocks, and the dilation rates of the dilated convolutions in each stage are set to 1, 2 and 4, respectively. The state-space model (Mamba) branch will Treating it as a state-space sequence, we perform deep mining of global temporal dependencies by stacking three MambaBlocks. We introduce a selective scan mechanism to accurately capture the global evolution trajectory of micro-expressions throughout the video segment, thus obtaining global temporal features: ; S22. Directly concatenate the local temporal output features and the global temporal features along the feature dimension to obtain the joint temporal representation. Subsequently, information is exchanged between the joint temporal representations through a linear mapping layer, and a gating weight matrix for dynamic weighting is generated using an activation function. ; in, This represents the gated weight tensor generated by the Sigmoid function. This reflects the importance of different characteristic channels at each time step; S23. Using the gate weight matrix for dynamic weighting, the output features of the multi-stage temporal convolutional network and the state-space model Mamba are summed element-wise to calculate the final gated temporal fusion features: ; in, This indicates element-wise multiplication.

[0013] Furthermore, step S3 specifically includes the following steps: S31. Calculate the i-th element in the sequence through a linear mapping layer containing learnable parameters. Frame-gated temporal fusion features The initial importance score is then used, followed by the Softmax activation function throughout the entire time dimension. The initial importance score is then probability-normalized to generate attention weights for each frame: ; in, This is the learned weight matrix of the attention network, after Softmax processing, the attention weights. satisfy ; S32. After obtaining the temporal attention weights, a weighted summation operation is performed on the entire gated temporal fusion feature to compress and aggregate the two-dimensional gated temporal fusion feature into a one-dimensional high-dimensional video-level semantic representation. : ; S33. High-dimensional video-level semantic representation The input is fed into a classification head consisting of fully connected layers. Through a linear transformation and mapping to the final Softmax function, the output is the probability distribution of micro-expression samples belonging to various predefined emotion categories (such as positive, negative, surprised, etc.). ; in, and These represent the weight matrix and bias vector of the fully connected classification layer, respectively. This represents the final predicted class probability.

[0014] This application also provides a high-precision micro-expression recognition device, comprising: The adaptive bimodal feature fusion module is used to obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames from the input micro-expression video segment, and extract the corresponding RGB features and optical flow features. The enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain the adaptive bimodal fusion features. The gated temporal fusion computation module is used to simultaneously feed the adaptive dual-modal fusion features into the multi-stage temporal convolutional network (MS-TCN) branch and the state space model (Mamba) branch to extract short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. The micro-expression prediction module is used to compress and aggregate the final gated temporal fusion features and high-dimensional video-level semantic representations. Then, through linear transformation and function mapping, it adaptively evaluates the probability distribution of each frame for the final emotion classification, thus completing the emotion category prediction of micro-expressions.

[0015] This application also provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the high-precision micro-expression recognition method.

[0016] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the high-precision micro-expression recognition method.

[0017] This application also provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the high-precision micro-expression recognition method.

[0018] Compared with the prior art, this application can produce the following beneficial effects: 1. This application significantly enhances the fine-grained representation capability of subtle facial deformations: relying on the designed saliency-aware motion core fusion mechanism, the underlying dynamic features are used as spatiotemporal priors to modulate the global appearance features. While fully preserving the facial spatial topology, this mechanism can adaptively focus on and amplify transient subtle deformation information in key local areas such as the periorbital area and the glabella. In terms of training configuration, the model is optimized using an end-to-end approach. 2. This application effectively suppresses redundant noise during multimodal feature fusion: It proposes an adaptive bimodal fusion gating strategy that can dynamically balance the contributions of apparent texture and motion vectors according to different spatiotemporal states, thereby significantly reducing the negative impact of illumination changes and identity differences; 3. This application can achieve efficient temporal coordination between local details and global evolution: a collaborative modeling mechanism of local multi-stage temporal convolutional network and global state space model is constructed. While maintaining linear computational complexity, it not only uses multi-scale convolutional network to capture short-term transient changes in micro-expressions, but also uses state space model to grasp the global long-term evolution law of micro-expressions. 4. This application offers flexible parameter configuration and is practical for engineering implementation: Key parameters such as time window, step size, threshold, and loss weight can be flexibly set according to the characteristics of the dataset, achieving a controllable trade-off between accuracy and efficiency, which facilitates actual deployment and subsequent expansion.

[0019] In addition to the purposes, features, and advantages described above, this application has other purposes, features, and advantages. A further detailed description of this application will be provided below with reference to the figures. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating a preferred embodiment of the high-precision micro-expression recognition method of this application; Figure 2 This is a schematic diagram of a high-precision micro-expression recognition network structure according to a preferred embodiment of this application; Figure 3 This is a schematic diagram of a module of a high-precision micro-expression recognition device according to a preferred embodiment of this application; Figure 4 This is a schematic block diagram of an electronic device according to a preferred embodiment of this application; Figure 5 This is a schematic diagram of the internal structure of a computer device according to a preferred embodiment of this application. Detailed Implementation

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the execution subject of this embodiment can be a computing service system with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc., or a high-precision micro-expression recognition device that can realize the above functions. The following uses a high-precision micro-expression recognition device as the execution subject to describe this embodiment and the following embodiments.

[0024] like Figure 1 As shown, to address the aforementioned technical problems, a preferred embodiment of this application provides a high-precision micro-expression recognition method, comprising the following steps: S1. For the input micro-expression video segment, obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames, and extract the corresponding RGB features and optical flow features. Use fusion weights to perform weighted fusion of the enhanced RGB features and optical flow features to obtain adaptive dual-modal fusion features. S2. The adaptive dual-modal fusion features are simultaneously fed into the multi-stage temporal convolutional network (MS-TCN) branch and the state space model (Mamba) branch to extract short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. S3. After compressing and aggregating the final gated temporal fusion features and high-dimensional video-level semantic representations, the probability distribution of each frame for the final emotion classification is adaptively evaluated through linear transformation and function mapping to complete the emotion category prediction of micro-expressions.

[0025] This embodiment provides a micro-expression recognition method based on motion saliency perception and local-global temporal coordination, aiming to solve problems such as the difficulty in effectively representing weak facial motion signals, susceptibility to noise interference from static textures and identity information, and the difficulty in co-modeling local short-term dynamics and global long-term temporal dependencies.

[0026] Preferably, step S1 specifically includes the following steps: S11. Dual-stream initial spatial feature extraction: For the input micro-expression video clip, preprocessing operations such as facial keypoint detection and alignment cropping are performed to unify the spatial resolution. The RGB original image sequences corresponding to the video frames are obtained, and the optical flow sequence between adjacent frames is calculated using the TV-L1 optical flow algorithm. The RGB original image sequences and corresponding optical flow sequences are input into their respective residual network branches. This embodiment uses a dual-stream architecture for initial spatial feature encoding, selecting ResNet-18 as the backbone feature extractor of the shared architecture. The RGB images and corresponding optical flow fields are input into their respective ResNet-18 branches to extract deep, high-dimensional spatial semantic features, denoted as RGB features. and optical flow characteristics ,in, For batch size, The number of feature channels, and These are the height and width of the feature map, respectively. S12. Optical flow-guided spatial attention map generation: A spatial attention map is generated by constructing joint motion response features using optical flow features, based on the initial RGB features. Residual enhancement is performed to obtain the enhanced RGB features. This step designs a spatial attention mechanism based on optical flow components and motion amplitude to extract attention maps with high spatial discriminativeness from optical flow modes. The design of this mechanism is based on the fact that facial muscle movements corresponding to micro-expressions usually manifest as subtle lateral deformation, longitudinal stretching, and their combinations, which respectively characterize the horizontal motion component, vertical motion component, and overall motion intensity, thus helping to enhance the network's response to regions with weak motion.

[0027] Specifically, step S12 includes the following steps: S121. Extract the horizontal optical flow component from the original optical flow field calculated based on the preprocessing stage. With vertical optical flow component Calculate their absolute values ​​to obtain the horizontal motion intensity diagram. and vertical motion intensity diagram Simultaneously, calculate the optical flow amplitude map: ; in, Used to characterize the overall motion intensity at each spatial location; S122, will , and The joint motion response features are constructed by concatenating them along the channel dimension, and a spatial attention map is generated by using a two-dimensional convolutional layer and a sigmoid activation function. : ; in, This indicates a channel splicing operation. This represents the Sigmoid activation function; S123, Based on the generated spatial attention map , initial features of RGB Perform residual enhancement to obtain the enhanced features. : ; in, This represents element-wise multiplication. As a learnable scaling factor, the optical flow mode participates in RGB feature enhancement as a motion prior through the above method. While preserving the original facial appearance information, it improves the representation ability of key motion areas and amplifies the transient subtle deformation information of key local areas such as the eye area and the glabella.

[0028] S13, Adaptive dual-modal gating fusion, enhances RGB features With optical flow characteristics The adaptive fusion weights generated by concatenating features along the feature dimension are used to weight and fuse the enhanced RGB features and optical flow features to obtain the adaptive dual-modal fusion features. The specific steps include: S131, Enhanced RGB features With optical flow characteristics By concatenating the features along the feature dimension, we obtain the joint features: ; Generate adaptive fusion weights based on joint features: ; in, Indicates global average pooling. Indicates a fully connected mapping; S132. Finally, the enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain adaptive dual-modal fusion features. : ; in, This represents element-wise multiplication. This fusion method can adaptively adjust the contribution ratio of appearance information and motion information according to the feature distribution of different samples, thereby reducing the impact of redundant noise on the recognition results.

[0029] Preferably, step S2 constructs a local-global temporal collaborative modeling mechanism, aiming to fully explore the complementary information between fine-grained short-term dynamics and long-term temporal dependencies in micro-expression videos. Step S2 specifically includes the following steps: S21. The Multi-Stage Temporal Convolutional Network (MS-TCN) branch is primarily dedicated to capturing multi-scale local temporal patterns in micro-expression movements (e.g., instantaneous twitching of facial muscles in the initial stage). Therefore, the features output by the adaptive bimodal fusion module are used... Simultaneously, the input features are fed into a multi-stage temporal convolutional network (MS-TCN) branch and a state-space model (Mamba) branch for parallel modeling. The multi-stage temporal convolutional network (MS-TCN) branch will input features... Short-term dynamic information is extracted using several dilated convolutional blocks containing residual connections, thereby obtaining local temporal output features: ; In terms of network structure configuration, in order to balance the feature representation performance and computational complexity of the model, this branch adopts a lightweight design concept. MS-TCN consists of three stages, each of which deploys several dilated residual blocks. The dilation rates of the dilated convolutions in each stage are set to 1, 2 and 4, respectively. This step-like dilation rate setting can effectively expand the network's temporal receptive field exponentially without adding additional network parameters, thereby accurately adapting to the sequence characteristics of short evolution cycles and rapid dynamic changes in micro-expressions. Meanwhile, the Mamba branch of the state-space model focuses on addressing the shortcomings of convolutional neural networks in long-range temporal modeling capabilities. Treating it as a state-space sequence, we perform deep mining of global temporal dependencies by stacking three Mamba Blocks. We introduce a selective scan mechanism to accurately capture the global evolution trajectory of micro-expressions throughout the video segment, thus obtaining global temporal features: ; S22, Fine-grained local features were acquired in parallel. With long-range global features Subsequently, to fully leverage the complementary advantages of these two temporal modeling methods, this step further designs a gated temporal fusion mechanism to achieve adaptive deep integration of local and global information. This mechanism directly concatenates local temporal output features and global temporal features along the feature dimension to obtain a joint temporal representation. Subsequently, information is exchanged between the joint temporal representations through a linear mapping layer, and a gating weight matrix for dynamic weighting is generated using an activation function. ; in, This represents the gated weight tensor generated by the Sigmoid function. This reflects the importance of different characteristic channels at each time step; S23. Using the gate weight matrix for dynamic weighting, the output features of the multi-stage temporal convolutional network and the state-space model Mamba are summed element-wise to calculate the final gated temporal fusion features: ; in, This represents element-wise multiplication. Through this collaborative optimization process, a more robust and highly discriminative temporal feature representation is ultimately formed, laying a solid foundation for subsequent classification tasks.

[0030] Preferably, step S3 specifically includes the following steps: S31. Calculate the i-th element in the sequence through a linear mapping layer containing learnable parameters. Frame-gated temporal fusion features The initial importance score is then used, followed by the Softmax activation function throughout the entire time dimension. The initial importance score is then probability-normalized to generate attention weights for each frame: ; in, This is the learned weight matrix of the attention network, after Softmax processing, the attention weights. satisfy ; S32. After obtaining the precise temporal attention weights, a weighted summation operation is performed on the entire gated temporal fusion feature to compress and aggregate the two-dimensional gated temporal fusion feature into a one-dimensional high-dimensional video-level semantic representation. : ; S33. High-dimensional video-level semantic representation The input is fed into a classification head consisting of fully connected layers. Through a linear transformation and mapping to the final Softmax function, the output is the probability distribution of micro-expression samples belonging to various predefined emotion categories (such as positive, negative, surprised, etc.). ; in, and These represent the weight matrix and bias vector of the fully connected classification layer, respectively. This represents the final predicted class probability.

[0031] This embodiment serves as the final decision-making stage of the entire micro-expression recognition network. Its core task is to map the high-dimensional temporal sequence features extracted by the preceding network into a compact video-level global representation, and based on this, to predict the sentiment category of micro-expressions. In actual observation of micro-expression videos, a complete sequence often contains a large number of "expressionless" resting state frames, while the truly subtle facial muscle movements usually only exist in a very small number of keyframe intervals (e.g., from the Onset phase at the beginning of the action to the Apex phase at the peak of the action). Therefore, this invention designs and employs a data-driven temporal attention weighted pooling mechanism, which can adaptively evaluate the contribution of each frame to the final sentiment classification.

[0032] In another preferred embodiment, such as Figure 2 As shown, we implemented the micro-expression recognition model of this application on an NVIDIA RTX 3090 GPU server using the PyTorch framework for high-precision micro-expression recognition. The implementation process and main parameters are described below: First, the micro-expression video sequence was processed. The original video was organized according to the subjects and video segments, and a unified index was established after frame-by-frame decoding. In the video frame-level preprocessing stage, face detection, key point localization, and face alignment were performed on each frame. Specifically, RetinaFace was used to detect the face region in each frame and obtain the face bounding box; then Dlib was used to detect facial key points, extract the coordinates of key points such as the corners of the eyes, the tip of the nose, and the corners of the mouth, and perform face alignment. The aligned face regions were then cropped and scaled to a uniform spatial size to obtain a standardized face image sequence.

[0033] After obtaining the standardized face sequence, a dual-modal input of appearance flow and motion flow is further constructed. The appearance flow consists directly of the aligned RGB image sequence of the face, used to preserve facial texture, structure, and grayscale distribution. The motion flow is obtained by calculating the dense optical flow between adjacent frames, specifically using the TV-L1 optical flow algorithm to estimate the pixel displacement components in the horizontal and vertical directions. To reduce the impact of background changes on motion estimation, the optical flow calculation is performed within the cropped and aligned face region, maintaining a strict correspondence with the RGB input in the temporal dimension. For each pair of adjacent frames, the corresponding two-dimensional optical flow field is obtained, and then the optical flow fields of adjacent frames of the entire video segment are stacked in temporal order to form a temporal sequence of optical flow.

[0034] To facilitate model input construction, the processed video frame sequences are further converted into fixed-length training samples. For original video sequences of inconsistent lengths, a sliding window approach is used for sample segmentation. The window slides along the time axis, with each window corresponding to a local video subsequence, and simultaneously extracting the corresponding RGB frame sequences and optical flow sequences. In the CASME3 data scenario, the sliding window length is set to 50 frames, and the sliding step size is set to 25 frames; in the SAMM data scenario, the window length is set to 30 frames, and the step size is set to 15 frames. This method converts original videos of inconsistent lengths into temporal samples of uniform length, facilitating batch training. For video segments shorter than the window length, length padding can be achieved using methods such as end-frame padding, repeating boundary frames, or zero-padding; for videos exceeding the window length, multiple overlapping samples are generated through sliding segmentation to increase the number of training samples while preserving temporal continuity.

[0035] In terms of model structure, the input original appearance sequence and underlying dynamics sequence first enter dual-stream feature extraction branches. The appearance branch is used to extract global spatial topology and texture details of the face, while the dynamics branch is used to decouple the pure motion features of facial muscle groups. Subsequently, in the saliency-aware motion core fusion module, the extracted dynamics prior features are used to generate a spatially saliency attention map, which is then used to perform residual modulation enhancement on the appearance features, significantly improving the network's ability to represent subtle facial deformation regions with fine-grained detail. The modulated and enhanced appearance features are fused with the original dynamics features through an adaptive bimodal gating strategy to obtain a high-confidence unified spatiotemporal collaborative feature. The residual enhancement coefficients in this module are set as learnable parameters so that the model can adaptively adjust the saliency-awareness amplitude according to the distribution of the training data.

[0036] In the temporal modeling stage, the fused spatiotemporal collaborative feature sequence is input into the local-global temporal collaborative modeling module. The local multi-stage temporal convolutional network branch focuses on capturing multi-scale local short-term transient abrupt changes in micro-expression movements. It is configured with three stages (blocks), each containing expanded residual blocks with expansion rates of 1, 2, and 4, respectively, exponentially expanding the temporal receptive field in a stepwise manner while accurately preserving local dynamic details. The global state space model branch focuses on establishing global long-term temporal dependencies, configured with three Mamba blocks, using a selective scanning mechanism to accurately model the global long-term temporal evolution trajectory of micro-expressions from onset to apex. Finally, the two complementary temporal features are deeply gated and integrated through adaptive temporal fusion and a decision mechanism, outputting a unified, highly discriminative temporal representation. The core temporal feature dimension in the entire network is uniformly set to C=256. In terms of training configuration, the model is optimized end-to-end. The cross-entropy loss function is used to constrain the difference between the predicted class and the true label. The optimizer used is AdamW, with an initial learning rate of 0.0001. During training, parameters such as batch size, weight decay coefficient, number of training epochs, and early stopping strategy can be set to improve the stability and generalization ability of the model training.

[0037] For model evaluation, a Leave-One-Subject-Out (LOSO) cross-validation strategy was employed. In this strategy, all samples from one subject were selected as the test set, while the remaining samples were used as the training set to verify the model's generalization ability to unknown individuals. This strategy effectively avoids the same subject's data appearing in both the training and test sets, thus more realistically reflecting the model's cross-individual recognition performance in real-world scenarios. For each round of LOSO validation, classification accuracy, F1 score, recall, and precision were recorded, and the average results across all subjects were used as the final performance evaluation.

[0038] As can be seen, this application addresses the problems of weak micro-expression signals, susceptibility to noise interference, and the complex temporal dependence of improving the representation ability of key motion regions. It proposes a micro-expression recognition method based on motion saliency perception and local-global temporal collaboration. Its key technical points include: 1. A saliency-aware motion core fusion mechanism. A spatial attention map is constructed using optical flow components and motion amplitude, and residual enhancement is performed on appearance texture features, enabling the model to accurately focus on areas of weak facial motion and effectively suppress interference from static textures and identity information.

[0039] 2. Dual-modal gating fusion mechanism. A dual-modal collaborative modeling framework for appearance texture and motion optical flow is constructed. Adaptive fusion weights are generated through global context to achieve dynamic balance and optimal integration of appearance and motion information.

[0040] 3. Local-Global Temporal Co-modeling Mechanism. A multi-stage temporal convolutional network is used to capture fine-grained local dynamics, while a state-space model is introduced to model long-range temporal dependencies, thereby achieving a collaborative expression of local and global temporal information.

[0041] 4. Adaptive Temporal Fusion Mechanism. A gated temporal fusion module is designed to dynamically weight and fuse local multi-stage temporal convolutional networks with the global state-space model, significantly enhancing the discriminative power and robustness of temporal features.

[0042] 5. Decision-making mechanism. A data-driven temporal attention-weighted pooling method is used to highlight key frames from Onset to Apex, generating highly discriminative video-level features and completing sentiment classification.

[0043] like Figure 3 As shown, another preferred embodiment of this application also provides a high-precision micro-expression recognition device, including: The adaptive bimodal feature fusion module is used to obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames from the input micro-expression video segment, and extract the corresponding RGB features and optical flow features. The enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain the adaptive bimodal fusion features. The gated temporal fusion computation module is used to simultaneously feed the adaptive dual-modal fusion features into the multi-stage temporal convolutional network (MS-TCN) branch and the state space model (Mamba) branch to extract short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. The micro-expression prediction module is used to compress and aggregate the final gated temporal fusion features and high-dimensional video-level semantic representations. Then, through linear transformation and function mapping, it adaptively evaluates the probability distribution of each frame for the final emotion classification, thus completing the emotion category prediction of micro-expressions.

[0044] The high-precision micro-expression recognition device provided in this embodiment employs the high-precision micro-expression recognition method described in the above embodiments, solving the technical problems of existing high-precision micro-expression recognition methods, such as difficulty in accurately perceiving micro-expressions with weak motion information, low automation, low recognition accuracy, and low robustness. Compared with the prior art, the beneficial effects of the high-precision micro-expression recognition device provided in this embodiment are the same as those of the high-precision micro-expression recognition method described in the above embodiments, and other technical features in the high-precision micro-expression recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0045] like Figure 4 As shown, a preferred embodiment of this example also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the high-precision micro-expression recognition method described in the above embodiment.

[0046] This embodiment also provides an electronic device that uses the high-precision micro-expression recognition method described in the above embodiments to solve the technical problems of existing high-precision micro-expression recognition methods, such as difficulty in accurately perceiving micro-expressions with weak motion information, low automation, low recognition accuracy, and low robustness. Compared with the prior art, the beneficial effects of the electronic device provided in this embodiment are the same as those of the high-precision micro-expression recognition method described in the above embodiments, and other technical features in the electronic device are the same as those disclosed in the method described in the above embodiments, which will not be repeated here.

[0047] like Figure 5 As shown, a preferred embodiment of this application also provides a computer device, which may be a terminal or a liveness detection server, and its internal structure diagram may be as follows. Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with other external computer devices via a network connection. When the computer program is executed by the processor, it implements the steps of the aforementioned high-precision micro-expression recognition method.

[0048] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the solution of this embodiment, and does not constitute a limitation on the computer device to which the solution of this embodiment is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0049] The computer device provided in this embodiment adopts the high-precision micro-expression recognition method in the above embodiment to solve the technical problems of existing high-precision micro-expression recognition being unable to accurately perceive micro-expressions with weak motion information, and having low automation, recognition accuracy and robustness. Compared with the prior art, the beneficial effects of the computer device provided in this embodiment are the same as the beneficial effects of the high-precision micro-expression recognition method provided in the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the method of the above embodiment, which will not be repeated here.

[0050] A preferred embodiment of this application also provides a storage medium including a stored program that, when the program is executed, controls the device where the storage medium is located to perform the steps of the high-precision micro-expression recognition method described above.

[0051] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0052] If the functions described in this embodiment are implemented as software functional units and sold or used as independent products, they can be stored in one or more computing device-readable storage media. Based on this understanding, the parts of this embodiment that contribute to the prior art or the technical solution can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, mobile computing device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this embodiment. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0053] Those skilled in the art will understand that the embodiments of this example can be provided as methods, systems, or computer program products. Therefore, this example can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this example can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code. The solutions in this example can be implemented using various computer languages, such as the object-oriented programming language C++ and the embedded programming language C.

[0054] This embodiment is described with reference to flowchart illustrations and / or block diagrams of the method, apparatus (system), and computer program product according to this embodiment. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0057] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the high-precision micro-expression recognition method described above.

[0058] The computer program product provided in this embodiment solves the technical problems of existing high-precision micro-expression recognition methods, such as difficulty in accurately perceiving micro-expressions with weak motion information, low automation, low recognition accuracy, and low robustness. Compared with the prior art, the beneficial effects of the computer program product provided in this embodiment are the same as those of the high-precision micro-expression recognition method provided in the above embodiments, and will not be repeated here.

[0059] Obviously, those skilled in the art can make various modifications and variations to this embodiment without departing from the spirit and scope of this embodiment. Therefore, if these modifications and variations of this embodiment fall within the scope of the claims of this embodiment and their equivalents, this embodiment is also intended to include these modifications and variations.

Claims

1. A high-precision micro-expression recognition method, characterized in that, Including the following steps: S1. For the input micro-expression video segment, obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames, and extract the corresponding RGB features and optical flow features. Use fusion weights to perform weighted fusion of the enhanced RGB features and optical flow features to obtain adaptive dual-modal fusion features. S2. The adaptive dual-modal fusion features are simultaneously fed into the multi-stage temporal convolutional network branch and the state space model branch to extract short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. S3. After compressing and aggregating the final gated temporal fusion features and high-dimensional video-level semantic representations, the probability distribution of each frame for the final emotion classification is adaptively evaluated through linear transformation and function mapping to complete the emotion category prediction of micro-expressions.

2. The high-precision micro-expression recognition method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Initial Spatial Feature Extraction of Dual Streams: For the input micro-expression video segment, obtain the RGB original image sequence corresponding to the video frame, and use the optical flow algorithm to calculate the optical flow sequence between adjacent frames. Input the RGB original image sequence and the corresponding optical flow sequence into their respective residual network branches to extract deep high-dimensional spatial semantic features, denoted as RGB features. and optical flow characteristics ,in, For batch size, The number of feature channels, and These are the height and width of the feature map, respectively. S12. Optical flow-guided spatial attention map generation: A spatial attention map is generated by constructing joint motion response features using optical flow features, based on the initial RGB features. Residual enhancement is performed to obtain the enhanced RGB features. ; S13, Adaptive dual-modal gating fusion, enhances RGB features With optical flow characteristics The adaptive fusion weights generated by concatenating features along the feature dimension are used to weight and fuse the enhanced RGB features and optical flow features to obtain the adaptive dual-modal fusion features. .

3. The high-precision micro-expression recognition method according to claim 2, characterized in that, In step S11, before obtaining the RGB original image sequence corresponding to the video frame and calculating the optical flow sequence between adjacent frames using the TV-L1 algorithm, preprocessing operations such as face key point detection and alignment cropping are performed to unify the spatial resolution.

4. The high-precision micro-expression recognition method according to claim 3, characterized in that, Step S12 specifically includes the following steps: S121. Extract the horizontal optical flow component from the original optical flow field calculated based on the preprocessing stage. With vertical optical flow component Calculate their absolute values ​​to obtain the horizontal motion intensity diagram. and vertical motion intensity diagram Simultaneously, calculate the optical flow amplitude map: ； in, Used to characterize the overall motion intensity at each spatial location; S122, will , and The joint motion response features are constructed by concatenating them along the channel dimension, and a spatial attention map is generated by using a two-dimensional convolutional layer and a sigmoid activation function. : ； in, This indicates a channel splicing operation. This represents the Sigmoid activation function; S123, Based on the generated spatial attention map , for RGB initial features Perform residual enhancement to obtain the enhanced features. : ； in, This represents element-wise multiplication. This is the learnable scaling factor.

5. The high-precision micro-expression recognition method according to claim 4, characterized in that, Step S13 specifically includes the following steps: S131, Enhanced RGB features With optical flow characteristics By concatenating the features along the feature dimension, we obtain the joint features: ； Generate adaptive fusion weights based on joint features: ； in, Indicates global average pooling. Indicates a fully connected mapping; S132. Finally, the enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain adaptive dual-modal fusion features. : ； in, This indicates element-wise multiplication.

6. The high-precision micro-expression recognition method according to claim 5, characterized in that, Step S2 specifically includes the following steps: S21. The features output by the adaptive dual-modal fusion module... Simultaneously, the input features are fed into a multi-stage temporal convolutional network branch and a state-space model branch for parallel modeling. The multi-stage temporal convolutional network branch will input features... Short-term dynamic information is extracted by using several dilated convolutional blocks containing residual connections, thereby obtaining local temporal output features: ； State-space model branching Treating it as a state-space sequence, we perform deep mining of global temporal dependencies by stacking three Mamba Blocks. We introduce a selective scanning mechanism to accurately capture the global evolution trajectory of micro-expressions throughout the video segment, thus obtaining global temporal features: ； S22. Directly concatenate the local temporal output features and the global temporal features along the feature dimension to obtain the joint temporal representation. Subsequently, information is exchanged between the joint temporal representations through a linear mapping layer, and a gating weight matrix for dynamic weighting is generated using an activation function. ； in, This represents the gated weight tensor generated by the Sigmoid function. This reflects the importance of different characteristic channels at each time step; S23. Using the gate weight matrix for dynamic weighting, the output features of the multi-stage temporal convolutional network and the state-space model Mamba are summed element-wise to calculate the final gated temporal fusion features: ； in, This indicates element-wise multiplication.

7. The high-precision micro-expression recognition method according to claim 6, characterized in that, Step S3 specifically includes the following steps: S31. Calculate the i-th element in the sequence through a linear mapping layer containing learnable parameters. Frame-gated temporal fusion features The initial importance score is then used, followed by the Softmax activation function throughout the entire time dimension. The initial importance score is then probability-normalized to generate attention weights for each frame: ； in, This is the learned weight matrix of the attention network, after Softmax processing, the attention weights. satisfy ; S32. After obtaining the temporal attention weights, a weighted summation operation is performed on the entire gated temporal fusion feature to compress and aggregate the two-dimensional gated temporal fusion feature into a one-dimensional high-dimensional video-level semantic representation. : ； S33. High-dimensional video-level semantic representation The input is fed into a classification head consisting of fully connected layers. Through a linear transformation and mapping to the final Softmax function, the output is the probability distribution of micro-expression samples belonging to various predefined emotion categories: ； in, and These represent the weight matrix and bias vector of the fully connected classification layer, respectively. This represents the final predicted class probability.

8. A high-precision micro-expression recognition device, characterized in that, include: The adaptive bimodal feature fusion module is used to obtain the original RGB image sequence corresponding to the video frame and the optical flow sequence between adjacent frames from the input micro-expression video segment, and extract the corresponding RGB features and optical flow features. The enhanced RGB features and optical flow features are weighted and fused using fusion weights to obtain the adaptive bimodal fusion features. The gated temporal fusion computation module is used to simultaneously feed the adaptive dual-modal fusion features into the multi-stage temporal convolutional network branch and the state space model branch, respectively extracting short-time dynamic information to obtain local temporal output features and global temporal features that represent the global evolution trajectory of micro-expressions in the entire video segment. Based on dynamic gating weights, the local temporal output features and global temporal features are summed element-wise to obtain the final gated temporal fusion features. The micro-expression prediction module is used to compress and aggregate the final gated temporal fusion features and high-dimensional video-level semantic representations. Then, through linear transformation and function mapping, it adaptively evaluates the probability distribution of each frame for the final emotion classification, thus completing the emotion category prediction of micro-expressions.

9. An electronic device, the electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the high-precision micro-expression recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the high-precision micro-expression recognition method as described in any one of claims 1 to 7.

Images