Micro-expression recognition method and system based on optical flow enhancement and spatio-temporal relationship modeling

Abstract

The application discloses a micro-expression recognition method and system based on optical flow enhancement and space-time relationship modeling, and belongs to the technical field of micro-expression recognition. In view of the problem that the prior art cannot actively amplify the cross-stage subtle motion difference and accurately position the key muscle area under strong interference environment, the application acquires a micro-expression video sequence and extracts a starting frame, a peak frame and an ending frame; extracts optical flow features between two groups of key frame pairs; performs cross-stage residual enhancement through a direction residual optical flow enhancement module, and accurately positions the key muscle area through a two-stage direction position coding module; and then, the space-time dependence is jointly modeled through a space-time relationship modeling module, and high-level discriminative features are output for classification. The application realizes high-accuracy and high-robustness recognition of micro-expression under complex environment, and can be widely applied to scenes such as safe interrogation, psychological diagnosis and human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of micro-expression recognition technology, and in particular to a micro-expression recognition method and system based on optical flow enhancement and spatiotemporal relationship modeling. Background Technology

[0002] The statements in this section are merely to provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In scenarios such as security interrogation, psychological diagnosis, and driver emotion monitoring, micro-expressions are considered key non-verbal signals revealing an individual's true emotional state. However, existing micro-expression recognition methods generally rely on optical flow between adjacent frames or only focus on the static features of peak frames, making it difficult to cope with the problems of extremely subtle motion amplitudes, numerous interference factors, and complex stages of evolution in practical applications. On the one hand, irrelevant actions such as head movements, sudden changes in lighting, and speaking can easily obscure weak muscle movement information; on the other hand, the dynamic differences and intrinsic connections between the contraction (from the beginning to the peak) and relaxation (from the peak to the end) of facial expressions are often ignored, leading to inaccurate localization of key muscle areas.

[0004] Therefore, existing technologies cannot actively amplify the subtle movement differences of micro-expressions across stages in environments with strong interference, nor can they combine spatiotemporal modeling to achieve precise focusing on key areas. This will lead to weak real emotional signals being submerged by strong interference such as head movements and changes in lighting in practical applications such as lie detection and psychological diagnosis. The system is prone to missed judgments and misjudgments, making it difficult to improve the accuracy of micro-expression recognition in complex scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a micro-expression recognition method and system based on optical flow enhancement and spatiotemporal relationship modeling. By constructing optical flow features between key frame pairs and combining directional residual optical flow enhancement, a two-stage directional position encoding module, and spatiotemporal relationship modeling, the method achieves automatic recognition of micro-expression categories, thereby improving the accuracy of micro-expression recognition in complex scenarios.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the present invention provides a micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling, comprising: The micro-expression video sequence is acquired and preprocessed to obtain the start frame, peak frame, and end frame. Extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; The optical flow features are processed by the directional residual optical flow enhancement module to obtain enhanced cross-stage motion difference features; the cross-stage motion difference features are processed by the dual-stage directional position encoding module to extract key region features; the key region features are jointly modeled by the spatiotemporal relationship modeling module to obtain high-level discrimination features; The high-level discriminative features are processed to obtain micro-expression recognition results.

[0007] Furthermore, the TV-L1 optical flow algorithm is used to calculate the optical flow maps between the start frame and the peak frame, and between the peak frame and the end frame. Each optical flow map is composed of three channels: horizontal displacement component, vertical displacement component, and optical flow strain amplitude.

[0008] Furthermore, the directional component is extracted from the optical flow during the relaxation phase between the peak frame and the end frame, and the directional residual is calculated. Then, the directional residual is superimposed on the optical flow during the contraction phase between the start frame and the peak frame to obtain the enhanced cross-phase motion difference feature.

[0009] Furthermore, the enhanced cross-stage motion difference features are input into the dual-stage orientation and position encoding module. By introducing a cross-attention mechanism of local neighborhood mask and absolute position bias, the orientation optical flow of the two stages of contraction and relaxation is interactively modeled to locate key muscle regions related to facial action units and extract the corresponding key region features.

[0010] Furthermore, the cross-attention mechanism of local neighborhood mask and absolute position bias specifically includes: A query matrix is ​​generated using the directional optical flow features of one stage, and a key matrix and a value matrix are generated using the directional optical flow features of another stage. After calculating the similarity between the query matrix and the key matrix, a local neighborhood mask matrix is ​​introduced to restrict attention to interact only within a given neighborhood radius, and an absolute position bias matrix is ​​introduced to penalize positions outside the neighborhood range. The obtained attention scores are normalized and multiplied by the value matrix to output features that fuse cross-stage directional relationships and spatial location information.

[0011] Furthermore, the spatiotemporal relationship modeling module includes at least a spatial saliency enhancement block and a dynamic temporal fusion block; The spatial saliency enhancement block processes the key region features of the input through parallel channel attention branches and spatial attention branches to obtain spatially enhanced features; The dynamic temporal fusion block inputs historical features and current features into the gated temporal branch and attention branch, and after processing, obtains temporal enhanced features.

[0012] Furthermore, the channel attention branch learns channel weights through global pooling and convolution, and the spatial attention branch constructs a spatial attention map through a convolutional sequence. The two branches are then fused to obtain spatial enhancement features. The gated temporal branch captures short-term dynamic changes through update and reset gates, while the attention branch models global dependencies through queries, keys, and values. The two branches are then weighted and fused to obtain temporal enhancement features.

[0013] In a second aspect, the present invention provides a micro-expression recognition system based on optical flow enhancement and spatiotemporal relationship modeling, comprising: Data acquisition module: used to acquire micro-expression video sequences and perform preprocessing to obtain the start frame, peak frame and end frame; Optical flow feature extraction module: used to extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; Directional residual optical flow enhancement module: used to process the optical flow features to obtain enhanced cross-stage motion difference features; Dual-stage orientation and position encoding module: used to process the cross-stage motion difference features and extract key region features; Spatiotemporal relationship modeling module: used to jointly model the key region features through the spatiotemporal relationship modeling module to obtain high-level discrimination features; Classification and recognition module: used to process the high-level discriminative features to obtain micro-expression recognition results.

[0014] In a third aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0015] In a fourth aspect, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention addresses the problems of existing micro-expression recognition technologies, such as difficulty in capturing subtle movements in complex environments, susceptibility to interference, and neglect of phase correlation. It replaces the traditional adjacent-frame optical flow by selecting two sets of key frame optical flow: the start frame and the peak frame, and the peak frame and the end frame. This focuses on characterizing the key motion differences in micro-expressions from contraction to relaxation, highlighting the core dynamic information of the expression. Furthermore, through a directional residual optical flow enhancement module, the directional residual of the optical flow during the relaxation phase is extracted and fused with the optical flow during the contraction phase, actively amplifying subtle facial deformations that are difficult to detect, while effectively suppressing changes in illumination and head movement. The model effectively mitigates strong interference noise such as facial swaying. Furthermore, by introducing local neighborhood masks and absolute position biases through a two-stage directional position encoding module, the model can accurately focus on action unit-related areas such as eyebrows, eye area, and corners of the mouth, avoiding interference from irrelevant backgrounds. The spatiotemporal relationship modeling module jointly optimizes spatial saliency, short-term dynamic changes, and long-term temporal dependencies, significantly improving the model's ability to express the complex evolution of micro-expressions. This enhances the accuracy and robustness of micro-expression recognition under strong interference environments, providing reliable support for practical applications such as lie detection interrogation, psychological assessment, and human-computer interaction. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention; Figure 2 This is a structural diagram of the dual-stage directional position encoding module of the present invention; Figure 3 This is a structural diagram of the spatiotemporal relationship modeling module of the present invention. Detailed Implementation

[0019] The following detailed description is exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0021] In this embodiment of the invention, the data collection and processing strictly adhere to the requirements of relevant laws and regulations, obtaining informed consent or separate consent from the data subject, and conducting subsequent data use and processing within the scope of laws, regulations, and the data subject's authorization. All data acquisition in this embodiment is based on compliance with laws and regulations and user consent, representing the lawful application of the data.

[0022] Example 1 As described in the background section, existing technologies cannot actively amplify the subtle differences in micro-expression movements across different stages under strong interference environments, nor can they achieve precise focusing on key areas through spatiotemporal modeling. This leads to the following consequences: For example, in secure interrogation scenarios, when test subjects deliberately conceal their true emotions, slight head movements or changes in lighting can mask already subtle facial movements, making it easy for existing methods to miss crucial clues of lies, resulting in misjudgments or missed judgments. Furthermore, in psychological diagnosis scenarios, unconscious head movements caused by patient tension can interfere with the localization of key muscle areas, making it difficult for doctors to rely on automated analysis systems to obtain genuine emotional feedback; they still need to manually review videos frame by frame, which is inefficient. Therefore, existing technologies lack the ability to actively enhance the differences in movement between the contraction and relaxation stages of micro-expressions and lack a precise focusing mechanism for key areas, leading to a sharp decline in recognition performance in complex dynamic environments.

[0023] To address the aforementioned issues, this invention employs the following approach: First, it replaces the traditional adjacent-frame optical flow with a dual-stage keyframe optical flow consisting of a start frame and a peak frame, and a peak frame and an end frame, to centrally characterize the motion differences between the contraction and relaxation phases. Second, it introduces a directional residual optical flow enhancement module to extract directional residuals from the relaxation phase optical flow and superimpose them onto the contraction phase optical flow, actively amplifying subtle cross-phase motion differences while suppressing background noise. Finally, through a dual-stage directional position encoding module, it accurately locates key muscle regions such as eyebrows and corners of the eyes using cross-attention based on local neighborhood masks and absolute position offsets, and combines this with a spatiotemporal relationship modeling module to jointly capture local details and dynamic dependencies. Thus, even in environments with strong interference, the system can still focus on realistic facial expression signals, improving the accuracy and robustness of recognition.

[0024] In a typical embodiment of the present invention, such as Figure 1 As shown, a micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling is disclosed. The specific steps include: S1: Acquire the micro-expression video sequence and preprocess it to obtain the start frame, peak frame and end frame; S2: Extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; S3: The optical flow features are processed by the directional residual optical flow enhancement module to obtain enhanced cross-stage motion difference features; the cross-stage motion difference features are processed by the dual-stage directional position encoding module to extract key region features; the key region features are jointly modeled by the spatiotemporal relationship modeling module to obtain high-level discrimination features; S4: Process the high-level discriminative features to obtain micro-expression recognition results.

[0025] The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling described above will be explained in detail below with reference to specific implementation methods.

[0026] In step S1, after acquiring the micro-expression video sequence, preprocessing operations such as face detection, alignment, cropping, and normalization are performed, and the start frame, peak frame, and end frame are output via a keyframe prediction network. The preprocessing specifically includes: performing face detection on each frame of the video sequence to extract facial regions; aligning the facial regions based on facial key points; cropping and adjusting the aligned facial regions to a uniform size; and normalizing the cropped facial images.

[0027] Acquiring micro-expression video sequences is typically done using high-speed cameras in a controlled experimental environment. Because micro-expressions are extremely short-lived (usually only 1 / 25 to 1 / 3 of a second), ordinary cameras struggle to capture their complete dynamic changes; therefore, specialized equipment with high frame rates (e.g., 200 frames per second) is required. During the acquisition process, researchers induce genuine emotions in the subjects (e.g., watching videos or receiving question-and-answer stimuli), causing them to unconsciously produce subtle, difficult-to-fake facial muscle movements. Furthermore, publicly available micro-expression datasets (such as CASME II, SMIC, SAMM, etc.) provide standardized video samples annotated by professional psychologists, containing clear start, peak, and end frame identifiers, which can be directly used for algorithm research. The raw video data obtained using the above acquisition methods is then used in subsequent processes.

[0028] In this embodiment, the input micro-expression video sequence is represented as: ,in, Let t represent the image at frame t, and T represent the total number of frames in the video.

[0029] To eliminate adverse factors such as background interference, subtle head pose changes, and inconsistencies in facial region positions between different samples in the original micro-expression videos, this invention first performs face detection on each frame of the video sequence to accurately extract the facial region. Then, based on the detected facial key points (such as the eyes, nose tip, and corners of the mouth), the facial region is aligned to ensure spatial consistency across different frames. Next, the aligned facial region is cropped and uniformly scaled to a preset size (e.g., fixed width and height, 28x28) to meet the input dimension requirements of subsequent optical flow calculations and network feature extraction. Afterward, the cropped and scaled facial image is normalized to effectively reduce interference from illumination variations, contrast fluctuations, and image intensity differences in feature extraction. After the above preprocessing, the system extracts motion features from the video sequence frame by frame and inputs them into the keyframe prediction network. The network receives candidate temporal features frame by frame and uses a built-in temporal modeling module (which can be one or more of Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), Temporal Convolutional Networks (TCN), or Transformers, without specific limitations) to learn the dynamic relationships between frames, and then outputs the micro-expression intensity response value corresponding to each frame. Finally, based on these response values, the key stage frames of the micro-expression are determined: the onset frame (corresponding to the moment when the micro-expression begins to appear), the apex frame (corresponding to the moment when the facial muscle movement amplitude is the largest), and the offset frame (corresponding to the moment when the micro-expression gradually fades back to neutral).

[0030] Through the above method, the present invention can accurately locate the complete change process of facial muscles from contraction to relaxation when micro-expressions occur, laying the foundation for subsequent modeling of the motion relationship between key frame pairs. That is, it adopts two sets of features: optical flow between the start frame and the peak frame (onset-apex) and optical flow between the peak frame and the end frame (apex-offset), instead of the traditional optical flow between adjacent frames, so as to more effectively characterize the key motion differences between each stage of micro-expression.

[0031] In step S2, the optical flow features between the start frame and the peak frame, and between the peak frame and the end frame are extracted. This includes: using the TV-L1 optical flow algorithm (an optical flow estimation algorithm based on total variation and L1 norm constraints) to calculate the optical flow maps between the start frame and the peak frame, and between the peak frame and the end frame, respectively. Each optical flow map is composed of three channels: horizontal displacement component, vertical displacement component, and optical flow strain amplitude.

[0032] In this embodiment, based on the selected keyframes, the TV-L1 optical flow algorithm is used to calculate the following: 1) Optical flow graph between the starting frame onset and the peak frame apex, denoted as .

[0033] 2) Optical flow graph between the peak frame apex and the end frame offset, denoted as... .

[0034] Each optical flow map consists of three channels: (1) horizontal displacement component u; (2) vertical displacement component v; and (3) optical flow strain amplitude os.

[0035] The optical flow strain amplitude os(x, y) is calculated based on the spatial gradient of the optical flow field, and its formula is as follows:

[0036] By concatenating u, v, and os along the channel dimension, the final motion representation is formed: Where H and W represent the feature map height and width, respectively. This method allows for the simultaneous encoding of motion direction, motion amplitude, and local deformation, thus providing a more comprehensive description of subtle facial movements related to micro-expressions.

[0037] In S3, micro-expressions typically manifest as facial muscle contraction during the initial to peak phase and as facial muscle relaxation during the peak to end phase, while the corresponding activated muscle regions are generally relatively consistent in spatial location. To utilize this inter-phase difference and consistency, this invention introduces a directional residual optical flow enhancement module to enhance subtle motion patterns associated with micro-expressions while suppressing background noise and irrelevant disturbances.

[0038] The optical flow features are processed by the directional residual optical flow enhancement module to obtain enhanced cross-stage motion difference features. This includes: extracting directional components from the optical flow of the relaxation stage between the peak frame and the end frame and calculating the directional residual; then superimposing the directional residual onto the optical flow of the contraction stage between the start frame and the peak frame to highlight the consistency and motion difference of micro-expressions between the contraction and relaxation stages, thereby obtaining enhanced cross-stage motion difference features.

[0039] In this embodiment, It can correspond to the optical flow map from the start frame to the peak frame. Optical flow maps corresponding to the peak frame to the end frame. Optical flow from the relaxation phase. Extract its directional component This directional component is used to represent the overall motion direction information from the peak to the end phase, and to calculate the directional residual:

[0040] This operation preserves more discriminative local deformation differences by removing the directional bias from the optical flow during the relaxation phase. This operation can remove the global directional bias while preserving local deformation variations.

[0041] Integrate directional residual information with optical flow during the contraction phase By fusing the two components, we obtain the enhanced optical flow representation:

[0042] in, The weighting coefficient for controlling the contribution of residual information across stages is preferably 0.8. This can be achieved by adjusting... This allows for a balance between enhancing key motions and suppressing noise.

[0043] Through the above processing, the key motion patterns of micro-expressions that are consistent in the contraction and relaxation phases can be enhanced, the ability to express subtle motion features can be improved, and the interference of background noise, lighting disturbances and non-expression motion on the recognition results can be reduced.

[0044] To further locate key muscle movement regions related to micro-expressions, this invention introduces a two-stage orientation position encoding module after orientation residual enhancement. This module receives the enhanced orientation optical flow features from the two stages and the corresponding orientation maps. By modeling the orientation change relationship between the contraction and relaxation phases, it learns the regional position cues related to the action unit, thereby suppressing interference from redundant regions.

[0045] The enhanced cross-stage motion difference features are input into the two-stage orientation and position encoding module. By introducing a cross-attention mechanism of local neighborhood mask and absolute position bias, the orientation optical flow of the two stages of contraction and relaxation is interactively modeled to accurately locate key muscle regions related to facial action units, such as eyebrows, eye area, and corner of mouth, and extract the corresponding key region features.

[0046] Specifically, the cross-attention mechanism of local neighborhood masking and absolute position bias includes: generating a query matrix using the directional optical flow features of one stage, and generating a key matrix and a value matrix using the directional optical flow features of another stage; after calculating the similarity between the query matrix and the key matrix, a local neighborhood mask matrix is ​​introduced to restrict attention to interact only within a given neighborhood radius, and an absolute position bias matrix is ​​introduced to impose a penalty on positions outside the neighborhood range that decreases with increasing distance; the attention score after adding the mask and bias is normalized by Softmax and multiplied by the value matrix to output a feature that integrates cross-stage directional relationships and spatial position information.

[0047] The local neighborhood mask is used to restrict attention computation to interact only within a local spatial range within a given neighborhood radius to highlight local relevance, while the absolute position offset is used to impose a distance-based attenuation penalty on positions outside the neighborhood range. Together, they work to achieve cross-stage feature alignment and precise localization of key muscle regions while preserving spatial structure constraints.

[0048] like Figure 2As shown, in this embodiment, the specific implementation of the cross-attention mechanism of local neighborhood mask and absolute position bias is as follows: First, the directional optical flow features and directional maps of the two stages are input into the directional encoder to obtain the corresponding feature representations x1 and x2; then, the feature x1 of one stage (such as the contraction stage) is used to generate a query matrix through 1×1 convolution. The key matrix is ​​generated by 1×1 convolution using features x2 from another stage (such as the diastolic stage). Sum matrix This establishes a cross-feature interaction relationship between the two stages. Then, the similarity between the query matrix and the key matrix is ​​calculated, and a local neighborhood mask matrix M and an absolute position bias matrix P are introduced into the similarity matrix. The local neighborhood mask matrix M is used to restrict attention to only within a given neighborhood radius. The attention is processed within a certain range to highlight local correlations. An absolute position bias matrix P is used to impose a decay penalty on positions outside the neighborhood, thus explicitly encoding spatial location information. Then, the attention scores after adding the mask and position bias are Softmax normalized to obtain the attention weights under position constraints. Finally, the attention weights are compared with the value matrix... The product is multiplied to output a feature representation that integrates cross-stage directional relationships and absolute positional information. This method allows for feature alignment and information exchange between the contraction and relaxation phases while preserving local spatial structure constraints, thereby enabling more accurate localization of key muscle movement regions related to micro-expressions.

[0049]

[0050]

[0051] in, , and These represent the query, key, and value, respectively. and It is obtained by convolving x2 with a 1×1 matrix. It is obtained by convolving x1 with a 1×1 matrix; Indicates the scaling factor; This represents element-wise multiplication; Used to restrict feature interactions within a local neighborhood; Used to introduce an absolute position bias into the attention matrix; It is a positive attenuation parameter; Let be the neighborhood radius, preferably 1.

[0052] This process enables the model to focus on key motion regions with causal relationships, such as the eyebrows, the area around the eyes, the nostrils, and the corners of the mouth—locations where micro-expressions are prone to local deformation—thus providing more discriminative input features for subsequent spatiotemporal relationship modeling.

[0053] In micro-expression recognition, facial muscle movements are not only highly localized spatially (concentrated only in tiny areas such as eyebrows, corners of the eyes, and corners of the mouth), but also exhibit dynamic characteristics of "rapid evolution and distinct stages" in time (often less than a second from start to peak and then to end). Traditional convolutional neural networks excel at extracting spatial features but struggle to capture inter-frame temporal dependencies, while recurrent neural networks, although capable of modeling time series, tend to overlook local spatial details. More importantly, there is an inherent causal relationship between the contraction and relaxation phases of micro-expressions: they have spatial consistency in the movement area, but their movement directions are exactly opposite. If only the two phases are modeled independently, this cross-stage dynamic evolution pattern will be lost. Therefore, this invention designs a spatiotemporal relationship modeling module, which precisely focuses on key muscle regions through spatial saliency enhancement blocks, simultaneously captures short-term inter-frame changes and long-term dependencies through dynamic temporal fusion blocks, and explicitly models the motion transition patterns between contraction and relaxation through stage association enhancement blocks. This achieves a joint representation of the complex spatiotemporal dynamics of micro-expressions, providing more discriminative high-level features for subsequent accurate classification.

[0054] The extracted key regional features are input into the spatiotemporal relationship modeling module. The spatial saliency enhancement block highlights the channels and spatial responses of key muscle regions. The dynamic temporal fusion block uses gating mechanism and multi-head self-attention to capture short-term changes and long-term dependencies. The stage association enhancement block models the dynamic relationship between the contraction and relaxation stages, thereby obtaining high-level discriminative features that integrate local details and global temporal information.

[0055] The spatiotemporal relationship modeling module is used to jointly model the spatial and temporal dependencies of features, so as to simultaneously characterize local detail changes, global dynamic trends, and cross-stage correlations. The spatiotemporal relationship modeling module includes at least a spatial saliency enhancement block and a dynamic temporal fusion block.

[0056] Specifically, the spatial saliency enhancement block processes the key region features of the input through parallel channel attention branches and spatial attention branches. The channel attention branch uses global average pooling and bottleneck convolution to learn the dependencies between channels, while the spatial attention branch uses convolutional sequences to construct a spatial attention map. The spatial enhancement features are obtained by fusing the outputs of the two branches.

[0057] like Figure 3As shown, in this embodiment, the channel attention branch is used to learn inter-channel dependencies and assign weights to different channels. Two optical flow maps are first fed into the encoder to obtain high-level representations of the contraction phase (i.e., Onset-Apex) and relaxation phase (i.e., Apex-Offset) motion. Then, the optical flow maps generated by CMPL are... , Add these features to the corresponding motion features to inject spatial cues related to the AU. Finally, the augmentation features are arranged in two temporal sequences: positive (starting vertex to vertex offset) features. and backward (vertices offset to the starting vertex) feature .

[0058] First, spatial information is compressed using global average pooling, then channel dependency is modeled using two 1×1 convolutional layers with bottleneck structures. For a given temporal feature... The channel attention weight is calculated as follows:

[0059] in, Indicates global average pooling. and This represents two 1×1 convolutional layers. Represents the ReLU activation function. This represents the Sigmoid function. This represents the channel attention weight.

[0060] The spatial attention branch is used to emphasize key local locations; it constructs a spatial attention map using a sequence of convolutions.

[0061] in, , Represents a 3×3 convolution. . This represents the ReLU activation function.

[0062] The results of the channel enhancement branch and the spatial enhancement branch are fused to obtain the output features:

[0063] By using channel-space attention blocks, the model can simultaneously highlight key micro-expression regions from both the channel and spatial dimensions, thereby improving its spatial feature discrimination capabilities.

[0064] Microexpressions are characterized by short duration and rapid evolution, making it difficult to fully characterize their dynamic processes by modeling only local temporal continuity or only long-range dependencies. Therefore, this invention introduces a gated multi-head temporal block, which combines a gated temporal mechanism and a multi-head self-attention mechanism in the same module to simultaneously capture short-term changes and long-term dependencies.

[0065] The dynamic temporal fusion block inputs historical features and current features into a gated temporal branch and a multi-head self-attention branch. The gated temporal branch selectively updates historical information through update and reset gates to capture short-term dynamic changes, while the multi-head self-attention branch obtains queries, keys, and values ​​through linear projection to model global long-term dependencies. The two branches are then weighted and fused to obtain the temporal enhancement features. The historical features refer to the feature representation input at the previous time or stage, and the current features refer to the feature representation input at the current time or stage.

[0066] The dynamic temporal fusion block will come from the forward augmentation representation. and backward enhancement representation The two-stage features are used as input. In each time sequence, the first stage is considered as historical information. The second phase is considered the current information. For a forward sequence, the starting point-vertex represents historical information, while the vertex-offset represents current information; for a reverse sequence, the order is reversed.

[0067] First, generate update and reset gates based on the current input: (11) in, and These represent updating the door and resetting the door, respectively. , This represents a learnable linear transformation matrix. , Indicates the bias term. This represents the Sigmoid function. Subsequently, a reset gate is applied to historical features, generating candidate states: (12) in, and For learnable parameters, Let represent the hyperbolic tangent activation function. Finally, the gating time is expressed as: (13) This operation is used to interpolate and update between historical features and candidate states to balance the preservation of old information with the injection of new information.

[0068] To model the global dependencies between different positions within the current sequence, the current features... Perform a linear projection to obtain the query Q, key K, and value V. For the first... Each attention head outputs the following: (14) By concatenating and linearly mapping the results of all attention heads, we obtain the attention branch output: (15) in, Indicates the output projection matrix. This indicates splicing based on feature dimensions.

[0069] The gated timing representation and the multi-head attention representation are fused together by weights to obtain the output: (16) in, This is a balancing coefficient used to control the contribution ratio of the gating branch and the attention branch.

[0070] After directional residual optical flow enhancement, a two-stage directional position encoding module, and spatiotemporal relationship modeling, a high-level feature representation with strong discriminative ability, namely high-level discriminative features, is obtained. These high-level discriminative features are then input into a classification module, which includes at least a fully connected layer and a Softmax classification function. The fully connected layer maps the features to a preset class space, and the Softmax classification function outputs the predicted probability for each class. The class with the highest probability is taken as the micro-expression recognition result.

[0071] Example 2 In a typical embodiment of the present invention, a micro-expression recognition system based on optical flow enhancement and spatiotemporal relationship modeling is provided, comprising: Data acquisition module: used to acquire micro-expression video sequences and perform preprocessing to obtain the start frame, peak frame and end frame; Optical flow feature extraction module: used to extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; Directional residual optical flow enhancement module: used to process the optical flow features to obtain enhanced cross-stage motion difference features; Dual-stage orientation and position encoding module: used to process the cross-stage motion difference features and extract key region features; Spatiotemporal relationship modeling module: used to jointly model the key region features through the spatiotemporal relationship modeling module to obtain high-level discrimination features; Classification and recognition module: used to process the high-level discriminative features to obtain micro-expression recognition results.

[0072] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0073] Example 4 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0074] The steps and methods involved in Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0075] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0077] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling, characterized in that, include: The micro-expression video sequence is acquired and preprocessed to obtain the start frame, peak frame, and end frame. Extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; The optical flow features are processed by the directional residual optical flow enhancement module to obtain enhanced cross-stage motion difference features; the cross-stage motion difference features are processed by the dual-stage directional position encoding module to extract key region features; the key region features are jointly modeled by the spatiotemporal relationship modeling module to obtain high-level discrimination features; The high-level discriminative features are processed to obtain micro-expression recognition results.

2. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 1, characterized in that, The TV-L1 optical flow algorithm is used to calculate the optical flow maps between the start frame and the peak frame, and between the peak frame and the end frame. Each optical flow map is composed of three channels: horizontal displacement component, vertical displacement component, and optical flow strain amplitude.

3. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 1, characterized in that, The directional component is extracted from the optical flow during the relaxation phase between the peak frame and the end frame, and the directional residual is calculated. Then, the directional residual is superimposed on the optical flow during the contraction phase between the start frame and the peak frame to obtain the enhanced cross-phase motion difference feature.

4. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 1, characterized in that, The enhanced cross-stage motion difference features are input into the two-stage orientation and position encoding module. By introducing a cross-attention mechanism of local neighborhood mask and absolute position bias, the orientation optical flow of the two stages of contraction and relaxation is interactively modeled to locate key muscle regions related to facial action units and extract the corresponding key region features.

5. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 4, characterized in that, The cross-attention mechanism of local neighborhood mask and absolute position bias specifically includes: A query matrix is ​​generated using the directional optical flow features of one stage, and a key matrix and a value matrix are generated using the directional optical flow features of another stage. After calculating the similarity between the query matrix and the key matrix, a local neighborhood mask matrix is ​​introduced to restrict attention to interact only within a given neighborhood radius, and an absolute position bias matrix is ​​introduced to penalize positions outside the neighborhood range. The obtained attention scores are normalized and multiplied by the value matrix to output features that fuse cross-stage directional relationships and spatial location information.

6. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 1, characterized in that, The spatiotemporal relationship modeling module includes at least a spatial saliency enhancement block and a dynamic temporal fusion block; The spatial saliency enhancement block processes the key region features of the input through parallel channel attention branches and spatial attention branches to obtain spatially enhanced features; The dynamic temporal fusion block inputs historical features and current features into the gated temporal branch and attention branch, and after processing, obtains temporal enhanced features.

7. The micro-expression recognition method based on optical flow enhancement and spatiotemporal relationship modeling as described in claim 6, characterized in that, The channel attention branch learns channel weights through global pooling and convolution, and the spatial attention branch constructs a spatial attention map through a convolutional sequence. The two branches are fused to obtain spatial enhancement features. The gated temporal branch captures short-term dynamic changes through update and reset gates, while the attention branch models global dependencies through queries, keys, and values. The two branches are then weighted and fused to obtain temporal enhancement features.

8. A micro-expression recognition system based on optical flow enhancement and spatiotemporal relationship modeling, characterized in that, include: Data acquisition module: used to acquire micro-expression video sequences and perform preprocessing to obtain the start frame, peak frame and end frame; Optical flow feature extraction module: used to extract optical flow features between the start frame and the peak frame, and between the peak frame and the end frame; Directional residual optical flow enhancement module: used to process the optical flow features to obtain enhanced cross-stage motion difference features; Dual-stage orientation and position encoding module: used to process the cross-stage motion difference features and extract key region features; Spatiotemporal relationship modeling module: used to jointly model the key region features through the spatiotemporal relationship modeling module to obtain high-level discrimination features; Classification and recognition module: used to process the high-level discriminative features to obtain micro-expression recognition results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

Images