Micro-expression recognition method and device based on multi-scale optical flow attention mechanism

Abstract

The application provides a micro-expression recognition method and device based on a multi-scale optical flow attention mechanism, and comprises the following steps: constructing a pre-trained micro-expression recognition model, wherein the micro-expression recognition model comprises a preprocessing module, a dynamically moving branch and a space-time appearance branch arranged in parallel, a cross-modal fusion module and a classification head; and inputting a micro-expression video to be recognized into the micro-expression recognition model to obtain an emotion category recognition result. According to the scheme, the dynamic information of the optical flow is used as a guide to significantly improve the attention sensitivity of the model to the micro-movement of key regions such as the orbicularis oculi muscle and the zygomaticus minor muscle, thereby improving the accuracy of emotion category prediction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a micro-expression recognition method and apparatus based on a multi-scale optical flow-guided attention mechanism. Background Technology

[0002] Microexpressions are involuntary facial movements driven by the subconscious, typically lasting 40-500ms. They involve subtle contractions of facial muscles such as the orbicularis oculi and zygomaticus minor, with a range of motion often less than 1 pixel. They can bypass conscious control and directly reflect an individual's true emotional state, such as fear, disgust, or surprise. Compared to macroexpressions, which last longer, the transient nature and low salience of microexpressions give them unique value in the field of affective computing. However, the implementation of these applications heavily relies on the accuracy of micro-expression recognition (MER) technology, and the inherent characteristics of microexpressions make them a challenging problem in computer vision and affective computing.

[0003] The core challenges of micro-expression recognition are concentrated in three aspects: First, it is difficult to capture subtle features. The muscle movements involved in micro-expressions are extremely small and easily masked by noise such as facial texture and lighting changes. Traditional visual feature extraction methods are difficult to distinguish between effective motion and interference information. Second, dynamic temporal modeling is complex. The complete evolution of micro-expressions requires going through three stages: "initiation-peak-shift". The nonlinear dynamic changes within a short time series require the model to capture both inter-frame correlations and high-frequency details. Third, data is scarce and its distribution is unbalanced. Existing public datasets such as SAMM, CASME II, and SMIC are limited in size and have uneven class distribution, which limits the generalization ability of the model.

[0004] Early micro-expression recognition methods were mainly based on manual feature extraction. For example, LBP-TOP describes motion through spatiotemporal local binary patterns, but it relies on manually designed rules and struggles to capture the high-frequency dynamics of micro-expressions. With the widespread application of the start-frame-peak-frame optical flow method, methods based on convolutional neural networks (CNNs) have gradually become mainstream. Early deep learning methods, such as CNN-based models, focused on static feature extraction and lacked temporal modeling capabilities, failing to characterize the evolutionary patterns between frames. While optical flow methods can capture dynamic information, most studies only utilize horizontal (u) and vertical (v) components, ignoring the strain component (z) reflecting muscle stretching / compression, resulting in the loss of dynamic information. Although some studies have begun to explore image-optical flow multimodal fusion, most adopt a single peak frame or a "start-frame + peak frame" scheme, failing to cover the complete "start-peak-offset" temporal sequence and losing key dynamic transition information. Even the few methods that attempt multi-frame fusion often employ simple splicing or weighted summation strategies, failing to address the modal heterogeneity between static image texture and dynamic optical flow motion, causing key features to be overwhelmed by redundant information.

[0005] The dynamic evolution of micro-expressions demands accurate temporal modeling capabilities from the model. Early methods often employed recurrent neural networks (RNNs) and their variants, such as LSTM, to capture inter-frame dependencies through gating mechanisms, but these suffer from the vanishing gradient problem over long sequences. In recent years, Transformers have demonstrated advantages in temporal modeling due to their self-attention mechanism. Some studies have introduced retention mechanisms to balance performance and efficiency, but they remain insufficient in uncovering cross-modal temporal correlations and have not fully integrated frequency domain information to capture high-frequency dynamic details of micro-expressions, such as the instantaneous changes in muscle contraction. These issues collectively hinder the practical application of micro-expression recognition technology, necessitating a technical solution that enables multi-scale dynamic modeling and deep cross-modal fusion. Summary of the Invention

[0006] This application provides a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism. By using dynamic optical flow information as a guide, the method significantly improves the model's sensitivity to micro-movements in key areas such as the orbicularis oculi muscle and zygomaticus minor muscle, thereby improving the accuracy of emotion category prediction.

[0007] In a first aspect, embodiments of this application provide a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism, the method comprising:

[0008] Construct a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head;

[0009] The micro-expression video to be identified is input into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame, and end frame of the video. Based on the start frame and peak frame, three-channel optical flow dynamic features are obtained. Non-facial regions are covered on the grayscaled start frame, peak frame, and end frame to obtain three grayscale appearance features, including the start frame grayscale appearance features, peak frame grayscale appearance features, and end frame grayscale appearance features. The dynamic motion branch uses multiple Transformer blocks of different scales connected in series to extract the facial muscle motion trajectory from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal extrapolation... The observation branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame, thus obtaining temporal difference features. Multiple feature enhancement blocks of different scales are then used to enhance these temporal difference features, resulting in multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, while the outputs of the remaining feature enhancement blocks are second intermediate features. An optical flow-guided fusion block uses attention weighting to fuse each second intermediate feature with a first intermediate feature of the corresponding scale, and then inputs this fusion into the next feature enhancement block. The cross-modal fusion module uses a dual-path global attention mechanism to fuse the multi-scale spatiotemporal appearance features and multi-scale optical flow features to obtain fused features. The classification head outputs the emotion category recognition result of the micro-expression based on the fused features.

[0010] Secondly, embodiments of this application provide a micro-expression recognition device based on a multi-scale optical flow-guided attention mechanism, comprising:

[0011] A construction module is used to build a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head.

[0012] The recognition module is used to input the micro-expression video to be recognized into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame, and end frame of the micro-expression video to be recognized. Based on the start frame and peak frame, it obtains three-channel optical flow dynamic features. The grayscale start frame, peak frame, and end frame are then masked for non-facial regions to obtain three grayscale appearance features, including the grayscale appearance features of the start frame, peak frame, and end frame. The dynamic motion branch uses multiple Transformer blocks of different scales connected in series to extract the facial muscle motion trajectory from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal appearance branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame, thus obtaining temporal difference features. Multiple concatenated feature enhancement blocks of different scales are then used to enhance these temporal difference features, resulting in multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, while the outputs of the remaining feature enhancement blocks are second intermediate features. An optical flow-guided fusion block uses attention weighting to fuse each second intermediate feature with a first intermediate feature of the corresponding scale, and then inputs this fusion into the next feature enhancement block. The cross-modal fusion module uses a dual-path global attention mechanism to fuse the multi-scale spatiotemporal appearance features and multi-scale optical flow features to obtain fused features. The classification head outputs the emotion category recognition result for micro-expressions based on the fused features.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism.

[0014] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism.

[0015] The main contributions and innovations of this invention are as follows:

[0016] This scheme employs multiple Transformer blocks of different scales in a series to accurately capture the subtle movement trajectories of facial muscles in its dynamic motion branch. In the spatiotemporal appearance branch, multiple optical flow guidance modules are used to significantly enhance the sensitivity of the method to the micro-movements of key areas such as the orbicularis oculi and zygomaticus minor muscles. The feature enhancement module uses a spatial-frequency dual-path feedforward network layer to effectively extract high-frequency dynamic details of muscle contraction, thus avoiding noise interference with subtle features. This scheme uses a temporal difference extraction module to extract the temporal deviation between the start frame, peak frame, and end frame, thereby integrating feature associations from different temporal stages and addressing the problem of existing methods focusing only on the "start-peak" stage and losing dynamic transition information in the offset stage. The cross-modal fusion module in this scheme deeply mines the cross-modal association between dynamic motion and spatiotemporal appearance features through bidirectional global attention, and refines discriminative information through channel self-attention, effectively solving the redundancy interference problem caused by modal heterogeneity and significantly improving the robustness of the fused features.

[0017] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram of the model structure of a micro-expression recognition model according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the structure of a feature enhancement block according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the HSM-SSD layer structure in a feature enhancement block according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of a space-frequency dual-path feedforward network in a feature enhancement block according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an optical flow-guided fusion block according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of a cross-modal fusion module according to an embodiment of this application;

[0025] Figure 7 This is a branch structure diagram of optical flow features of a dual-path global attention layer according to an embodiment of this application;

[0026] Figure 8 This is a structural diagram of an optical flow channel attention unit and an appearance channel attention unit according to embodiments of this application;

[0027] Figure 9 This is a structural block diagram of a micro-expression recognition device based on a multi-scale optical flow-guided attention mechanism according to an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0030] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0031] Example 1

[0032] This application provides a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism. Specifically, refer to... Figure 1 The method includes:

[0033] Construct a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head;

[0034] The micro-expression video to be identified is input into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame, and end frame of the video. Based on the start frame and peak frame, three-channel optical flow dynamic features are obtained. Non-facial regions are covered on the grayscale start frame, peak frame, and end frame to obtain three grayscale appearance features. The dynamic motion branch uses multiple Transformer blocks of different scales in series to extract facial muscle motion trajectories from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal appearance branch uses temporal difference extraction... The temporal difference features are obtained by extracting the inter-frame differences of the start frame, peak frame, and end frame. Multiple feature enhancement blocks of different scales are used to enhance the full temporal appearance changes in the temporal difference features to obtain multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, and the outputs of the remaining feature enhancement blocks are second intermediate features. An optical flow-guided fusion block is used to fuse each second intermediate feature with the first intermediate feature of the corresponding scale in an attention-weighted manner. The cross-modal fusion module fuses the multi-scale spatiotemporal appearance features and multi-scale optical flow features based on a dual-path global attention mechanism to obtain fused features. The classification head outputs the emotion category recognition result of micro-expressions based on the fused features.

[0035] In the current embodiment, the start frame is the image frame at the moment when the micro-expression begins, the peak frame is the image frame at the moment when the micro-expression intensity reaches its peak, and the end frame is the image frame at the moment when the micro-expression ends.

[0036] Specifically, the start frame, peak frame, and end frame are automatically acquired by the trained preprocessing module. The preprocessing module preprocesses the micro-expression video dataset with labeled start frames, peak frames, and end frames during the training process, thereby accurately capturing the start frame, peak frame, and end frame of the micro-expression video.

[0037] Specifically, if the micro-expression video dataset used for training does not label the start frame, peak frame, and end frame, then the frame index of each frame in the micro-expression video to be identified is obtained. The image corresponding to the first frame index in the micro-expression video to be identified is taken as the start frame, the image corresponding to the last frame index in the micro-expression video to be identified is taken as the end frame, the average of the first frame index and the last frame index is taken as the intermediate frame index, and the image corresponding to the intermediate frame index is taken as the peak frame.

[0038] In the current embodiment, the horizontal optical flow field, vertical optical flow field, and motion strain component are obtained based on the start frame and the peak frame. The horizontal optical flow field, vertical optical flow field, and motion strain component are then stitched together to obtain the three-channel optical flow dynamic features.

[0039] For example, this scheme uses the TV-L1 optical flow algorithm to extract the total variational regularized optical flow with L1 norm based on the start frame and the peak frame to obtain the horizontal optical flow field and the vertical optical flow field, and then calculates the motion strain component based on the horizontal optical flow field and the vertical optical flow field.

[0040] Specifically, the kinematic strain components are calculated using matrices applied to normal strain and shear.

[0041] Specifically, before obtaining the horizontal optical flow field, vertical optical flow field, and motion strain components based on the start frame and peak frame, the start frame, peak frame, and end frame are clipped and aligned.

[0042] In the current embodiment, key facial feature points are extracted from the grayscale start frame, peak frame, and end frame. The convex hull algorithm is used to generate facial masks for the start frame, peak frame, and end frame based on the key facial feature points. The area outside the facial mask is covered to obtain three grayscale appearance features.

[0043] For example, using the dlib toolkit, 68 key facial feature points are extracted from the grayscale start frame, peak frame, and end frame respectively. For each frame image after extracting facial features, frame dynamic smoothing mechanism is used to reduce inter-frame jitter. Then, with the center of the two eyes as the rotation center, the face is rotated to the horizontal by affine transformation based on the slope of the line connecting the two eyes. The face is then cropped into a square area and uniformly scaled to a size of 224×224. At the same time, a facial mask corresponding to each frame image is generated by convex hull algorithm. The background, hair, and other areas outside the facial mask are set to black to cover them, thus obtaining three frames of grayscale appearance features.

[0044] In the current embodiment, the Transformer block in the dynamic motion branch uses the Manhattan self-attention mechanism to calculate the three-channel optical flow dynamic features to obtain the first intermediate feature. The first intermediate feature is input into the next Transformer block for calculation, and the output of the last Transformer block is the multi-scale optical flow feature.

[0045] Specifically, the Transformer block extends RetNet's temporal decay mechanism to the spatial domain, introducing a spatial decay matrix based on Manhattan distance to calculate the Manhattan distance at different locations within the dynamic features of the three-channel optical flow, thus embedding explicit spatial priors into the self-attention mechanism. At the same time, it extends Manhattan self-attention (MaSA) in RetNet to two-dimensional space, enabling bidirectional modeling of two-dimensional images.

[0046] Specifically, in the initial stage of the dynamic motion branch, the Manhattan self-attention mechanism decomposes the self-attention and spatial attenuation matrix along the horizontal and vertical axes of the three-channel optical flow dynamic characteristics, calculates the attention fractions in the two directions respectively, and applies a one-dimensional bidirectional attenuation matrix to reduce the computational burden.

[0047] The dynamic motion branch of this scheme uses four Transformer blocks connected in series. The channel dimensions of the four Transformer blocks are configured as 64, 128, 256, and 512 respectively. The number of attention heads in the four Transformer blocks are 1, 2, 8, and 16 respectively. Specifically, the first Transformer block uses 1 attention head to focus on local subtle motions to avoid noise interference, the second Transformer block uses 2 attention heads to initially expand the receptive field, the third Transformer block uses 8 attention heads to enhance feature interaction capabilities, and the fourth Transformer block uses 16 attention heads to capture complex motion patterns.

[0048] Specifically, a 3×3 convolutional layer with a stride of 2 is connected after each Transformer block to achieve feature scale downsampling. At the same time, the three-channel optical flow dynamic features are mapped to 64 channels through multi-layer convolution stacking, thereby realizing the initial feature embedding of the three-channel optical flow dynamic features.

[0049] In other words, the first intermediate feature channel output by the first Transformer block is 64, the first intermediate feature channel output by the second Transformer block is 128, the first intermediate feature channel output by the third Transformer block is 256, and the multi-scale optical flow feature channel output by the fourth and last Transformer block is 512.

[0050] In the current embodiment, the temporal difference extraction module performs pixel-level difference calculations on the grayscale appearance features of the starting frame and the grayscale appearance features of the peak frame to obtain the first inter-frame difference, performs pixel-level difference calculations on the grayscale appearance features of the peak frame and the grayscale appearance features of the ending frame to obtain the second inter-frame difference, obtains the original frame features, and fuses the original frame features, the first inter-frame difference, and the second inter-frame difference channel by channel to obtain the temporal difference features. The original frame features provide the corresponding global facial contour and static information of fixed wrinkles.

[0051] Specifically, the original frame is obtained by processing the result of stacking the start frame, peak frame and end frame in the time dimension using a 3D convolutional layer. For example, the start frame, peak frame and end frame are stacked in the time dimension to obtain a 5D tensor, and the original frame features are obtained by processing this 5D tensor using a 3D convolutional layer.

[0052] Specifically, the temporal difference extraction block targets the dynamic features of the two key transition stages of micro-expression: the start to the peak and the peak to the end. It performs pixel-level difference operations on the start frame to the peak frame and the peak frame to the end frame, thereby amplifying the grayscale changes caused by muscle movement to enhance the discriminativeness of dynamic texture regions. At the same time, it integrates the inter-frame difference features with the original frame features in a complementary manner. That is, the original frame features retain static structural information such as the global facial contour and fixed wrinkles, while focusing on the dynamic change regions in the temporal difference features, ultimately obtaining temporal difference features that combine transient muscle changes with global facial correlation.

[0053] In the current embodiment, the structure of the feature enhancement block is as follows: Figure 2 As shown, the feature enhancement block includes a multi-scale attention layer, an HSM-SSD layer, and a space-frequency dual-path feedforward network layer connected in sequence. The multi-scale attention layer divides the input features into multiple groups of sub-features based on the number of channels, and calculates the horizontal and vertical spatial attention of each group of sub-features to obtain directional attention weight features. The HSM-SSD layer processes the directional attention weight features to obtain the weighted features of the prediction results. The space-frequency dual-path feedforward network layer uses depthwise separable convolution to extract the local structural features of the weighted features of the prediction results. After lightweighting the local structural features through nonlinear activation, lightweight interactive features are obtained. The lightweight interactive features are divided into multiple sub-regions, and a fast Fourier transform is performed on each sub-region to obtain dynamically weighted sub-features. Then, all dynamically weighted sub-features are recombined through inverse Fourier transform to obtain the output result. The output result is a second intermediate feature or a multi-scale spatiotemporal appearance feature.

[0054] Specifically, the feature enhancement block is an improvement on the EfficientViM Block structure. The conventional settings in the EfficientViM Block structure, such as the HSM-SSD layer and some preprocessing operations for features, will not be described in detail in this solution.

[0055] Specifically, this scheme uses a multi-scale attention layer (EMA) to divide the input features into 16 sub-features on an average scale according to the channel dimension. During the independent attention calculation process for each sub-feature, attention enhancement in the spatial dimension is combined to calculate the directional attention weight features for horizontal and vertical spatial attention, thereby achieving dynamic enhancement of key discrimination regions and suppression of redundant information.

[0056] Specifically, in the multi-scale attention layer, one-dimensional global average pooling is used to compress each group of sub-features along the horizontal and vertical directions to obtain horizontal and vertical sub-features. Attention is then calculated on the horizontal and vertical sub-features to obtain horizontal and vertical sub-feature attention weights. The horizontal and vertical sub-feature attention weights are then multiplied element-wise with the corresponding sub-features to obtain the directional attention weight features. In other words, this scheme uses one-dimensional global average pooling to capture long-distance dependencies. After concatenation, shared convolution transformation, and nonlinear activation, horizontal and vertical attention calculations are performed to obtain the corresponding horizontal and vertical sub-feature attention weights. Finally, the weights in these two directions are multiplied element-wise with the original input sub-features to encode precise spatial location information into the features, thus obtaining the directional attention weight features.

[0057] Specifically, the schematic diagram of the HSM-SSD layer structure in the feature enhancement block is as follows: Figure 3 As shown, the HSM-SSD layer transfers the channel mixing operation from the input feature space to the hidden state space (where the hidden state sequence length N) The number of input channels (D) significantly reduces computational complexity. Simultaneously, a multi-stage hidden state fusion (MSF) mechanism is introduced. At the end of each stage, stage-level features are extracted using global average pooling. These features are then normalized and linearly projected to generate stage predictions. Finally, all stage predictions are weighted and fused using softmax normalization of learnable weights β. This integrates low-order details and high-order semantics of micro-expressions to obtain weighted features of the prediction results. The weight calculation formula is as follows:

[0058]

[0059] in, Let t be the learnable raw weights for stage s, where S is the total number of stages, s and t are both stage indices, and exp is an exponential function.

[0060] The formula for weighted features of the prediction results is expressed as:

[0061]

[0062] in, To predict weighted features, For the predicted features of stage s,

[0063] Specifically, the schematic diagram of the spatial-frequency dual-path feedforward network in the feature enhancement block is as follows: Figure 4As shown, the spatial-frequency dual-path feedforward network performs convolutional upscaling and depthwise separable convolution on the weighted features of the prediction results to obtain local structural features. Nonlinear activation is then applied to the local structural features to obtain nonlinear activation results. The nonlinear activation results are multiplied point-by-point with the local structural features to achieve lightweight feature interaction and obtain lightweight interactive features. The lightweight interactive features are rearranged into multiple 4×4 sub-regions. The frequency distribution is analyzed by performing a fast Fourier transform on each sub-region, and the frequency domain components are dynamically weighted using learnable parameters to obtain dynamically weighted sub-features. Finally, the spatial feature structure is restored by inverse Fourier transform and dimension recombination to obtain the output features.

[0064] In the current embodiment, the structure of the optical flow guided fusion block is as follows: Figure 5 As shown, the optical flow guidance module takes a first intermediate feature and a second intermediate feature of the same scale as input, performs depth convolution and point convolution on the first intermediate feature to obtain a first motion intensity map, uses the relative coordinate difference between pixels within the first intermediate feature as a position bias, fuses the first motion intensity map with the position bias to obtain a second motion intensity map, constructs a gate weight based on the first intermediate feature, multiplies the second motion intensity map with the gate weight element-wise to obtain a comprehensive guidance weight, multiplies the second motion intensity map with the comprehensive guidance weight to obtain an intermediate scale feature, and performs a residual connection between the intermediate scale feature and the second intermediate feature to obtain an optical flow guidance fusion feature, which is the output of the optical flow guidance fusion block.

[0065] Specifically, when inputting the first intermediate feature and the second intermediate feature into the optical flow-guided fusion block, adaptive average pooling is used to downsample the first intermediate feature and the second intermediate feature simultaneously, thereby reducing the complexity of subsequent calculations and preserving the global motion trend. For example, this scheme downsamples the first intermediate feature and the second intermediate feature simultaneously to 14×14.

[0066] Specifically, the relative coordinate difference between pixels within the first intermediate feature is processed using an embedding layer to obtain the positional bias, and the first intermediate feature is processed using a convolutional layer to obtain the gating weights.

[0067] Specifically, this scheme processes the first intermediate features through depthwise convolution and pointwise convolution to generate a first motion intensity map to quantify the intensity of pixel motion, and strengthens the spatial context association of high-motion regions by fusing positional bias.

[0068] Specifically, when multiplying the intermediate-scale features with the second intermediate feature, a learnable parameter is added to control the enhancement intensity to obtain the optical flow-guided fusion feature. When outputting the optical flow-guided fusion feature, bilinear interpolation is used to restore the scale of the optical flow-guided fusion feature to the scale of the first and second intermediate features, thereby ensuring edge alignment in areas such as the eye area and corners of the mouth. For example, the learnable parameter (gamma) acts as a weighting coefficient, determining how much "optical flow-guided attention information" is injected into the "second intermediate feature." When gamma is large, the model significantly amplifies the enhancement effect of optical flow guidance, making the second intermediate feature (appearance) focus more on the motion region. When gamma is close to 0, the enhancement term is suppressed, and the output is almost equal to the original second intermediate feature, meaning that optical flow guidance is ignored. It is a scalar learnable parameter. During training, the model automatically adjusts the value of gamma according to the loss function to find the optimal balance between appearance features and motion guidance.

[0069] In the current embodiment, the structure of the cross-modal fusion module is as follows: Figure 6 As shown, the cross-modal fusion module includes a dual-path global attention layer, a channel self-attention layer, and a cross-modal fusion layer. The dual-path global attention layer includes an optical flow feature branch and an appearance feature branch. In the optical flow feature branch, attention is calculated using multi-scale optical flow features as queries and multi-scale spatiotemporal appearance features as values ​​and queries to obtain an optical flow-appearance attention feature map. In the appearance feature branch, attention is calculated using multi-scale spatiotemporal appearance features as queries and multi-scale optical flow features as keys and values ​​to obtain an appearance-optical flow attention feature map. The channel self-attention layer includes an optical flow channel attention unit and an appearance channel attention unit. In the optical flow channel attention unit, self-attention is calculated using the global average pooling result and the global max pooling result of the optical flow-appearance attention feature map to obtain an optical flow self-attention feature. In the appearance channel attention unit, self-attention is calculated using the global average pooling result and the global max pooling result of the appearance-optical flow attention feature map to obtain an appearance self-attention feature. In the cross-modal fusion layer, the optical flow self-attention feature and the appearance self-attention feature are fused to obtain a fused feature.

[0070] Specifically, the optical flow feature branch structure of the dual-path global attention layer is as follows: Figure 7As shown, in the dual-path global attention layer, the optical flow feature branch and appearance feature branch respectively perform bidirectional attention mechanism calculations for optical flow → appearance and appearance → optical flow to obtain appearance-optical flow attention feature maps and optical flow-appearance attention feature maps. When calculating the attention mechanism to obtain the optical flow-appearance attention feature map, the optical flow attention map is first constructed using the corresponding query and key, and then applied to the corresponding value to obtain the optical flow-appearance attention feature map. The formula for obtaining the optical flow attention map is expressed as:

[0071]

[0072] in, This is an optical flow attention map. For queries in the optical flow feature branch, For bonds in optical flow feature branches, This refers to the temperature parameter.

[0073] The appearance feature branch structure of the dual-path global attention layer is the same as the optical flow feature branch structure. When calculating the attention mechanism to obtain the appearance-optical flow attention feature map, the appearance attention map is first constructed using the corresponding query and key. Then, the appearance attention map is applied to the corresponding values ​​to obtain the appearance-optical flow attention feature map. The formula for obtaining the appearance attention map is expressed as:

[0074]

[0075] in, For appearance attention diagram, For queries in the appearance feature branch, For the key in the appearance feature branch, This refers to the temperature parameter.

[0076] Specifically, during attention calculation, the gating mechanism generates channel-level weights through a learnable network to suppress noise in the enhanced features, and then combines residual connections to strengthen the robustness of the features.

[0077] Specifically, the structures of the optical flow channel attention unit and the appearance channel attention unit are as follows: Figure 8 As shown, channel statistics are extracted from the appearance channel attention unit and the optical flow channel attention unit, and then fused by a lightweight convolutional network to generate channel attention weights, thereby dynamically strengthening the discriminative channels related to micro-expressions and suppressing redundant background channels.

[0078] Specifically, the cross-modal fusion layer uses 1×1 convolution, batch normalization, and ReLU activation to fuse optical flow self-attention features and appearance self-attention features.

[0079] In the current embodiment, the classification head is constructed based on a fully connected layer, and the emotion category recognition results output by the classification head in this scheme are three categories: positive, negative, and surprised.

[0080] For example, in the classification head, global average pooling is performed on the fused features to obtain a 1×1×512 feature vector, which is then output through a fully connected layer to generate the corresponding class prediction probability.

[0081] In the current embodiment, when training the micro-expression recognition model, the open-source CASMEII dataset and SAMM dataset are obtained, and the annotation files provided with the datasets are called to obtain the start frame index, peak frame index and end frame index to form training samples.

[0082] Specifically, the micro-expression recognition model uses a weighted cross-entropy loss function to handle class imbalance during the training phase. The training loss function expression is as follows:

[0083]

[0084] in, Let N be the loss function, i be the sample index, C be the number of classes, and c be the class index. For the true label of sample i, Let be the predicted probability of sample i. For category weights, .

[0085] In the validation phase, standard cross-entropy loss without weights is used to objectively reflect the model's performance on the original data distribution; the final output is the emotion category recognition result of micro-expressions.

[0086] To verify the effectiveness of the method of this invention, leave-one-out cross-validation (LOSO) was used to evaluate it on four benchmark datasets: SMIC, CASMEII, SAMM, and 3DB-composite. The experimental results are shown in Table 1.

[0087] Table 1 Results of the three-class classification experiment

[0088]

[0089] As shown in Table 1, on the 3DB-composite dataset, the unweighted F1 score (UF1) of our method reaches 91.93% and the unweighted average recall (UAR) reaches 92.95%, which are 3.93 percentage points and 4.34 percentage points higher than the previous best method, respectively. On the CASMEII dataset, UF1 and UAR reach 98.82% and 98.96%, respectively. On the SMIC dataset, UF1 and UAR are 86.45% and 87.03%, respectively. On the SAMM dataset, UF1 and UAR are 89.89% and 92.34%, respectively, all of which are better than the current mainstream methods.

[0090] Ablation experiments show that when only the dynamic motion branch is enabled, the UF1 of the CASMEII dataset is 96.64%. When only the spatiotemporal appearance branch (including SAFG) is enabled, the UF1 increases to 97.08%. When the dual branches are combined with the CMCRF module, the UF1 further increases to 98.82%. This confirms that the dynamic motion branch, spatiotemporal appearance branch, SAFG module, and CMCRF module all make key contributions to the recognition performance. The synergistic effect of each module is the core of improving the recognition accuracy, ensuring that the method can stably perform under different datasets and application scenarios.

[0091] Example 2

[0092] Based on the same concept, referencing Figure 9 This application also proposes a micro-expression recognition device based on a multi-scale optical flow-guided attention mechanism, comprising:

[0093] A construction module is used to build a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head.

[0094] The recognition module is used to input the micro-expression video to be recognized into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame, and end frame of the micro-expression video to be recognized. Based on the start frame and peak frame, it obtains three-channel optical flow dynamic features. The grayscale start frame, peak frame, and end frame are then masked for non-facial regions to obtain three grayscale appearance features, including the grayscale appearance features of the start frame, peak frame, and end frame. The dynamic motion branch uses multiple Transformer blocks of different scales connected in series to extract the facial muscle motion trajectory from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal appearance branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame, thus obtaining temporal difference features. Multiple concatenated feature enhancement blocks of different scales are then used to enhance these temporal difference features, resulting in multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, while the outputs of the remaining feature enhancement blocks are second intermediate features. An optical flow-guided fusion block uses attention weighting to fuse each second intermediate feature with a first intermediate feature of the corresponding scale, and then inputs this fusion into the next feature enhancement block. The cross-modal fusion module uses a dual-path global attention mechanism to fuse the multi-scale spatiotemporal appearance features and multi-scale optical flow features to obtain fused features. The classification head outputs the emotion category recognition result for micro-expressions based on the fused features.

[0095] Example 3

[0096] This embodiment also provides an electronic device, see reference. Figure 10 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0097] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0098] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0099] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0100] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the micro-expression recognition methods based on the multi-scale optical flow-guided attention mechanism in the above embodiments.

[0101] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0102] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0103] The input / output device 408 is used to input or output information. In this embodiment, the input information may be a video of micro-expressions to be identified, and the output information may be the emotion category recognition result, etc.

[0104] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0105] Construct a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head;

[0106] The micro-expression video to be identified is input into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame, and end frame of the video. Based on the start frame and peak frame, three-channel optical flow dynamic features are obtained. Non-facial regions are covered on the grayscaled start frame, peak frame, and end frame to obtain three grayscale appearance features, including the start frame grayscale appearance features, peak frame grayscale appearance features, and end frame grayscale appearance features. The dynamic motion branch uses multiple Transformer blocks of different scales connected in series to extract the facial muscle motion trajectory from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal extrapolation... The observation branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame, thus obtaining temporal difference features. Multiple feature enhancement blocks of different scales are then used to enhance these temporal difference features, resulting in multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, while the outputs of the remaining feature enhancement blocks are second intermediate features. An optical flow-guided fusion block uses attention weighting to fuse each second intermediate feature with a first intermediate feature of the corresponding scale, and then inputs this fusion into the next feature enhancement block. The cross-modal fusion module uses a dual-path global attention mechanism to fuse the multi-scale spatiotemporal appearance features and multi-scale optical flow features to obtain fused features. The classification head outputs the emotion category recognition result of the micro-expression based on the fused features.

[0107] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0108] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0109] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 10 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0110] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A micro-expression recognition method based on multi-scale optical flow guided attention mechanism, characterized in that, Includes the following steps: Construct a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head; The micro-expression video to be identified is input into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame and end frame of the micro-expression video to be identified. Based on the start frame and peak frame, the three-channel optical flow dynamic features are obtained. The non-facial area is covered on the grayscale start frame, peak frame and end frame to obtain three grayscale appearance features. The three grayscale appearance features include the grayscale appearance features of the start frame, the grayscale appearance features of the peak frame and the grayscale appearance features of the end frame. The dynamic motion branch uses multiple Transformer blocks of different scales concatenated to extract facial muscle motion trajectories from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal appearance branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame to obtain temporal difference features. Multiple feature enhancement blocks of different scales concatenated are used to enhance the temporal difference features to obtain multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, and the outputs of the remaining feature enhancement blocks are the second intermediate features. An optical flow-guided fusion block is used to fuse each second intermediate feature with the first intermediate feature of the corresponding scale using an attention-weighted method before inputting it into the next feature enhancement block. The cross-modal fusion module fuses the multi-scale spatiotemporal appearance features and multi-scale optical flow features based on a dual-path global attention mechanism to obtain fused features. The classification head outputs the emotion category recognition result of micro-expressions based on the fused features.

2. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, The starting frame is the image frame at the beginning of the micro-expression, the peak frame is the image frame at the peak of the micro-expression intensity, and the ending frame is the image frame at the end of the micro-expression. Based on the starting frame and the peak frame, the horizontal optical flow field, the vertical optical flow field, and the motion strain component are obtained. The horizontal optical flow field, the vertical optical flow field, and the motion strain component are stitched together to obtain the three-channel optical flow dynamic features. Key facial feature points are extracted from the grayscale start frame, peak frame, and end frame. The convex hull algorithm is used to generate facial masks for the start frame, peak frame, and end frame based on the key facial feature points. The area outside the facial mask is covered to obtain three grayscale appearance features.

3. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, The Transformer block in the dynamic motion branch uses the Manhattan self-attention mechanism to calculate the three-channel optical flow dynamic features to obtain the first intermediate feature. The first intermediate feature is input into the next Transformer block for calculation, and the output of the last Transformer block is the multi-scale optical flow feature.

4. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, In the temporal difference extraction module, pixel-level difference calculations are performed on the grayscale appearance features of the starting frame and the grayscale appearance features of the peak frame to obtain the first inter-frame difference. Pixel-level difference calculations are performed on the grayscale appearance features of the peak frame and the grayscale appearance features of the ending frame to obtain the second inter-frame difference. The original frame features are then obtained. The original frame features, the first inter-frame difference, and the second inter-frame difference are fused channel by channel to obtain the temporal difference features. The original frame features provide the corresponding global facial contour and static information of fixed wrinkles.

5. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, The feature enhancement block includes a multi-scale attention layer, an HSM-SSD layer, and a space-frequency dual-path feedforward network layer connected in sequence. The multi-scale attention layer divides the input features into multiple groups of sub-features based on the number of channels, and calculates the horizontal spatial attention and vertical spatial attention of each group of sub-features to obtain the directional attention weight features. The HSM-SSD layer processes the directional attention weight features to obtain the weighted features of the prediction result; the space-frequency dual-path feedforward network layer uses depthwise separable convolution to extract the local structural features of the weighted features of the prediction result, and then performs lightweight processing on the local structural features through nonlinear activation to obtain lightweight interactive features. The lightweight interactive features are divided into multiple sub-regions, and a fast Fourier transform is performed on each sub-region to obtain dynamic weighted sub-features. Then, all dynamic weighted sub-features are recombined through inverse Fourier transform to obtain the output result. The output result is a second intermediate feature or a multi-scale spatiotemporal appearance feature.

6. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, The optical flow guidance module takes a first intermediate feature and a second intermediate feature of the same scale as input. It performs depth convolution and point convolution on the first intermediate feature to obtain a first motion intensity map. The relative coordinate difference between pixels within the first intermediate feature is used as a position bias. The first motion intensity map is fused with the position bias to obtain a second motion intensity map. A gate weight is constructed based on the first intermediate feature. The second motion intensity map is multiplied element-wise with the gate weight to obtain a comprehensive guidance weight. The second motion intensity map is multiplied with the comprehensive guidance weight to obtain an intermediate scale feature. The intermediate scale feature is residually concatenated with the second intermediate feature to obtain an optical flow guidance fusion feature. The optical flow guidance fusion feature is the output of the optical flow guidance fusion block.

7. The micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism according to claim 1, characterized in that, The cross-modal fusion module includes a dual-path global attention layer, a channel self-attention layer, and a cross-modal fusion layer. The dual-path global attention layer includes an optical flow feature branch and an appearance feature branch. In the optical flow feature branch, attention is calculated using multi-scale optical flow features as queries and multi-scale spatiotemporal appearance features as values ​​and queries to obtain an optical flow-appearance attention feature map. In the appearance feature branch, attention is calculated using multi-scale spatiotemporal appearance features as queries and multi-scale optical flow features as keys and values ​​to obtain an appearance-optical flow attention feature map. The channel self-attention layer includes an optical flow channel attention unit and an appearance channel attention unit. In the optical flow channel attention unit, self-attention is calculated using the global average pooling result and the global max pooling result of the optical flow-appearance attention feature map to obtain an optical flow self-attention feature. In the appearance channel attention unit, self-attention is calculated using the global average pooling result and the global max pooling result of the appearance-optical flow attention feature map to obtain an appearance self-attention feature. In the cross-modal fusion layer, the optical flow self-attention feature and the appearance self-attention feature are fused to obtain a fused feature.

8. A micro-expression recognition device based on a multi-scale optical flow-guided attention mechanism, characterized in that, include: A construction module is used to build a pre-trained micro-expression recognition model, which includes a preprocessing module, a parallel dynamic motion branch and a spatiotemporal appearance branch, a cross-modal fusion module, and a classification head. The recognition module is used to acquire the micro-expression video to be recognized and input it into the micro-expression recognition model. The preprocessing module extracts the start frame, peak frame and end frame of the micro-expression video to be recognized, obtains three-channel optical flow dynamic features based on the start frame and peak frame, and performs non-facial area occlusion on the grayscale start frame, peak frame and end frame to obtain three grayscale appearance features. The three grayscale appearance features include the grayscale appearance features of the start frame, the grayscale appearance features of the peak frame and the grayscale appearance features of the end frame. The dynamic motion branch uses multiple Transformer blocks of different scales concatenated to extract facial muscle motion trajectories from the three-channel optical flow dynamic features to obtain multi-scale optical flow features. The output of the last Transformer block is the multi-scale optical flow feature, and the outputs of the remaining Transformer blocks are the first intermediate features. The spatiotemporal appearance branch uses a temporal difference extraction block to obtain inter-frame differences in the grayscale appearance features of the starting frame, the peak frame, and the ending frame to obtain temporal difference features. Multiple feature enhancement blocks of different scales concatenated are used to enhance the temporal difference features to obtain multi-scale spatiotemporal appearance features. The output of the last feature enhancement block is the multi-scale spatiotemporal appearance feature, and the outputs of the remaining feature enhancement blocks are the second intermediate features. An optical flow-guided fusion block is used to fuse each second intermediate feature with the first intermediate feature of the corresponding scale using an attention-weighted method before inputting it into the next feature enhancement block. The cross-modal fusion module fuses the multi-scale spatiotemporal appearance features and multi-scale optical flow features based on a dual-path global attention mechanism to obtain fused features. The classification head outputs the emotion category recognition result of micro-expressions based on the fused features.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements a micro-expression recognition method based on a multi-scale optical flow-guided attention mechanism as described in any one of claims 1-7.

Images