A micro-expression recognition method based on gradient isolation leakage integral attention

Abstract

This invention relates to a micro-expression recognition method based on gradient-isolated leakage integral attention, comprising: A) preprocessing micro-expression videos; B) selecting peak frames and start frames from the preprocessed video sequence in step A, and performing pixel difference calculations on them to obtain facial keyframe differences; C) constructing a facial motion pattern feature extraction main network; D) constructing a gradient-isolated leakage integral attention module to enhance the ability to capture subtle changes in facial muscle activity; E) model training; F) inputting the preprocessed micro-expression video to be recognized into the trained facial motion pattern feature extraction main network and gradient-isolated leakage integral attention module to achieve micro-expression classification and recognition. This invention utilizes the difference information between peak frames and start frames of micro-expression video samples to centrally process representative features generated by subtle facial movements.

Description

Technical Field

[0001] This invention relates to a micro-expression recognition method based on gradient isolation leakage integral attention, which focuses on the extraction of facial motion patterns and belongs to the field of deep learning and pattern recognition technology. Background Technology

[0002] Microexpressions, as subtle, fleeting, and spontaneous emotional representations of human facial expressions, have been a significant topic in psychology and human-computer interaction research since their initial discovery by Professor Paul Ekman in the 1960s. These brief facial movements, lasting no more than half a second, can reveal the most authentic and uncontrollable aspects of human emotion that are being concealed. Therefore, accurate identification of microexpressions has extremely important applications in security interrogations, clinical diagnosis, business negotiations, and everyday social interactions.

[0003] However, the spontaneity, transience, and subtlety of micro-expressions make their recognition extremely challenging (they typically last only 1 / 25 to 1 / 3 of a second), and even trained experts struggle to accurately identify them. The best recognition rate achieved by professionally trained experts in tests was only 47%, highlighting the significant difficulty of manually identifying micro-expressions.

[0004] In recent years, with the rapid development of computer vision and multimedia technologies, automatic micro-expression recognition using computers has become possible and a research hotspot, leading to the emergence of various micro-expression recognition algorithms. Early micro-expression recognition algorithms largely relied on traditional machine learning models that manually designed micro-expression features. These models utilized existing human knowledge of micro-expressions to design potentially more representative features. However, recognition methods based on manually designed features struggle to capture the complete and comprehensive changes in micro-expressions; they often focus on specific types of data distributions (such as facial texture, lighting variations, etc.). This incomplete representation makes it difficult for the model to achieve the expected results. Subsequently, with the success of deep learning-based methods in feature extraction, more researchers began exploring the possibilities of applying deep learning models to micro-expression recognition tasks. While deep learning-based methods have made progress in feature extraction, effectively improving recognition accuracy and reducing false positives remains a key problem that current research needs to address.

[0005] Early on, Jianzheng Liu et al. proposed a novel facial expression recognition method based on motion templates and a 4-layer deep learning neural network. They used motion templates and AdaBoost to identify action units as features, then input the extracted features into the 4-layer deep learning neural network to recognize facial expressions. At this point, the use of deep learning models for micro-expression recognition began to enter the research field. Later, Peng et al. proposed a dual-stream time-scale convolutional neural network (DTSCNN), using independent shallow convolutional networks to adapt to micro-expression video samples at different frame rates for different streams. However, because it used a dual-stream network to calculate the optical flow between adjacent frames, it inevitably introduced a large amount of computation. Liong et al. further proposed a shallow three-stream three-dimensional CNN (STSTNet), combining the optical flow of the start frame and peak frame as input to the model. Verma et al. proposed a lateral-accretive hybrid network (LEARNet) based on dynamic imaging, compressing a complete video sequence into a single frame as input to preserve complete muscle movement information. In recent years, methods for micro-expression recognition using deep learning models have increasingly incorporated well-established algorithms validated in other fields. In 2021, Rahil Kadakia et al. explored the performance of different deep learning models (such as VGG16, ResNet50, MobileNet, InceptionV3, and Xception) on the SAMM dataset for micro-expression recognition, finding that MobileNet outperformed other models in efficiency. This work reduced the computational cost of combining traditional handcrafted features with deep learning methods. In December of the same year, Zhao et al. summarized research on micro-expression recognition based on deep learning methods, discussing the development issues and trends of mainstream algorithms such as DBN and CNN, and pointing out the application prospects of deep learning in the field of micro-expression recognition. Subsequently, in 2022, Zhao et al. proposed a deep prototype learning framework, ME-PLAN, which addresses key challenges in micro-expression recognition through a local attention mechanism, demonstrating its superiority and effectiveness on composite databases. Lei et al. proposed a graph-temporal convolutional network based on graph structure feature representation, utilizing attention modules of different scales to determine the motion intensity in different facial regions to identify subtle facial expression changes. In the same year, Xie et al. also adopted a deep mutual learning network, combining AU with emotion categories, and obtained relevant feature representations of micro-expression input through the difference images of key frames, which improved the recognition accuracy and speed.Recently, Fan et al. proposed the Self-Supervised Motion Learning for Micro-Expression (SelfME) framework in 2023, which primarily captures the changing patterns of human micro-expression facial movements, attempting to overcome the limitations of models' dependence on handcrafted features. Nguyen et al. proposed the BERT-based Facial Micro-Expression Recognition (Micron-BERT) framework, leveraging the sequence modeling capabilities of the BERT framework to locate patches of interest (POIs) in micro-expression frames, thereby reducing the impact of background noise on model recognition performance.

[0006] Currently, deep learning-based recognition algorithms often rely on the powerful feature modeling capabilities of neural networks, neglecting the subtle and transient nature of micro-expression features. This means that the performance of the model is often determined by the sophistication of the deep model used, making it difficult to further improve the recognition rate. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a micro-expression recognition method based on gradient-isolated leakage integral attention for facial motion pattern extraction. This invention emphasizes the effective integration of deep learning frameworks with subtle facial motion features, rather than simply relying on the modeling capabilities of neural networks for abstract features. Invention Overview:

[0009] A micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction includes a dataset preprocessing module, a facial motion pattern feature extraction module, a gradient isolation leakage integral attention module, and a classification and recognition module.

[0010] The technical problem addressed by this invention is that existing methods struggle to model the subtle dynamic features of micro-expressions, resulting in low recognition accuracy. Compared to existing methods, this invention utilizes neural networks to model the difference between the peak frame and the starting frame of a micro-expression, effectively capturing minute changes in facial muscles. Furthermore, this invention proposes gradient-isolated leakage integral attention, which enhances the model's ability to model subtle facial changes by strengthening the subtle movement patterns exhibited by specific facial regions over a given duration.

[0011] Terminology Explanation:

[0012] 1. Dlib Vision Library: Dlib is an open-source C++ toolkit containing machine learning algorithms. Dlib can help create many complex machine learning software applications to help solve practical problems. Currently, Dlib is widely used in industry and academia, including robotics, embedded devices, mobile phones, and large-scale high-performance computing environments.

[0013] 2. 68 key facial feature points: These 68 key facial feature points are mainly distributed in the eyebrows, eyes, nose, mouth, and facial contours, such as... Figure 1 As shown, detection is performed using the Dlib vision library, which is an existing technology.

[0014] 3. Loss Function: The loss function is used to evaluate the degree of inconsistency between the model's predicted values ​​and the true values. The smaller the loss function, the better the robustness of the model. The loss function can guide the model's learning.

[0015] 4. Convolutional Residual Network (ResNet): This is a deep learning model, originally proposed by Kaiming He et al. in 2015. Its various variants have been widely used in video classification and behavior recognition.

[0016] 5. Spiking Neuron: The spiking neuron is a neuron model designed to more closely resemble the behavior of biological neurons. Compared to the neuron models used in traditional artificial neural networks, the spiking neuron can process and generate time-discrete signals.

[0017] 6. Leakage Integral: The leakage integral mechanism is a spiking neuron model used to simulate the electrical activity of neurons. It can establish independent time domains and encode information.

[0018] The technical solution of the present invention is as follows:

[0019] A micro-expression recognition method based on gradient-isolated leakage integral attention for facial motion pattern extraction includes:

[0020] A. Preprocessing of micro-expression videos, including: acquiring video frame sequences, face detection and localization, and face alignment;

[0021] B. Select the peak frame and the starting frame from the video sequence after preprocessing in step A, and perform pixel difference calculation on them to obtain the facial key frame difference;

[0022] C. Construct a main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction utilizes the facial keyframe differences obtained in step B and uses ResNet to further learn and model the micro-expression information of the facial keyframe differences.

[0023] D. Construct a gradient-isolated leakage current integral attention module to enhance the ability to capture subtle changes in facial muscle activity;

[0024] E. Model training: Iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integral attention module constructed in steps C and D using cross-entropy loss.

[0025] F. After preprocessing the micro-expression video to be identified, input it into the trained facial motion pattern feature extraction main network and gradient isolation leakage integral attention module to realize micro-expression classification and recognition.

[0026] According to a preferred embodiment of the present invention, in step A, the micro-expression video is preprocessed, which includes:

[0027] 1) Obtain video frames: Perform frame segmentation on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames, and save them in chronological order;

[0028] 2) Face detection and localization: The Dlib vision library is used to detect and localize faces in the separated micro-expression video frames, and the distance between the face and the image boundary in the detected micro-expression video frames is obtained.

[0029] 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib vision library is used to determine 68 key feature points of the face in the micro-expression video frame, and then rectangular boxes are used for segmentation.

[0030] According to a preferred embodiment of the present invention, in step A, face correction is performed after face alignment, including:

[0031] Based on the characteristics of the selected micro-expression dataset, adjustments were made to some of the 68 key feature points detected.

[0032] According to a preferred embodiment of the present invention, in step B, facial keyframe differences Defined as peak frame X σ The pixel difference between the starting frame X1 and the starting frame X1 is shown in Equation (1):

[0033]

[0034] According to a preferred embodiment of the present invention, in step C, the facial motion pattern feature extraction main network utilizes the facial keyframe differences obtained in step B, and uses ResNet to further learn and model the micro-expression information based on the facial keyframe differences; including:

[0035] The main network for facial motion pattern feature extraction consists of convolutional layers, pulse residual blocks, and a feature classifier; the pulse residual block consists of two independent convolutional layers.

[0036] The convolutional layer extracts preliminary spatial features; the output features of the convolutional layer are further processed by the pulsed residual block; the pulsed residual block further enhances the features, and its output is expressed as Equation (2) and Equation (3):

[0037]

[0038] Among them, W i 1 W i 2 and b i 2 F represents the weights and biases of the two convolutional layers in the i-th pulse residual block, respectively; i The feature output by the convolution of the i-th pulse residual block;

[0039] ReLU(·) represents activating features using a non-linear activation function; BN(·) represents normalizing features using batch normalization.

[0040] After l pulse residual blocks, the final feature map F I Used for the feature classifier, which predicts the sentiment state through a fully connected layer, as shown in Equation (4):

[0041]

[0042] Among them, F l Here, c represents the feature output of the last pulse residual block, and c is the micro-expression category to be predicted. and The weights and biases of the fully connected layer are denoted by Softmax, which is the probability output function to ensure that the sum of the output probability distributions is 1, thereby classifying between the three states of Positive, Negative, and Surprise.

[0043] According to a preferred embodiment of the present invention, a gradient-isolated leakage current integration attention module is constructed to enhance the ability to capture subtle changes in facial muscle activity; the gradient-isolated leakage current integration attention module includes leakage current integration attention and gradient isolation.

[0044] According to a preferred embodiment of the present invention, the spiking neuron calculates the leakage integral attention based on the motion characteristics of the input, including:

[0045] The membrane potential state of each spiking neuron is updated using equations (5) and (6):

[0046]

[0047] V i (0)=V i-1(T) (6)

[0048] Among them, V i Let w be the membrane potential of the i-th spiking neuron at time t, where τ is the leakage coefficient, which determines the influence of the potential from the previous time step on the current time step. ij Is input x j (t) represents the synaptic weight of spiking neuron i, where δ is a binary variable indicating whether a pulse occurred at the previous time step t-1. Whenever the potential exceeds the threshold, the spiking neuron fires a pulse and transmits the excess potential portion to the potential at the next time step as its initial value. The spiking neuron retains the membrane potential generated at the last time step of the previous layer of attention and uses it as the initial potential value V for the current layer of leakage integral attention. i (0).

[0049] According to a preferred embodiment of the present invention, the calculation of leakage current integral attention is specifically implemented as follows:

[0050] First, a gradient-isolated leakage integral attention module is constructed using the feature map X as input; the parameters of the gradient-isolated leakage integral attention module include weights W. att Bias b att And the potential leakage parameter τ;

[0051] Secondly, initialize the relevant parameters, including the membrane potential Mem[t] and the initial attention state Att. i Weight W att and bias b att ;

[0052] Then, the forward propagation process begins, and the state of the leakage integral attention neuron is updated by setting a time step T. The process is described by equation (7):

[0053] Mem[t]=τMem[t-1]+f(X,W att ,b att (7)

[0054] Where f(i) is the linear computation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att... i A value of 1 indicates the active state; otherwise, a value of 0 indicates the inactive state.

[0055] When Mem[t] is in an active state, it is reset back to its initial state.

[0056] Finally, the backpropagation process for gradient isolation is performed according to equation (8):

[0057] grad=grad / (ctx.k*|Mem[T]+1.0| 2 (8)

[0058] Where grad is the gradient of the leakage integral attention neuron, and ctx.k is the attention state Att during the forward process. i (i.e., 0 or 1).

[0059] According to a preferred embodiment of the present invention, gradient isolation is implemented as follows:

[0060] First, spiking neurons Responsible for generating the corresponding feature layer (i.e. Attention value Att i spiking neurons The forward generation process means that after the time step cycle of the spiking neuron ends, the spiking signal will be transmitted to the next neuron in the same layer; at the same time, the facial motion pattern feature extraction main network will transmit the extracted features to the next layer.

[0061] Secondly, the remaining potential Mem in the current layer continues to propagate to the next layer, affecting the spiking neurons in that layer. Perform initialization;

[0062] Finally, the spiking neurons are independently updated and optimized using the second cross-entropy loss, while the parameters of the spiking neurons are updated using the first cross-entropy loss.

[0063] Further optimized formulas for the first cross-entropy loss and the second cross-entropy loss are shown in equation (9):

[0064]

[0065] Where N is the total number of samples to be tested, and M is the number of categories to be classified; y ic Let y be a sign function (0 or 1). If the true class of sample i is m, then y ic If y is 1, otherwise y ic p is 0; ic Let be the probability that observed sample i belongs to category m.

[0066] According to a preferred embodiment of the present invention, the gradient optimization of the main network for isolated gradient leakage integral attention and facial motion pattern feature extraction is described by equations (10)-(12):

[0067]

[0068] Among them, △W att ΔW represents the updated weights of the spiking neurons, and ΔW represents the updated weights of the main network convolutional units. Let be the pulse value generated by the spiking neuron of the i-th layer pulse residual block at time t, which is determined by... And δ is calculated to obtain, Here, δ represents the corresponding input potential value, η represents the activation threshold, and η represents the learning rate. For loss function, Micro-expression features extracted by the main network for facial motion pattern feature extraction.

[0069] According to a preferred embodiment of the present invention, after preprocessing the micro-expression video to be identified, it is input into a trained facial motion pattern feature extraction main network and a gradient isolation leakage integral attention module to achieve micro-expression classification and recognition; including:

[0070] First, the micro-expression video to be identified is preprocessed and labeled as peak frame and start frame, and then motion features are obtained after pixel difference calculation.

[0071] Next, the motion features are first reduced in dimensionality using a 3x3 convolutional kernel. The dimensionality-reduced motion features are then sequentially fed into a spiking residual block and connected using residual connections. The spiking residual block consists of two convolutional layers and a batch normalization operation after each convolutional layer. The input motion features, after processing by the spiking residual block, are temporarily stored as intermediate features. Simultaneously, each spiking residual block is paired with a spiking neuron module. The spiking neuron module contains a single spiking neuron, which calculates the leakage integral attention based on the input motion features.

[0072] Finally, the obtained leakage current integral attention is multiplied and weighted with the obtained intermediate features;

[0073] The above process continues until the last pulse residual block. The output features of the last pulse residual block are fed into the feature classifier for micro-expression recognition and classification.

[0074] The beneficial effects of this invention are as follows:

[0075] 1. To enhance the network's ability to model micro-expression features, this invention proposes a deep model focused on extracting dynamic facial features. It utilizes the difference between peak frames and starting frames in micro-expression video samples to centrally process representative features generated by subtle facial movements.

[0076] 2. To capture subtle changes in facial muscle activity, this invention proposes a gradient-isolated leakage integral attention mechanism. By emphasizing the subtle movement patterns exhibited by specific facial regions during the duration of micro-expressions, the model learns features within specific facial regions more effectively. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of 68 key facial feature points of the present invention;

[0078] Figure 2 The flowchart shows a micro-expression recognition method based on gradient isolation leakage integral attention.

[0079] Figure 3 A schematic diagram of the facial motion pattern extraction network structure based on gradient-isolated leakage integral attention;

[0080] Figure 4 A diagram illustrating the visualization effect of traditional channel attention weights;

[0081] Figure 5 A schematic diagram illustrating the visualization effect of traditional spatial attention weights;

[0082] Figure 6 A diagram illustrating the visualization of multi-head attention weights;

[0083] Figure 7 This is a schematic diagram illustrating the visualization effect of the gradient isolation leakage current integral attention weight in this invention;

[0084] Figure 8 This is a schematic diagram illustrating the implementation of attention gradient isolation. Detailed Implementation

[0085] The present invention will be further described below with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0086] Example 1

[0087] A micro-expression recognition method based on gradient-isolated leakage integral attention for facial motion pattern extraction, such as... Figure 2 As shown, it includes:

[0088] A. Preprocessing of micro-expression videos, including: acquiring video frame sequences, face detection and localization, and face alignment;

[0089] B. Select the peak frame and the starting frame from the video sequence after preprocessing in step A, and perform pixel difference calculation on them to obtain the facial key frame difference;

[0090] C. Construct a main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction utilizes the facial keyframe differences obtained in step B and uses ResNet to further learn and model the micro-expression information of the facial keyframe differences.

[0091] D. Construct a gradient-isolated leakage current integral attention module to enhance the ability to capture subtle changes in facial muscle activity;

[0092] E. Model training: Iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integral attention module constructed in steps C and D using cross-entropy loss.

[0093] F. After preprocessing the micro-expression video to be identified, input it into the trained facial motion pattern feature extraction main network and gradient isolation leakage integral attention module to realize micro-expression classification and recognition.

[0094] Example 2

[0095] The micro-expression recognition method based on gradient isolation leakage integral attention facial motion pattern extraction described in Example 1 differs in that:

[0096] In step A, the micro-expression video is preprocessed, which includes:

[0097] 1) Obtain video frames: Perform frame segmentation on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames, and save them in chronological order as I(0,1,2,…,I).png format;

[0098] 2) Face detection and localization: The Dlib vision library is used to detect and localize faces in the separated micro-expression video frames, and the distance between the face and the image boundary in the detected micro-expression video frames is obtained.

[0099] 3) Face Alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib vision library is used to determine 68 key feature points of the face in the micro-expression video frame, and then rectangular boxes are used for segmentation. The size of the rectangular boxes is set to 256*256.

[0100] In step A, after face alignment, face correction is performed, including:

[0101] Based on the characteristics of the selected micro-expression dataset (including ethnicity, gender, shooting angle, etc.), adjustments are made to certain key feature points among the 68 detected key feature points. For example, a line can be drawn connecting key feature point 37 (marked at the left corner of the left eye) and key feature point 46 (marked at the right corner of the right eye). This line may form an angle α with the standard horizontal line. This angle can be used to rotate the segmented face, making the line connecting key feature points 37 and 46 parallel to the standard horizontal line, thus calibrating the face pose.

[0102] In step B, facial keyframe differences Defined as peak frame X σ The pixel difference between the starting frame X1 and the starting frame X1 is shown in Equation (1):

[0103]

[0104] In step C, the facial motion pattern feature extraction main network utilizes the facial keyframe differences obtained in step B, and uses ResNet to further learn and model micro-expression information from these keyframe differences; including:

[0105] The main network for facial motion pattern feature extraction consists of convolutional layers, pulse residual blocks, and a feature classifier; the pulse residual block consists of two independent convolutional layers.

[0106] The convolutional layer extracts preliminary spatial features; the output features of the convolutional layer are further processed by the pulsed residual block; the pulsed residual block further enhances the features, and its output is expressed as Equation (2) and Equation (3):

[0107]

[0108] Among them, W i 1 W i 2 and b i 2 These represent the weights and biases of the two convolutional layers in the i-th pulse residual block, respectively.

[0109] F i For the features output by the convolution of the i-th pulse residual block, Relu(·) represents the activation of the features using a non-linear activation function; BN(·) represents the normalization of the features using batch normalization.

[0110] After l pulse residual blocks, the final feature map F I Used for the feature classifier, which predicts the sentiment state through a fully connected layer, as shown in Equation (4):

[0111]

[0112] Among them, F l Here, c represents the feature output of the last pulse residual block, and c is the micro-expression category to be predicted. and The weights and biases of the fully connected layer are denoted by Softmax, which is the probability output function to ensure that the sum of the output probability distributions is 1, thereby classifying between the three states of Positive, Negative, and Surprise.

[0113] A gradient-isolated leakage integral attention module is constructed to enhance the ability to capture subtle changes in facial muscle activity. This module includes leakage integral attention and gradient isolation. Leakage integral attention: This mechanism is incorporated into the proposed main recognition network to improve the neural network's sensitivity to subtle facial muscle movements. The leakage integral attention mechanism combines the temporal dynamics of spiking neural networks with the representational capabilities of deep learning.

[0114] Spiking neurons calculate leakage integral attention based on the motion characteristics of the input, including:

[0115] The membrane potential state of each spiking neuron is updated using equations (5) and (6):

[0116]

[0117] V i (0)=V i-1 (T) (6)

[0118] Among them, V i Let w be the membrane potential of the i-th spiking neuron at time t, where τ is the leakage coefficient, which determines the influence of the potential from the previous time step on the current time step. ij Is input x j (t) represents the synaptic weight of spiking neuron i, where δ is a binary variable indicating whether a pulse occurred at the previous time step t-1. Whenever the potential exceeds the threshold, the spiking neuron fires a pulse and transmits the excess potential portion to the potential at the next time step as its initial value. Notably, the proposed spiking neuron retains the membrane potential generated at the last time step of the previous layer of attention and uses it as the initial potential value V for the current layer of leakage integral attention. i (0). Since the leakage integral provides an insufficient discrete representation of information, this approach helps to preserve shallow features and saves the modeling computation process of learned information per unit time step in the deep leakage integral attention unit.

[0119] The specific implementation process for calculating the leakage current integral attention is as follows:

[0120] First, a gradient-isolated leakage integral attention module is constructed using the feature map X as input; the parameters of the gradient-isolated leakage integral attention module include weights W. att Bias b att And the potential leakage parameter τ;

[0121] Secondly, initialize the relevant parameters, including the membrane potential Mem[t] and the initial attention state Att. i Weight W att and bias b att ;

[0122] Then, the forward propagation process begins, and the state of the leakage integral attention neuron is updated by setting a time step T. The process is described by equation (7):

[0123] Mem[t]=τMem[t-1]+f(X,W att ,b att (7)

[0124] Where f(i) is the linear computation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att... i A value of 1 indicates the active state; otherwise, a value of 0 indicates the inactive state.

[0125] When Mem[t] is in an active state, it is reset back to its initial state.

[0126] Finally, the backpropagation process for gradient isolation is performed according to equation (8):

[0127] grad=grad / (ctx.k*|Mem[T]+1.0| 2 (8)

[0128] Where grad is the gradient of the leakage integral attention neuron, and ctx.k is the attention state Att during the forward process. i (i.e., 0 or 1).

[0129] Since the leakage current integral attention output is a binary signal, the network is difficult to perform normal gradient descent to optimize the network weights. Based on this, gradient isolation is performed on the proposed attention unit. Figure 8 This is a schematic diagram illustrating gradient isolation for attention units. The specific implementation process of gradient isolation is as follows:

[0130] First, spiking neurons Responsible for generating the corresponding feature layer (i.e. Attention value Att i spiking neurons The forward generation process means that after the time step cycle of a spiking neuron ends, the pulse signal will be transmitted to the next neuron in the same layer (e.g., ...). Figure 8 (As shown by the dark solid line in the vertical direction); simultaneously, the facial motion pattern feature extraction main network passes the extracted features to the next layer (such as...). Figure 8 (As shown by the dark solid line in the horizontal direction);

[0131] Secondly, the remaining potential Mem in the current layer continues to be propagated to the next layer (e.g., Figure 8 (As shown by the light-colored solid line in the horizontal direction), for the spiking neurons of the next layer. Perform initialization;

[0132] Finally, the spiking neurons undergo independent weight updates and optimizations using the second cross-entropy loss (cross-entropy loss 2). Figure 8 (As shown by the dark dashed line in the vertical direction), in order to update the weights of the backbone classification network when the gradient of the spiking neuron is not directly fed back to the backbone network, a first cross-entropy loss (cross-entropy loss 1) is established to update the parameters of the spiking neuron (e.g., ...). Figure 8 (As shown by the dark dashed line in the middle horizontal direction).

[0133] The optimization formulas for the first cross-entropy loss and the second cross-entropy loss are shown in Equation (9):

[0134]

[0135] Where N is the total number of samples to be tested, and M is the number of categories to be classified; y ic Let y be a sign function (0 or 1). If the true class of sample i is m, then y ic If y is 1, otherwise y ic p is 0; ic Let be the probability that observed sample i belongs to category m.

[0136] The difference between cross-entropy loss 1 and cross-entropy loss 2 lies in the different weights they optimize. Cross-entropy loss 1 optimizes the weights ΔW of the main network convolutional units, while cross-entropy loss 2 optimizes the weights ΔW of the spiking neurons. att As shown in formulas (8) and (7).

[0137] Leakage integral attention of isolated gradient and facial motion pattern feature extraction main network gradient optimization are described by equations (10)-(12):

[0138]

[0139] Among them, △W att ΔW represents the updated weights of the spiking neurons, and ΔW represents the updated weights of the main network convolutional units. Let be the pulse value generated by the spiking neuron of the i-th layer pulse residual block at time t, which is determined by... And δ is calculated to obtain, Here, δ represents the corresponding input potential value, η represents the activation threshold, and η represents the learning rate. For loss function, Micro-expression features extracted by the main network for facial motion pattern feature extraction.

[0140] The gradient-isolated leakage integral attention mechanism not only preserves the temporal dynamic characteristics of spiking neurons, providing a natural way to capture subtle micro-expression information, but also ensures the training stability of the network and avoids interference from the non-traditional learning dynamics of spiking neurons on the optimization of the main network weights.

[0141] After preprocessing the micro-expression video to be identified, the input is a pre-trained facial motion pattern feature extraction main network and a gradient isolation leakage integral attention module to achieve micro-expression classification and recognition; including:

[0142] First, the micro-expression video to be identified is preprocessed and labeled as peak frame and start frame, and then motion features are obtained after pixel difference calculation.

[0143] Then, the motion features are first processed by a 3x3 convolution kernel. Figure 3 The dimensionality reduction is performed on Conv, and then the dimensionality-reduced motion features are fed into the pulse residual block in the order of the solid arrows. Figure 3 SRB is used in the middle, and residual linking is used. Figure 3 The middle part is a light-colored broken line); the pulse residual block includes two convolutional layers ( Figure 3 Conv1 and Conv2 in the middle, and batch normalization operation after each convolutional layer ( Figure 3 (BN1 and BN2); The input motion features are temporarily saved as intermediate features after processing by the pulse residual block, such as... Figure 3 As shown in the light-colored cuboid. Simultaneously, each pulse residual block is paired with a spiking neuron module ( Figure 3 The middle part is SNB). The spiking neuron module includes a spiking neuron ( Figure 3 The spiking neuron (SN) calculates the leakage integral attention based on the motion characteristics of the input. Figure 3 The Chinese character is represented as Att. i (T)), the calculation process is shown in equations (5) and (6) below;

[0144] Finally, the obtained leakage current integral attention is multiplied and weighted with the obtained intermediate features;

[0145] The above process continues until the last pulse residual block. The output features of the last pulse residual block are fed into the feature classifier for micro-expression recognition and classification.

[0146] In this embodiment, micro-expression recognition tests were conducted on original videos from the CASME II micro-expression database released by Fu Xiaolan's team at the Institute of Psychology, Chinese Academy of Sciences, the SAMM database released by Davison's team at Manchester Metropolitan University in the UK, and the SMIC database released by a research team at the University of Oulu in Finland. The test results of this invention were also compared with existing methods based on traditional learning and deep learning; Table 1 shows the test results.

[0147] Table 1

[0148]

[0149] In Table 1, Pre-P: preprocessing method; E: Euler image magnification; R: region of interest; T: temporal normalization; Pre-T: pre-training dataset.

[0150] In Table 1, the LBP-TOP method is from the literature Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29(6): 915-928.

[0151] FDM is from the literature Xu F, Zhang J, Wang J Z. Micro-expression identification and categorization using a facial dynamics map[J]. IEEE Transactions on Affective Computing, 2017, 8(2): 254-267.

[0152] MDMO is from the literature Liu Y J, Zhang J K, Yan W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299-310.

[0153] TSCNN is from the literature Song B, Li K, Zong Y, et al. Recognizing spontaneous micro-expression using a three-stream convolutional neural network[J]. IEEE Access, 2019, 7: 184537-184551.

[0154] STSTNet is from the literature "Liong S T, Gan Y S, See J, et al. Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition [C] / / 2019 14th IEEE international conference on automatic face&gesture recognition (FG 2019). IEEE, 2019:1-5."

[0155] DIKD is from the literature "Sun B, Cao S, Li D, et al. Dynamic micro-expression recognition using knowledge distillation [J]. IEEE Transactions on Affective Computing, 2020, 13(2):1037-1043."

[0156] DSTAN is from the literature "Wang Y, Huang Y, Liu C, et al. Micro expression recognition via dual-stream spatiotemporal attention network [J]. Journal of Healthcare Engineering, 2021, 2021."

[0157] GEME is from the literature "Nie X, Takalkar M A, Duan M, et al. GEME: Dual-stream multi-task GEnder-based micro-expression recognition [J]. Neurocomputing, 2021, 427:13-28."

[0158] ME-PLAN is from the literature "Zhao S, Tang H, Liu S, et al. ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition [J]. Neural Networks, 2022, 153:427-443."

[0159] BDCNN comes from the document Chen B, Liu KH, Xu Y, et al. Block division convolutional network with implicit deep features augmentation for micro-expressionrecognition[J]. IEEE Transactions on Multimedia, 2022, 25: 1345-1358.

[0160] As shown in Table 1, the method proposed in this invention outperforms other methods in all recognition rates (accuracy ACC, mean F1 score UF1, and mean recall UAR) on the CASMEII, SAMM, and SMIC datasets. The accuracy of this invention on the three datasets reached 92.9%, 78.9%, and 75.2%, respectively; the mean F1 score reached 90.5%, 68.7%, and 75.0%, respectively; and the mean recall reached 90.5%, 73.9%, and 77.9%, respectively. Figure 4 A diagram illustrating the visualization effect of traditional channel attention weights; Figure 5 A schematic diagram illustrating the visualization effect of traditional spatial attention weights; Figure 6 A diagram illustrating the visualization of multi-head attention weights; Figure 7 This is a visualization of the gradient isolation leakage integral attention weights of the present invention. This is because the algorithm of the present invention, compared to other algorithms, has more concentrated processing weights on information at specific locations in the facial feature map. It can lock onto the areas with the most significant changes in facial expressions, such as around the eyes and mouth, and thus identify subtle changes in facial muscles, rather than uniformly learning global information from the input features. This effectively avoids the problem of the network focusing excessively on global changes, causing the algorithm to ignore subtle facial expressions. Furthermore, by proposing the use of pixel differences between micro-expression video frames, the present invention further enhances its ability to identify key micro-expression features and significantly improves the overall model's sensitivity to subtle emotional changes.

Claims

1. A micro-expression recognition method based on gradient-isolated leakage integral attention for facial motion pattern extraction, characterized in that, include: A. Preprocessing of micro-expression videos, including: acquiring video frame sequences, face detection and localization, and face alignment; B. Select the peak frame and the starting frame from the video sequence after preprocessing in step A, and perform pixel difference calculation on them to obtain the facial key frame difference; C. Construct a main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction utilizes the facial keyframe differences obtained in step B and uses ResNet to further learn and model the micro-expression information of the facial keyframe differences. D. Construct a gradient-isolated leakage current integral attention module to enhance the ability to capture subtle changes in facial muscle activity; E. Model training: Iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integral attention module constructed in steps C and D using cross-entropy loss. F. After preprocessing the micro-expression video to be identified, input it into the trained facial motion pattern feature extraction main network and gradient isolation leakage integral attention module to realize micro-expression classification and recognition. A gradient-isolated leakage integral attention module is constructed to enhance the ability to capture subtle changes in facial muscle activity; the gradient-isolated leakage integral attention module includes leakage integral attention and gradient isolation. Spiking neurons calculate leakage integral attention based on the motion characteristics of the input, including: The membrane potential state of each spiking neuron is updated using equations (5) and (6): V i (0)=V i-1 (T) (6) Among them, V i Let w be the membrane potential of the i-th spiking neuron at time t, where τ is the leakage coefficient, which determines the influence of the potential from the previous time step on the current time step. ij Is input x j (t) represents the synaptic weight of spiking neuron i, where δ is a binary variable indicating whether a pulse occurred at the previous time step t-1. Whenever the potential exceeds the threshold, the spiking neuron fires a pulse and transmits the excess potential portion to the potential at the next time step as its initial value. The spiking neuron retains the membrane potential generated at the last time step of the previous layer of attention and uses it as the initial potential value V for the current layer of leakage integral attention. i (0); Gradient isolation is implemented as follows: First, spiking neurons Responsible for generating the attention value Att of the corresponding feature layer i spiking neurons The forward generation process means that after the time step cycle of the spiking neuron ends, the spiking signal will be transmitted to the next neuron in the same layer; at the same time, the facial motion pattern feature extraction main network will transmit the extracted features to the next layer. Secondly, the remaining potential Mem in the current layer continues to propagate to the next layer, affecting the spiking neurons in that layer. Perform initialization; Finally, the spiking neurons are independently updated and optimized using the second cross-entropy loss, while the parameters of the spiking neurons are updated using the first cross-entropy loss.

2. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, In step A, the micro-expression video is preprocessed, which includes: 1) Obtain video frames: Perform frame segmentation on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames, and save them in chronological order; 2) Face detection and localization: The Dlib vision library is used to detect and localize faces in the separated micro-expression video frames, and the distance between the face and the image boundary in the detected micro-expression video frames is obtained. 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib vision library is used to determine 68 key feature points of the face in the micro-expression video frame, and then rectangular boxes are used for segmentation.

3. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, In step A, after face alignment, face correction is performed, including: Based on the characteristics of the selected micro-expression dataset, adjustments were made to some of the 68 key feature points detected.

4. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, In step B, facial keyframe differences Defined as peak frame X σ The pixel difference between the starting frame X1 and the starting frame X1 is shown in Equation (1):

5. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, In step C, the facial motion pattern feature extraction main network utilizes the facial keyframe differences obtained in step B, and uses ResNet to further learn and model micro-expression information from these keyframe differences; including: The main network for facial motion pattern feature extraction consists of convolutional layers, pulse residual blocks, and a feature classifier; the pulse residual block consists of two independent convolutional layers. The convolutional layer extracts preliminary spatial features; the output features of the convolutional layer are further processed by the pulsed residual block; the pulsed residual block further enhances the features, and its output is expressed as Equation (2) and Equation (3): Among them, W i 1 W i 2 and b i 2 These represent the weights and biases of the two convolutional layers in the i-th pulse residual block, respectively. F i For the features output by the convolution of the i-th pulse residual block, Relu(·) represents the activation of the features using a non-linear activation function; BN(·) represents the normalization of the features using batch normalization. After l pulse residual blocks, the final feature map F I Used for the feature classifier, which predicts the sentiment state through a fully connected layer, as shown in Equation (4): Among them, F l Here, c represents the feature output of the last pulse residual block, and c is the micro-expression category to be predicted. and The weights and biases of the fully connected layer are denoted by Softmax, which is the probability output function to ensure that the sum of the output probability distributions is 1, thereby classifying the three states as positive, negative, and surprised.

6. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, The specific implementation process for calculating the leakage current integral attention is as follows: First, a gradient-isolated leakage integral attention module is constructed using the feature map X as input; the parameters of the gradient-isolated leakage integral attention module include weights W. att Bias b att And the potential leakage parameter τ; Secondly, initialize the relevant parameters, including the membrane potential Mem[t] and the initial attention state Att. i Weight W att and bias b att ; Then, the forward propagation process begins, and the state of the leakage integral attention neuron is updated by setting a time step T. The process is described by equation (7): Mem[t]=τMem[t-1]+f(X,W att ,b att )(7) Where f(i) is the linear computation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att i A value of 1 indicates the active state; otherwise, a value of 0 indicates the inactive state. When Mem[t] is in an active state, it is reset back to its initial state. Finally, the backpropagation process for gradient isolation is performed according to equation (8): grad=grad / (ctx.k*|Mem[T]+1.0| 2 )(8) Where grad is the gradient of the leakage integral attention neuron, and ctx.k is the attention state Att during the forward process. i .

7. A micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to any one of claims 1-6, characterized in that, The optimization formulas for the first cross-entropy loss and the second cross-entropy loss are shown in Equation (9): Where N is the total number of samples to be tested, and M is the number of categories to be classified; y ic Let y be a sign function; if the true class of sample i is m, then y ic If y is 1, otherwise y ic p is 0; ic Let i be the probability that observed sample i belongs to category m; Leakage integral attention of isolated gradient and facial motion pattern feature extraction main network gradient optimization are described by equations (10)-(12): Among them, △W att ΔW represents the updated weights of the spiking neurons, and ΔW represents the updated weights of the main network convolutional units. Let be the pulse value generated by the spiking neuron of the i-th layer pulse residual block at time t, which is determined by... And δ is calculated to obtain, Here, δ represents the corresponding input potential value, η represents the activation threshold, and η represents the learning rate. For loss function, Micro-expression features extracted by the main network for facial motion pattern feature extraction.

8. The micro-expression recognition method based on gradient isolation leakage integral attention for facial motion pattern extraction according to claim 1, characterized in that, After preprocessing the micro-expression video to be identified, it is input into a pre-trained facial motion pattern feature extraction main network and a gradient isolation leakage integral attention module to achieve micro-expression classification and recognition; including: First, the micro-expression video to be identified is preprocessed and labeled as peak frame and start frame, and then motion features are obtained after pixel difference calculation. Subsequently, the motion features are first reduced in dimensionality using a 3x3 convolutional kernel. The dimensionality-reduced motion features are then sequentially fed into a spiking residual block and connected using residual connections. The spiking residual block consists of two convolutional layers and a batch normalization operation after each convolutional layer. The input motion features are temporarily stored after processing by the spiking residual block as intermediate features. Simultaneously, each spiking residual block is paired with a spiking neuron module. Each spiking neuron module contains a spiking neuron, which calculates the leakage integral attention based on the input motion features. Finally, the obtained leakage current integral attention is multiplied and weighted with the obtained intermediate features; The output features of the final pulse residual block are fed into a feature classifier for micro-expression recognition and classification.

Images