Micro-expression recognition method based on gradient isolation leakage integral attention

Abstract

The invention relates to a micro-expression recognition method based on gradient isolation leakage integral attention. The method comprises the following steps: A, preprocessing a micro-expression video; b, picking out a peak frame and a start frame from the video sequence preprocessed in the step A, and performing pixel difference operation on the peak frame and the start frame to obtain a face key frame difference; c, constructing a facial motion mode feature extraction main network; d, constructing a gradient isolation leakage integral attention module, and enhancing the capability of capturing subtle changes in facial muscle activities; e, model training; and F, preprocessing a to-be-recognized micro-expression video, and inputting the preprocessed to-be-recognized micro-expression video into the trained facial motion mode feature extraction main network and the gradient isolation leakage integral attention module to realize micro-expression classification recognition. According to the method, representative features generated due to facial subtle motion are processed in a centralized manner by using difference information of a peak frame and a start frame of a micro-expression video sample.

Description

Technical Field

[0001] The present invention relates to a micro-expression recognition method based on gradient isolation leakage integral attention, which focuses on extracting facial motion patterns and belongs to the technical field of deep learning and pattern recognition. Background Art

[0002] Two studies on daily lying showed that 77 young subjects were detected to lie an average of 2 times a day, and 70 middle-aged subjects lied an average of 1 time a day. The results reveal a universal and interesting phenomenon: lying is an inevitable part of human communication. Although the frequency of lying may vary with age, environment, culture and personal differences. And people are not good at detecting deception, and accuracy is usually equivalent to guessing.

[0003] Microexpressions, as a subtle, short-lived and spontaneous emotional representation of human facial expressions, have been an important topic in psychology and human-computer interaction research since Professor Paul Ekman first discovered them in the 1960s. Although these short facial movements last no more than half a second, they can reveal the most real and uncontrollable part of people's emotional expressions that they try to hide. Therefore, the accurate recognition of microexpressions has extremely important application value in security interrogation, clinical diagnosis, business negotiations and daily social interactions.

[0004] However, the spontaneity, brevity and subtlety of micro-expressions make them extremely challenging to identify (they generally last only 1 / 25 to 1 / 3 of a second), and even trained experts have difficulty accurately identifying micro-expressions. The best recognition rate of professionally trained experts in the test is still only 47%, so it is still very difficult to manually identify micro-expressions.

[0005] In recent years, with the rapid development of computer vision and multimedia technology, it has become possible to automatically recognize micro-expressions using computers, which has also become a hot topic of research in recent years, and a variety of micro-expression recognition algorithms have emerged. Early micro-expression recognition algorithms mostly relied on traditional machine learning models that manually designed micro-expression features. By utilizing human prior knowledge of the field of micro-expressions, more representative micro-expression features can be designed. However, recognition methods based on artificial features are difficult to capture the complete and comprehensive changes in micro-expressions, and they tend to focus more on specific types of data distribution (such as facial texture, light changes, etc.). Incomplete representation makes it difficult for the model to achieve the expected effect. Later, with the success of deep learning-based methods in feature extraction, more researchers began to explore the possibility of micro-expression recognition tasks on deep learning models. Although deep learning-based methods have made progress in feature extraction, how to effectively improve recognition accuracy and reduce misjudgment is still a key issue that needs to be solved in current research.

[0006] In the early days, Jianzheng Liu et al. first proposed a new method for facial expression recognition based on motion templates and 4-layer deep learning neural networks. The motion template and adaboost were used to identify action units as features, and then the extracted features were input into the 4-layer deep learning neural network to recognize facial expressions. At this time, the use of deep learning models for micro-expression recognition began to enter the research field. Later, Peng et al. proposed a dual-stream dual-stream time-scale convolutional neural network (DTSCNN), which uses independent shallow convolutional networks to adapt to micro-expression video samples with different frame rates for different streams. However, since it uses a dual-stream network to calculate the optical flow between adjacent frames, it inevitably introduces a large amount of calculation. On this basis, Liong et al. further proposed a shallow triple stream three-dimensional convolutional network (Shallow Triple Stream Three-dimensional CNN, STSTNet), combining the optical flow of the start frame and the peak frame as the input of the model. Verma et al. proposed a lateral accretive hybrid network (LEARNet) based on dynamic imaging, which compresses the complete video sequence into one frame as input to preserve complete muscle movement information. In recent years, the method of using deep learning models for micro-expression recognition tends to combine excellent algorithms that have been verified in other fields. In 2021, Rahil Kadakia et al. explored the effect of using different deep learning models (such as VGG16, ResNet50, MobileNet, InceptionV3, and Xception) for micro-expression recognition on the SAMM dataset, and found that the MobileNet model exceeded other models in terms of efficiency. This work reduced the computational complexity of using traditional manual features combined with deep learning methods. In December of the same year, Zhao et al. summarized the research on micro-expression recognition based on deep learning methods, discussed the development problems and trends of mainstream algorithms such as DBN and CNN, and pointed out the application prospects of deep learning in the field of micro-expression recognition. Later, Zhao et al. proposed a deep prototype learning framework ME-PLAN in 2022, which solved the key challenges in micro-expression recognition through the local attention mechanism and demonstrated its superiority and effectiveness on composite databases. Lei et al. proposed a graph-time convolutional network based on graph structure feature representation, which used attention modules of different scales to discriminate the intensity of motion in different facial regions to identify subtle changes in expression. In the same year, Xie et al. also adopted a deep mutual learning network to combine AU with emotion categories, and obtained relevant feature representations of micro-expression inputs through difference images of key frames, thereby improving recognition accuracy and speed.Recently, according to the Self-Supervised Motion Learning for Micro-Expression (SelfME) framework proposed by Fan et al. in 2023, it mainly captures the changing patterns of human micro-expression facial movements, trying to overcome the model's reliance on manual features. Nguyen et al. proposed a BERT-based Facial Micro-Expression Recognition (Micron-BERT) framework based on the BERT model. With the sequence modeling capability of the BERT framework, the Patch of Interest (POI) in the micro-expression frame is located to reduce the impact of background noise on the model recognition performance.

[0007] At present, recognition algorithms based on deep learning often rely on the powerful feature modeling capabilities of neural networks, while ignoring the weak and instantaneous characteristics of micro-expression features. This makes the model recognition performance often determined by the advanced level of the deep model used, making it difficult to further improve its recognition rate. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a micro-expression recognition method for facial movement pattern extraction based on gradient isolation leakage integral attention. The present invention focuses on the effective combination of deep learning framework and facial subtle movement features, rather than simply relying on the modeling ability of neural networks for abstract features. Summary of the invention:

[0010] A micro-expression recognition method based on facial movement pattern extraction of gradient isolation leaky integral attention includes data set preprocessing, facial movement pattern feature extraction module, gradient isolation leaky integral attention and classification recognition module.

[0011] The technical problem solved by the present invention is that the existing methods have difficulty in modeling the weak dynamic features of micro-expressions, resulting in low recognition accuracy. Compared with the existing methods, the present invention uses a neural network to model the difference between the peak frame and the starting frame of micro-expressions to effectively capture the slight changes in facial muscles. In addition, the present invention proposes gradient isolation leakage integral attention, which enhances the model's ability to model subtle facial changes by enhancing the subtle movement patterns of specific facial areas within the duration of micro-expressions.

[0012] Terminology explanation:

[0013] 1. Dlib Visual Library: Dlib is a C++ open source toolkit that contains machine learning algorithms. Dlib can help create a lot of complex machine learning software to help solve practical problems. Currently, Dlib has been widely used in industry and academic fields, including robots, embedded devices, mobile phones and large high-performance computing environments.

[0014] 2. 68 key facial feature points. The 68 key facial feature points are mainly distributed in eyebrows, eyes, nose, mouth and facial contours, such as Figure 1 As shown, detection is performed using the Dlib visual library, which is a prior art.

[0015] 3. Loss function: The loss function is used to evaluate the degree of inconsistency between the model's predicted value and the true value. The smaller the loss function, the better the robustness of the model. The loss function can guide model learning.

[0016] 4. Convolutional Residual Network ResNet: It is a deep learning model originally proposed by He Kaiming et al. in 2015. Its various variants have been widely used in the fields of video classification and behavior recognition.

[0017] 5. Pulse neuron: Pulse neuron is a neuron model that is designed to be closer to the behavior of biological neurons. Compared with the neuron model used in traditional artificial neural networks, pulse neurons can process and generate temporally discrete signals.

[0018] 6. Leakage integration: The leakage integration mechanism is a pulse neuron model used to simulate the electrical activity of neurons. It can establish an independent time domain and encode information.

[0019] The technical solution of the present invention is as follows:

[0020] A micro-expression recognition method based on facial movement pattern extraction using gradient-isolated leaky integral attention, comprising:

[0021] A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment;

[0022] B. Select the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference operation on them to obtain the difference of facial key frames;

[0023] C. Constructing the main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction uses the facial key frame differences obtained in step B and uses ResNet to further learn and model micro-expression information of the facial key frame differences.

[0024] D. Construct a gradient-isolated leaky-integrated attention module to enhance the ability to capture subtle changes in facial muscle activity;

[0025] E. Model training, using cross entropy loss to iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integration attention module constructed in steps C and D;

[0026] F. After preprocessing the micro-expression video to be recognized, input the trained facial movement pattern feature extraction main network and gradient isolation leakage integration attention module to realize micro-expression classification and recognition.

[0027] Preferably, according to the present invention, in step A, the micro-expression video is preprocessed, and the step includes:

[0028] 1) Obtain video frames: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames and save them in chronological order;

[0029] 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames;

[0030] 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and then a rectangular frame is used for segmentation.

[0031] Preferably, according to the present invention, in step A, face correction is performed after face alignment, including:

[0032] According to the characteristics of the selected micro-expression dataset, some key feature points among the detected 68 key feature points are adjusted.

[0033] According to the preferred embodiment of the present invention, in step B, the facial key frame difference Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is as shown in formula (1):

[0034]

[0035] Preferably, according to the present invention, in step C, the facial movement pattern feature extraction main network uses the facial key frame differences obtained in step B, and uses ResNet to further learn and model micro-expression information of the facial key frame differences; comprising:

[0036] The main network for facial motion pattern feature extraction includes convolutional layers, impulse residual blocks and feature classifiers; the impulse residual block includes two independent convolutional layers;

[0037] The convolution layer extracts preliminary spatial features; the output features of the convolution layer are further processed by the impulse residual block; the impulse residual block further enhances the features, and its output is expressed as formula (2) and formula (3):

[0038]

[0039]

[0040] Among them, W i 1 , W i 2 and denote the weights and biases of the two convolutional layers in the i-th impulse residual block; F i It is the feature output by the convolution of the i-th impulse residual block;

[0041] Relu(·) represents the use of nonlinear activation function to activate feature operations; BN(·) represents the use of batch normalization to normalize features;

[0042] After l impulse residual blocks, the final feature F l The mapping is used for the feature classifier, which predicts the emotional state through a fully connected layer, as shown in formula (4):

[0043]

[0044] Among them, Fl is the feature output of the last pulse residual block, c is the micro-expression category to be predicted, and are the weights and biases of the fully connected layer, and Softmax is the probability output function to ensure that the sum of the output probability distribution is 1, thereby classifying between the three states of positive, negative, and surprise.

[0045] Preferably, according to the present invention, a gradient-isolated leakage-integrated attention module is constructed to enhance the ability to capture subtle changes in facial muscle activity; the gradient-isolated leakage-integrated attention module includes leakage-integrated attention and gradient isolation.

[0046] Preferably, according to the present invention, the spiking neuron calculates the leakage integrated attention according to the input motion characteristics, including:

[0047] The membrane potential state of each spiking neuron is updated as Equation (5) and Equation (6):

[0048]

[0049] V i (0) = V i-1(T) (6)

[0050] Among them, V i is the membrane potential of the ith pulse neuron at time, τ is the leakage coefficient, which determines the influence of the potential at the previous moment on the current moment, and w ij is the input x j (t) to the synaptic weight of the spiking neuron i, δ is a binary variable, indicating whether a pulse occurred at the previous time t-1; whenever the potential exceeds the threshold, the spiking neuron emits a pulse and passes the excess potential to the potential at the next moment as its initial value; the spiking neuron retains the membrane potential generated by the last time step of the previous layer of attention and uses it as the initial value of the potential V of the current layer of leakage integrated attention i (0).

[0051] Preferably, according to the present invention, the leakage integral attention is calculated, and the specific implementation process is as follows:

[0052] First, the feature map X is used as input to construct a gradient isolation leaky integral attention module; the parameters of the gradient isolation leaky integral attention module include the weight W att , bias b att and the potential leakage parameter τ;

[0053] Secondly, initialize relevant parameters, including membrane potential Mem[t], initial attention state Att i , weight W att and bias b att ;

[0054] After that, the forward propagation process starts, and the time step T is set to update the state of the leaky integral attention neuron. The process is described as formula (7):

[0055] Mem[t]=τMem[t-1]+f(X,W att ,b att ) (7)

[0056] Among them, f(i) is the linear calculation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att i If it is 1, it is in the activated state, otherwise, it will be set to 0, that is, in the inactivated state;

[0057] When in the activated state, Mem[t] is reset to the initial state;

[0058] Finally, the back propagation process of gradient isolation is performed according to formula (8):

[0059] grad=grad / (ctx.k*|Mem[T]+1.0| 2 ) (8)

[0060] Among them, grad is the gradient of the leakage integrated attention neuron, and ctx.k is the attention state Att in the forward process i (i.e. 0 or 1).

[0061] According to the preferred embodiment of the present invention, the gradient isolation is specifically implemented as follows:

[0062] First, the spiking neurons Responsible for generating the corresponding feature layer (i.e. )’s attention value Att i , spiking neurons The forward generation process means that when the time step cycle of the spike neuron ends, the spike signal will be passed to the next neuron in the same layer; at the same time, the main network for facial motion pattern feature extraction will pass the extracted features to the next layer;

[0063] Secondly, the remaining potential Mem of the current layer continues to be transmitted to the next layer, which affects the pulse neurons of the next layer. Initialize;

[0064] Finally, the pulse neuron is independently optimized for weight update by the second cross entropy loss, and the first cross entropy loss is set up to update the pulse neuron parameters.

[0065] Further preferably, the optimization formula of the first cross entropy loss and the second cross entropy loss is as shown in formula (9):

[0066]

[0067] Among them, N is the total number of samples to be tested, M is the number of categories that need to be classified; y ic is a sign function (0 or 1), if the true category of sample i is m, then y ic is 1, otherwise y ic is 0; p ic is the probability that the observed sample i belongs to category m.

[0068] Preferably, according to the present invention, the leakage integral attention of the isolated gradient and the main network gradient optimization of facial movement pattern feature extraction are described as formulas (10)-(12):

[0069]

[0070]

[0071]

[0072] Among them, △W att is the updated value of the pulse neuron weight, △W is the updated value of the main network convolution unit weight, is the pulse value generated by the pulse neuron of the pulse residual block in the i-th layer at time t, which is given by And δ is calculated, is the corresponding input potential value, δ is the activation threshold, η is the learning rate, is the loss function, Micro-expression features extracted by the main network for facial movement pattern feature extraction.

[0073] Preferably, according to the present invention, after pre-processing the micro-expression video to be recognized, the trained facial movement pattern feature extraction main network and gradient isolation leakage integral attention module are input to realize micro-expression classification and recognition; comprising:

[0074] First, the micro-expression video to be recognized is pre-processed and marked as the peak frame and the starting frame, and the motion features are obtained after pixel difference calculation;

[0075] After that, the motion features are first reduced in dimension through a convolution kernel with a convolution kernel size of 3x3, and then the motion features after dimensionality reduction are sequentially sent to the pulse residual block and residual links are used; the pulse residual block includes two convolution layers and a batch normalization operation after each convolution layer; the input motion features are temporarily saved after being processed by the pulse residual block as intermediate features, and at the same time, each pulse residual block is paired with a pulse neuron module. The pulse neuron module includes a pulse neuron, which calculates the leakage integral attention according to the input motion features;

[0076] Finally, the obtained leakage integral attention is multiplied and weighted with the obtained intermediate features;

[0077] The above process will continue until the last pulse residual block, and the output features of the last pulse residual block will be sent to the feature classifier for micro-expression recognition and classification.

[0078] The beneficial effects of the present invention are:

[0079] 1. In order to enhance the network's modeling ability for micro-expression features, the present invention proposes a deep model that focuses on facial dynamic feature extraction. The difference information between the peak frame and the starting frame of micro-expression video samples is used to centrally process representative features generated by subtle facial movements.

[0080] 2. In order to capture the subtle changes in facial muscle activity, the present invention proposes a gradient isolation leaky integral attention mechanism. By emphasizing the subtle movement patterns exhibited by specific facial regions during the duration of micro-expressions, the model can learn the features in specific facial regions more focusedly. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 Schematic diagram of 68 key feature points of the face of the present invention;

[0082] Figure 2 This is a flow chart of the micro-expression recognition method based on gradient-isolated leaky integral attention;

[0083] Figure 3 Schematic diagram of the network structure for facial motion pattern extraction based on gradient-isolated leaky integrated attention;

[0084] Figure 4 This is a diagram showing the visualization effect of traditional channel attention weights;

[0085] Figure 5 This is a diagram showing the visualization effect of traditional spatial attention weights;

[0086] Figure 6 This is a diagram showing the visualization effect of multi-head attention weights;

[0087] Figure 7 This is a schematic diagram of the visualization effect of the gradient isolation leakage integral attention weight of the present invention;

[0088] Figure 8 Schematic diagram of the attention gradient isolation implementation. DETAILED DESCRIPTION

[0089] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0090] Example 1

[0091] A micro-expression recognition method based on facial movement pattern extraction using gradient-isolated leaky integral attention, such as Figure 2 As shown, including:

[0092] A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment;

[0093] B. Select the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference operation on them to obtain the difference of facial key frames;

[0094] C. Constructing the main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction uses the facial key frame differences obtained in step B and uses ResNet to further learn and model micro-expression information of the facial key frame differences.

[0095] D. Construct a gradient-isolated leaky-integrated attention module to enhance the ability to capture subtle changes in facial muscle activity;

[0096] E. Model training, using cross entropy loss to iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integration attention module constructed in steps C and D;

[0097] F. After preprocessing the micro-expression video to be recognized, input the trained facial movement pattern feature extraction main network and gradient isolation leakage integration attention module to realize micro-expression classification and recognition.

[0098] Example 2

[0099] The difference between the micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention described in Example 1 is that:

[0100] In step A, the micro-expression video is preprocessed, which includes:

[0101] 1) Obtain video frames: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames, and save them in chronological order as I(0,1,2,…,I).png format;

[0102] 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames;

[0103] 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and then a rectangular frame is used for segmentation. The size of the rectangular frame is set to 256*256.

[0104] In step A, face alignment is followed by face correction, including:

[0105] According to the characteristics of the selected micro-expression data set (including race, gender, shooting angle, etc.), some key feature points are adjusted among the 68 key feature points detected. For example, a line can be drawn between the key feature point 37 marking the left corner of the left eye and the key feature point 46 marking the right corner of the right eye. This line may have an angle α with the standard horizontal line. Through this angle, the segmented face can be rotated so that the line between the key feature points 37 and 46 of the face is parallel to the standard horizontal line, thereby realizing the calibration of the face posture.

[0106] In step B, the facial keyframe differences Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is as shown in formula (1):

[0107]

[0108] In step C, the main facial motion pattern feature extraction network uses the facial key frame differences obtained in step B and uses ResNet to further learn and model micro-expression information of the facial key frame differences; including:

[0109] The main network for facial motion pattern feature extraction includes convolutional layers, impulse residual blocks and feature classifiers; the impulse residual block includes two independent convolutional layers;

[0110] The convolution layer extracts preliminary spatial features; the output features of the convolution layer are further processed by the impulse residual block; the impulse residual block further enhances the features, and its output is expressed as formula (2) and formula (3):

[0111]

[0112]

[0113] Among them, W i 1 , W i 2 and They represent the weights and biases of the two convolutional layers in the i-th impulse residual block respectively;

[0114] F i is the feature output by the convolution of the ith impulse residual block, Relu(·) represents the use of nonlinear activation function to activate the feature operation; BN(·) represents the use of batch normalization to normalize the feature;

[0115] After l impulse residual blocks, the final feature map F I For feature classifier, the feature classifier predicts the emotional state through a fully connected layer, as shown in formula (4):

[0116]

[0117] Among them, Fl is the feature output of the last pulse residual block, c is the micro-expression category to be predicted, and are the weights and biases of the fully connected layer, and Softmax is the probability output function to ensure that the sum of the output probability distribution is 1, thereby classifying between the three states of positive, negative, and surprise.

[0118] Construct a gradient-isolated leaky-integrated attention module to enhance the ability to capture subtle changes in facial muscle activity; the gradient-isolated leaky-integrated attention module includes leaky-integrated attention and gradient isolation. Leaky-integrated attention: The leaky-integrated attention mechanism is introduced into the proposed recognition main network to enhance the neural network's sensitivity to subtle facial muscle movements. The leaky-integrated attention mechanism combines the temporal dynamics of the spiking neural network with the representational power of deep learning.

[0119] The spiking neuron calculates leaky integrated attention based on the motion features of the input, including:

[0120] The membrane potential state of each spiking neuron is updated as Equation (5) and Equation (6):

[0121]

[0122] V i (0) = V i-1 (T) (6)

[0123] Among them, V i is the membrane potential of the ith pulse neuron at time, τ is the leakage coefficient, which determines the influence of the potential at the previous moment on the current moment, and w ij is the input x j (t) is the synaptic weight of the spiking neuron i, δ is a binary variable, indicating whether a pulse occurred at the previous time t-1; whenever the potential exceeds the threshold, the spiking neuron emits a pulse and passes the excess potential to the potential at the next moment as its initial value; it is worth noting that the proposed spiking neuron retains the membrane potential generated by the last time step of the previous layer of attention and uses it as the initial value of the potential V of the current layer of leakage integrated attention i (0). Due to the insufficient discrete representation of information by leaky integration, this method is beneficial to retain shallow features and save the modeling calculation process of the learned information per unit time step of the deep leaky integration attention unit.

[0124] Calculate the leakage integral attention. The specific implementation process is as follows:

[0125] First, the feature map X is used as input to construct a gradient isolation leaky integral attention module; the parameters of the gradient isolation leaky integral attention module include the weight W att , bias b att and the potential leakage parameter τ;

[0126] Secondly, initialize relevant parameters, including membrane potential Mem[t], initial attention state Att i , weight W att and bias b att ;

[0127] After that, the forward propagation process starts, and the time step T is set to update the state of the leaky integral attention neuron. The process is described as formula (7):

[0128] Mem[t]=τMem[t-1]+f(X,W att ,b att ) (7)

[0129] Among them, f(i) is the linear calculation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att i If it is 1, it is in the activated state, otherwise, it will be set to 0, that is, in the inactivated state;

[0130] When in the activated state, Mem[t] is reset to the initial state;

[0131] Finally, the back propagation process of gradient isolation is performed according to formula (8):

[0132] grad=grad / (ctx.k*|Mem[T]+1.0| 2 ) (8)

[0133] Among them, grad is the gradient of the leakage integrated attention neuron, and ctx.k is the attention state Att in the forward process i (i.e. 0 or 1).

[0134] Since the leaky integrated attention output is a binary signal, it is difficult for the network to perform normal gradient descent to optimize the network weights. Based on this, the proposed attention unit is gradient isolated. Figure 8 Schematic diagram of gradient isolation for attention units. The specific implementation process of gradient isolation is as follows:

[0135] First, the spiking neurons Responsible for generating the corresponding feature layer (i.e. )’s attention value Att i , spiking neurons The forward generation process means that when the time step cycle of the spike neuron ends, the spike signal will be passed to the next neuron in the same layer (such as Figure 8 At the same time, the facial motion pattern feature extraction main network passes the extracted features to the next layer (as shown in the vertical direction). Figure 8 The horizontal direction is shown by the dark solid line);

[0136] Secondly, the remaining potential Mem of the current layer continues to be transferred to the next layer (such as Figure 8 The horizontal direction is shown as a light solid line), which is used to pulse neurons in the next layer. Initialize;

[0137] Finally, the spike neurons are optimized with independent weight updates using the second cross entropy loss (cross entropy loss 2) (e.g. Figure 8 In order to update the weights of the backbone classification network when the pulse neuron gradient is not directly fed back to the backbone network, the first cross entropy loss (cross entropy loss 1) is set to update the pulse neuron parameters (as shown in the dark dashed line in the vertical direction). Figure 8 The horizontal dark dashed line in the middle).

[0138] The optimization formula of the first cross entropy loss and the second cross entropy loss is shown in formula (9):

[0139]

[0140] Among them, N is the total number of samples to be tested, M is the number of categories that need to be classified; y ic is a sign function (0 or 1), if the true category of sample i is m, then y ic is 1, otherwise y ic is 0; p ic is the probability that the observed sample i belongs to category m.

[0141] The difference between cross entropy loss 1 and cross entropy loss 2 is that they optimize different weights. Cross entropy loss 1 optimizes the weights △W of the main network convolution unit, while cross entropy loss 2 optimizes the weights △W of the spike neurons. att , as shown in formula (8) and formula (7).

[0142] The leaky integral attention of isolated gradient and the main network gradient optimization for facial motion pattern feature extraction are described as Eqs. (10)-(12):

[0143]

[0144]

[0145]

[0146] Among them, △W att is the updated value of the pulse neuron weight, △W is the updated value of the main network convolution unit weight, is the pulse value generated by the pulse neuron of the pulse residual block in the i-th layer at time t, which is given by And δ is calculated, is the corresponding input potential value, δ is the activation threshold, η is the learning rate, is the loss function, Micro-expression features extracted by the main network for facial movement pattern feature extraction.

[0147] The gradient-isolated leaky-integrated attention mechanism not only preserves the temporal dynamics of spiking neurons and provides a natural way to capture weak micro-expression information, but also ensures the training stability of the network and avoids the interference of the non-traditional learning dynamics of spiking neurons on the optimization of the main network weights.

[0148] After preprocessing the micro-expression video to be recognized, the trained facial movement pattern feature extraction main network and gradient isolation leakage integration attention module are input to realize micro-expression classification and recognition; including:

[0149] First, the micro-expression video to be recognized is pre-processed and marked as the peak frame and the starting frame, and the motion features are obtained after pixel difference calculation;

[0150] After that, the motion features are first passed through a convolution kernel with a size of 3x3 ( Figure 3 Conv in the figure) for dimensionality reduction, and then the motion features after dimensionality reduction are sent to the pulse residual block ( Figure 3 SRB in ), and using residual links ( Figure 3 The pulse residual block includes two convolutional layers ( Figure 3 Conv1 and Conv2), and batch normalization after each convolutional layer ( Figure 3 BN1 and BN2); the input motion features are temporarily saved after being processed by the pulse residual block as intermediate features, such as Figure 3 As shown in the light-colored rectangle. At the same time, each pulse residual block is paired with a pulse neuron module ( Figure 3 The spiking neuron module includes a spiking neuron ( Figure 3 SN in the figure), the spiking neuron calculates the leakage integral attention ( Figure 3 In the formula (5), it is represented as Atti(T), and the calculation process is shown in the following formula (6);

[0151] Finally, the obtained leakage integral attention is multiplied and weighted with the obtained intermediate features;

[0152] The above process will continue until the last pulse residual block, and the output features of the last pulse residual block will be sent to the feature classifier for micro-expression recognition and classification.

[0153] In this embodiment, the original videos of CASMEⅡ micro-expression database released by Fu Xiaolan's team of Institute of Psychology, Chinese Academy of Sciences, SAMM database released by Davison's team of Manchester Metropolitan University, UK, and SMIC database released by the research team of University of Oulu, Finland were tested for micro-expression recognition. At the same time, the test results of the present invention were compared with the existing methods based on traditional learning and deep learning, and Table 1 shows the test results.

[0154] Table 1

[0155]

[0156] In Table 1, Pre-P: preprocessing method; E: Euler image enlargement; R: region of interest; T: time domain normalization; Pre-T: pre-training dataset.

[0157] In Table 1, the LBP-TOP method comes from the literature Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions [J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29 (6): 915-928.

[0158] FDM comes from the literature Xu F, Zhang J, Wang J Z. Micro-expression identification and categorization using a facial dynamics map [J]. IEEE Transactions on AffectiveComputing, 2017, 8(2): 254-267.

[0159] MDMO comes from the literature Liu YJ, Zhang JK, Yan WJ, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition [J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299-310.

[0160] TSCNN comes from the literature Song B, Li K, Zong Y, et al. Recognizing spontaneous micro-expression using a three-stream convolutional neural network [J]. IEEE Access, 2019, 7: 184537-184551.

[0161] STSTNet is from the literature Liong S T, Gan Y S, See J, et al. Shallow triple stream three - dimensional cnn (ststnet) for micro - expression recognition[C] / / 2019 14th IEEE international conference on automatic face&gesture recognition (FG 2019). IEEE, 2019:1 - 5.

[0162] DIKD is from the literature Sun B, Cao S, Li D, et al. Dynamic micro - expression recognition using knowledge distillation[J]. IEEE Transactions on Affective Computing, 2020, 13(2):1037 - 1043.

[0163] DSTAN is from the literature Wang Y, Huang Y, Liu C, et al. Micro expression recognition via dual - stream spatiotemporal attention network[J]. Journal of Healthcare Engineering, 2021, 2021.

[0164] GEME is from the literature Nie X, Takalkar M A, Duan M, et al. GEME: Dual - stream multi - task GEnder - based micro - expression recognition[J]. Neurocomputing, 2021, 427:13 - 28.

[0165] ME - PLAN is from the literature Zhao S, Tang H, Liu S, et al. ME - PLAN: A deep prototypical learning with local attention network for dynamic micro - expression recognition[J]. Neural Networks, 2022, 153:427 - 443.

[0166] BDCNN comes from the document Chen B, Liu KH, Xu Y, et al.Block division convolutionalnetwork with implicit deep features augmentation for micro-expressionrecognition[J]. IEEE Transactions on Multimedia, 2022,25:1345-1358.

[0167] According to Table 1, the recognition rates (accuracy ACC., average F1 score UF1 and average recall rate UAR) of the proposed method on CASMEII, SAMM and SMIC datasets are better than those of other methods. The accuracy of the proposed method on the three datasets reached 92.9%, 78.9% and 75.2% respectively, the average F1 score reached 90.5%, 68.7% and 75.0% respectively, and the average recall rate reached 90.5%, 73.9% and 77.9% respectively. Figure 4 This is a diagram showing the visualization effect of traditional channel attention weights; Figure 5 This is a diagram showing the visualization effect of traditional spatial attention weights; Figure 6 This is a diagram showing the visualization effect of multi-head attention weights; Figure 7 This is a schematic diagram of the visualization effect of the gradient isolation leakage integral attention weight of the present invention; this is because the algorithm of the present invention has a more concentrated processing weight for the information at a specific position in the facial feature map compared to other algorithms. It can lock the area with the most significant changes in facial expressions, such as around the eyes and mouth, and then identify subtle facial muscle changes, rather than uniformly learning the global information of the input features, effectively avoiding the problem that the network pays too much attention to global changes and causes the algorithm to ignore subtle facial expressions. At the same time, due to the invention of using micro-expression video frame pixel difference, the recognition ability of the present invention for key micro-expression features is further enhanced, and the sensitivity of the overall model to subtle emotional changes is also significantly improved.

Claims

1. A micro-expression recognition method based on facial movement pattern extraction using gradient-isolated leaky integral attention, characterized in that: include: A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment; B. Select the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference operation on them to obtain the difference of facial key frames; C. Constructing the main network for facial motion pattern feature extraction. The main network for facial motion pattern feature extraction uses the facial key frame differences obtained in step B and uses ResNet to further learn and model micro-expression information of the facial key frame differences. D. Construct a gradient-isolated leaky-integrated attention module to enhance the ability to capture subtle changes in facial muscle activity; E. Model training, using cross entropy loss to iteratively update the facial motion pattern feature extraction main network and gradient isolation leakage integration attention module constructed in steps C and D; F. After preprocessing the micro-expression video to be recognized, input the trained facial movement pattern feature extraction main network and gradient isolation leakage integration attention module to realize micro-expression classification and recognition.

2. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: In step A, the micro-expression video is preprocessed, which includes: 1) Obtain video frames: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames and save them in chronological order; 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames; 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and then a rectangular frame is used for segmentation.

3. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: In step A, face alignment is followed by face correction, including: According to the characteristics of the selected micro-expression dataset, some key feature points among the detected 68 key feature points are adjusted.

4. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: In step B, the facial keyframe differences Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is as shown in formula (1):

5. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: In step C, the main facial motion pattern feature extraction network uses the facial key frame differences obtained in step B and uses ResNet to further learn and model micro-expression information of the facial key frame differences; including: The main network for facial motion pattern feature extraction includes convolutional layers, impulse residual blocks and feature classifiers; the impulse residual block includes two independent convolutional layers; The convolution layer extracts preliminary spatial features; the output features of the convolution layer are further processed by the impulse residual block; the impulse residual block further enhances the features, and its output is expressed as formula (2) and formula (3): Among them, W i 1 , W i 2 and They represent the weights and biases of the two convolutional layers in the i-th impulse residual block respectively; F i is the feature output by the convolution of the ith impulse residual block, Relu(·) represents the use of nonlinear activation function to activate the feature operation; BN(·) represents the use of batch normalization to normalize the feature; After l impulse residual blocks, the final feature map F I For feature classifier, the feature classifier predicts the emotional state through a fully connected layer, as shown in formula (4): Among them, F l is the feature output of the last pulse residual block, c is the micro-expression category to be predicted, and are the weights and biases of the fully connected layer, and Softmax is the probability output function to ensure that the sum of the output probability distribution is 1, thereby classifying between the three states of positive, negative, and surprise.

6. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: Construct a gradient-isolated leaky-integral attention module to enhance the ability to capture subtle changes in facial muscle activity; the gradient-isolated leaky-integral attention module includes leaky-integral attention and gradient isolation.

7. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 6 is characterized in that: The spiking neuron calculates leaky integrated attention based on the motion features of the input, including: The membrane potential state of each spiking neuron is updated as Equation (5) and Equation (6): V i (0)=V i-1 (T) (6) Among them, V i is the membrane potential of the ith pulse neuron at time, τ is the leakage coefficient, which determines the influence of the potential at the previous moment on the current moment, and w ij is the input x j (t) to the synaptic weight of the spiking neuron i, δ is a binary variable, indicating whether a pulse occurred at the previous time t-1; whenever the potential exceeds the threshold, the spiking neuron emits a pulse and passes the excess potential to the potential at the next moment as its initial value; the spiking neuron retains the membrane potential generated by the last time step of the previous layer of attention and uses it as the initial value of the potential V of the current layer of leakage integrated attention i (0).

8. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 6 is characterized in that: Calculate the leakage integral attention. The specific implementation process is as follows: First, the feature map X is used as input to construct a gradient isolation leaky integral attention module; the parameters of the gradient isolation leaky integral attention module include the weight W att , bias b att and the potential leakage parameter τ; Secondly, initialize relevant parameters, including membrane potential Mem[t], initial attention state Att i , weight W att and bias b att ; After that, the forward propagation process starts, and the time step T is set to update the state of the leaky integral attention neuron. The process is described as formula (7): Mem[t]=τMem[t-1]+f(X,W att ,b att ) (7) Among them, f(i) is the linear calculation of the neural network; when Mem[t] exceeds the threshold δ, the attention state Att i If it is 1, it is in the activated state, otherwise, it will be set to 0, that is, in the inactivated state; When in the activated state, Mem[t] is reset to the initial state; Finally, the back propagation process of gradient isolation is performed according to formula (8): grad=grad / (ctx.k*|Mem[T]+1.0| 2 ) (8) Among them, grad is the gradient of the leakage integrated attention neuron, and ctx.k is the attention state Att in the forward process i .

9. A method for micro-expression recognition based on facial movement pattern extraction based on gradient isolation leakage integral attention according to any one of claims 6-8, characterized in that: Gradient isolation, the specific implementation process is as follows: First, the spiking neurons Responsible for generating the attention value Att of the corresponding feature layer i , spiking neurons The forward generation process means that when the time step cycle of the spike neuron ends, the spike signal will be passed to the next neuron in the same layer; at the same time, the main network for facial motion pattern feature extraction will pass the extracted features to the next layer; Secondly, the remaining potential Mem of the current layer continues to be transmitted to the next layer, which affects the pulse neurons of the next layer. Initialize; Finally, the pulse neuron is independently optimized for weight update by the second cross entropy loss, and the first cross entropy loss is set up to update the pulse neuron parameters; Further preferably, the optimization formula of the first cross entropy loss and the second cross entropy loss is as shown in formula (9): Among them, N is the total number of samples to be tested, M is the number of categories that need to be classified; y ic is a symbolic function. If the true category of sample i is m, then y ic is 1, otherwise y ic is 0; p ic is the probability that the observed sample i belongs to category m; Further preferably, the leakage integral attention of the isolated gradient and the main network gradient optimization for facial movement pattern feature extraction are described as equations (10)-(12): Among them, △W att is the updated value of the pulse neuron weight, △W is the updated value of the main network convolution unit weight, is the pulse value generated by the pulse neuron of the pulse residual block in the i-th layer at time t, which is given by And δ is calculated, is the corresponding input potential value, δ is the activation threshold, η is the learning rate, is the loss function, Micro-expression features extracted by the main network for facial movement pattern feature extraction.

10. The micro-expression recognition method based on facial movement pattern extraction based on gradient isolation leakage integral attention according to claim 1 is characterized in that: After preprocessing the micro-expression video to be recognized, the trained facial movement pattern feature extraction main network and gradient isolation leakage integration attention module are input to realize micro-expression classification and recognition; including: First, the micro-expression video to be recognized is pre-processed and marked as the peak frame and the starting frame, and the motion features are obtained after pixel difference calculation; After that, the motion features are first reduced in dimension through a convolution kernel with a convolution kernel size of 3x3, and then the motion features after dimensionality reduction are sequentially sent to the pulse residual block and residual links are used; the pulse residual block includes two convolution layers and a batch normalization operation after each convolution layer; the input motion features are temporarily saved after being processed by the pulse residual block as intermediate features, and at the same time, each pulse residual block is paired with a pulse neuron module; the pulse neuron module includes a pulse neuron, and the pulse neuron calculates the leakage integral attention according to the input motion features; Finally, the obtained leakage integral attention is multiplied and weighted with the obtained intermediate features; The above process will continue until the last pulse residual block, and the output features of the last pulse residual block will be sent to the feature classifier for micro-expression recognition and classification.

Images