Cross-time-domain micro-expression recognition method based on multi-tense information entropy

Through the multi-temporal information entropy cross-time domain method, using asymmetric autoencoder and pulse time domain matrix, combined with adaptive threshold function, the modeling problem of micro-expression recognition model for continuous transient characteristics of micro-expression is solved, and the micro-expression recognition model is realized Effective modeling and recognition of different scale features of expressions improves recognition accuracy and adaptability.

CN118918620APending Publication Date: 2024-11-08SHANDONG UNIV +1

Patent Information

Application Number
CN202410722223.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-05
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing micro-expression recognition models are difficult to effectively model the continuous transient characteristics of micro-expression, and the lack of modeling of time-domain characteristics hinders the research and development of micro-expression recognition models.

Method used

Using a cross-time domain method based on multi-temporal information entropy, asymmetric variational autoencoder and pulse time domain matrix are combined with adaptive threshold function to construct a micro-expression feature model in the long and short time domains to realize the characteristics of micro-expression at different scales. comprehensive modeling and identification.

Benefits of technology

The problem of difficult to capture the transient characteristics of micro-expression is overcome, and the model's adaptive selection ability of different data sets, samples and scale features is ensured, which improves the accuracy and effectiveness of micro-expression recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918620A_ABST
    Figure CN118918620A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-tense information entropy-based cross-time-domain micro-expression recognition method, and aims to comprehensively model features of micro-expressions on different scales. Firstly, an asymmetric variational auto-encoder is introduced, unsupervised modeling is carried out on micro-expression short-time domain feature distribution through a generation mode, and the problem that micro-expression instantaneous features are difficult to capture is solved. Secondly, designing a time domain pulse matrix, constructing a new time domain space by using pulse neurons, fusing and screening micro-expression features in two different time states, and ensuring that the model effectively extracts the features; and finally, proposing a self-adaptive threshold function, designing an activation threshold based on the mutual information entropy of the feature map, and enhancing the self-adaptive selection capability of the model for different data sets, samples and scale features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a micro-expression recognition method based on multi-temporal information entropy across time domains, which focuses on modeling features of micro-expressions at different scales and belongs to the technical field of deep learning and pattern recognition. Background Art

[0002] Although we live in an era of continuous technological advancement, language is still the main way for people to communicate. However, when the two parties in communication use different languages, observing the other party's facial expressions may be one of the important ways to obtain accurate communication information. Therefore, expression recognition has a wide range of applications in many fields such as medical monitoring, educational assessment, emotional computing, and fatigue driving monitoring. For example, in the medical field, by observing the patient's facial expression, the doctor can preliminarily infer whether the patient is in pain; teachers can evaluate the student's understanding of current knowledge through the student's facial expression characteristics; during driving, the driver's facial expression can be used to determine whether he is in a state of fatigue. These applications have made expression recognition a research hotspot in the fields of computer vision and human-computer interaction.

[0003] However, when people try to hide their true emotions, conventional facial expressions may lose their effectiveness. Micro-expressions, as a spontaneous and extremely short-lived facial expression, can better reflect real emotions and better understand and capture human emotions because they cannot be controlled by the conscious mind. In the field of healthcare, doctors can more accurately determine whether patients are feeling pain.

[0004] Although micro-expressions can provide more accurate and effective information, they cannot be effectively recognized by the human eye due to their inherent characteristics. Only trained experts can accurately identify micro-expressions. In addition, recognizing micro-expressions by the naked eye is a time-consuming and error-prone process. Therefore, it is of great significance to develop a micro-expression recognition system based on computer vision. In recent years, significant research progress has been made in the field of micro-expression recognition, covering two major directions: traditional machine learning and deep learning.

[0005] In terms of traditional machine learning, researchers widely use local binary patterns (LBP) as classic texture features for micro-expression recognition. Among them, Pfister et al. introduced the LBP algorithm based on three orthogonal planes (LBP-TOP), and combined with a multi-core support vector machine, successfully realized the recognition of natural micro-expressions. In addition, Huang et al. proposed spatiotemporal local binary patterns (STLBP) based on integral projection, which solved the problem of spatiotemporal features in micro-expression recognition. Optical flow is also one of the commonly used methods in micro-expression recognition. Xu et al. used facial dynamic maps to characterize the movement of micro-expressions and realized the division of spatiotemporal cuboids through pixel-level alignment algorithms. In addition, Liong et al. proposed a dual-weighted directional optical flow method to encode the basic expression features of micro-expressions.

[0006] With the development of deep learning technology, micro-expression recognition methods based on deep learning have received widespread attention. Patel et al. combined deep learning with micro-expression recognition for the first time, and performed facial expression transfer learning through convolutional neural networks to reduce the interference of deep features on micro-expression recognition. Peng et al. designed a dual-stream network (DTSCNN) to extract spatial and temporal features of micro-expression video clips through two convolutional neural networks at different time scales. In addition to CNN, networks such as LSTM are widely used in spatiotemporal modeling of micro-expressions. Hochreiter et al. proposed a feature representation method based on long short-term memory recurrent neural network (LSTM) to capture the action flow in micro-expression sequences. In addition, Li et al. proposed a convolutional neural network (3D CNN) based on 3D streams to obtain the fine action flow generated by micro-expressions and applied it to the field of video micro-expression recognition.

[0007] In recent years, the application of transfer learning in micro-expression recognition has attracted more and more attention. Zong et al. studied cross-database micro-expression recognition and proposed a domain regeneration framework to solve the problem of inconsistent feature distribution between different datasets. Jia et al. proposed a conversion model, which is a macro-to-micro learning conversion, and applied it to micro-expression feature extraction and recognition.

[0008] At present, the mainstream micro-expression recognition models are usually based on the micro-expression movement characteristics of long time intervals to highlight the subtle features of micro-expressions, but it is difficult to effectively model the continuous and short-term instantaneous characteristics of micro-expressions. The lack of modeling of time domain characteristics has hindered the research and development of micro-expression recognition models. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention proposes a micro-expression recognition method based on multi-temporal information entropy across time domains. The present invention respectively models the motion characteristics of micro-expressions in long-interval time domains with a residual network, models the continuous frames of micro-expressions in short-interval time domains with an asymmetric autoencoder network, and performs joint screening of long and short time domains with a pulse time domain matrix. By extracting and fusing multi-temporal micro-expression information, the complete modeling is used to identify meaningful feature distributions. Summary of the invention:

[0011] A cross-time domain micro-expression recognition method based on multi-temporal information entropy includes data set preprocessing, a residual network for modeling the motion characteristics of long-interval time domain micro-expressions, an asymmetric autoencoder network for modeling continuous frames of short-interval time domain micro-expressions, and a pulse time domain matrix for joint screening of long and short time domains.

[0012] The technical problem solved by the present invention is that the subtlety and instantaneousness of micro-expressions are unique features that are different from other video recognition tasks. The current mainstream micro-expression recognition models are usually based on the micro-expression motion characteristics of long time intervals, focusing on the subtle features of micro-expressions, but it is difficult to effectively model the continuous and short instantaneous characteristics of micro-expressions. The lack of modeling of time domain characteristics has hindered the research and development of micro-expression recognition models.

[0013] In response to the above technical problems, the present invention proposes a cross-time domain micro-expression recognition method based on multi-temporal information entropy, aiming to comprehensively model the characteristics of micro-expressions at different scales. First, an asymmetric variational autoencoder is introduced to perform unsupervised modeling of the short-time domain feature distribution of micro-expressions in a generative manner, thereby overcoming the problem that the instantaneous characteristics of micro-expressions are difficult to capture. Secondly, a time domain pulse matrix is ​​designed, and a new time domain space is constructed using pulse neurons to fuse and filter the micro-expression features of two different temporal states to ensure that the model effectively extracts features. Finally, an adaptive threshold function is proposed to design the activation threshold based on the mutual information entropy of the feature graph to enhance the model's adaptive selection ability for different data sets, samples, and scale features.

[0014] Terminology explanation:

[0015] 1. Dlib Visual Library: Dlib is a C++ open source toolkit that contains machine learning algorithms. Dlib can help create a lot of complex machine learning software to help solve practical problems. Currently, Dlib has been widely used in industry and academic fields, including robots, embedded devices, mobile phones and large high-performance computing environments.

[0016] 2. 68 key facial feature points. The 68 key facial feature points are mainly distributed in eyebrows, eyes, nose, mouth and facial contours, such as Figure 1 As shown, detection is performed using the Dlib visual library, which is a prior art.

[0017] 3. Loss function: The loss function is used to evaluate the degree of inconsistency between the model's predicted value and the true value. The smaller the loss function, the better the robustness of the model. The loss function can guide model learning.

[0018] 4. Convolutional Residual Network ResNet: It is a deep learning model originally proposed by He Kaiming et al. in 2015. Its various variants have been widely used in the fields of video classification and behavior recognition.

[0019] 5. VAE model: Variational Autoencoders (VAE): is a generative model originally proposed by Kingma and Welling in 2013. It is a probabilistic model based on neural networks that aims to learn the potential representation of data and generate new data samples.

[0020] 6. Pulse neuron: Pulse neuron is a neuron model that is designed to be closer to the behavior of biological neurons. Compared with the neuron model used in traditional artificial neural networks, pulse neurons can process and generate temporally discrete signals.

[0021] 7. Information entropy: It is an important concept in information theory, used to measure the uncertainty or information content of random variables. It was proposed by Shannon in 1948 and is widely used in communications, data compression, cryptography and other fields.

[0022] 8. LSTM network, please explain and supplement; Long Short-Term Memory Network (LSTM): It is a variant of Recurrent Neural Network (RNN), originally proposed by Sepp Hochreiter and Jürgen Schmidhuber in 1997. Its various variants are widely used in time series prediction and natural language processing.

[0023] The technical solution of the present invention is as follows:

[0024] A micro-expression recognition method based on multi-temporal information entropy across time domains, comprising:

[0025] A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment;

[0026] B. Pick out the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference calculation on the peak frame and the starting frame to obtain the difference of facial key frames, and sample the micro-expression frame set adjacent to the peak frame.

[0027] C. Construct a residual network for modeling micro-expression motion characteristics in the long-interval time domain. Using the facial key frame differences obtained in step B, the subtlety of micro-expression changes is amplified in the long-term domain, and subtle and representative motion pattern features are extracted and modeled.

[0028] D. Construct an asymmetric autoencoder network that models continuous frames of micro-expressions in short intervals, and input a micro-expression frame set. Reconstruct peak frames, encode continuous and subtle data in extremely short time scales, and enhance its ability to encode time dimension information;

[0029] E. Construct a pulse time domain matrix module for joint screening of long and short time domains, map the features extracted from micro-expressions in the long time domain and the features extracted in the short time domain to a new time domain space, and perform unified feature modeling;

[0030] F. According to the adaptive calculation pulse threshold δ of the feature map mutual information, an adaptive function is constructed to adaptively generate new time domain pulse signals for micro-expression representations in long and short time domains under different samples and different network scales;

[0031] G. Model training, using cross entropy loss, peak frame reconstruction loss, KL divergence loss, and classification loss to iteratively update the residual network for modeling the long-interval time-domain micro-expression motion characteristics and the asymmetric autoencoder network for modeling the short-interval time-domain micro-expression continuous frames constructed in steps C and D;

[0032] H. The facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained to model the long-interval time domain micro-expression motion characteristics, and the input micro-expression frame set An asymmetric autoencoder network is used to model continuous frames of micro-expressions in the short-interval time domain to achieve micro-expression classification and recognition.

[0033] Preferably, according to the present invention, in step A, the micro-expression video is pre-processed, including:

[0034] 1) Obtain video frame sequence: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames and save them in chronological order;

[0035] 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames;

[0036] 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and a rectangular frame is used for segmentation.

[0037] Preferably according to the present invention, in step A, face correction is performed after face alignment.

[0038] According to the preferred embodiment of the present invention, in step B, the facial key frame difference Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is as shown in formula (1):

[0039]

[0040] At the same time, within a certain time range, the micro-expression frame set adjacent to the peak frame is periodically sampled. As shown in formula (2):

[0041]

[0042] In formula (2), x σ-△φ represents the peak frame x σ Before Frame, x σ+△φ represents the peak frame x σ After Frame, x σ-φ and x σ+φ Represents micro-expression frame sets The left and right borders of the midframe.

[0043] Preferably, in step C, a residual network for modeling micro-expression motion characteristics in the long-interval time domain is constructed, that is, a neural network is constructed with Resnet as the skeleton to extract micro-expression features in the long-time domain, and the peak frame x is compared. σ The difference between the frame x1 and the starting frame x1 highlights the subtle facial expressions. The difference of the facial key frames obtained in step B is used to amplify the subtlety of micro-expression changes in the long-term domain, and the subtle and representative motion pattern features are extracted and modeled; it is expressed as formula (3):

[0044]

[0045] Among them, the residual block R(·) is represented by the parameter Γ;

[0046] By magnifying the subtlety of micro-expression changes in the long-term domain, the subtle and representative movement pattern feature Y is extracted M , H, W, C represent the length, width, and number of channels of the current feature respectively, and N represents the number of residual blocks.

[0047] Preferably, in step D, an asymmetric autoencoder network for modeling continuous frames of short-interval time-domain micro-expressions is constructed, that is, a variational autoencoder is used to model continuous and short-term complex changes in a frame set, including an encoder and a decoder; the L-encoder, i.e., the basic unit of the encoder, includes an LSTM and a maximum pooling layer; the micro-expression frame set is input Modeling continuous and short-term complex changes between frames, reconstructing peak frames, encoding continuous and subtle data in extremely short time scales, and enhancing the ability to encode time dimension information;

[0048] In a variational autoencoder, is the input of the L-encoder, as shown in formula (4), is the reconstruction target of the decoder, as shown in formula (5);

[0049]

[0050] In formula (4), x σ-△φ represents the peak frame x σ Before Frame, x σ+△φ represents the peak frame x σ After Frame, x σ-φ and x σ+φ Represents micro-expression frame sets The left and right borders of the middle frame; in formula (5), x σ is the peak frame of the current micro-expression sequence. For each sampled micro-expression frame x σ±△φ , both expect to generate the peak frame x of the current micro-expression sequence σ ;

[0051] The process of L-encoder performing downsampling and aggregating features representing continuous time changes is shown in formulas (6) to (8):

[0052]

[0053] in, represents the encoder E with parameter φ, including LSTM and max pooling layers, and the input is a set of frames of continuous micro-expressions Y S represents the output of each layer of the encoder, Represents the features of the encoder's first layer output, with a size of H×W×C, Represents the features of the encoder's Nth layer output, with a size of Indicates that given input The approximate posterior distribution of the latent variable Z under the condition of , where the latent variable Z is the implicit representation learned from the input data by the encoder, represents the output of each layer after the L-encoder, Indicated in the parameter The mean of the latent variable Z under the condition, Indicated in the parameter The variance of the latent variable Z under the condition;

[0054] The decoder uses transposed convolution to perform upsampling operations; the decoder's reconstruction target is a set of consecutive input frames. The missing peak frame x σ , the whole process is shown in formula (9):

[0055] x=D Ψ (μ, log(σ 2 )) (9)

[0056] In formula (9), the decoding layer is composed of parameters ψ, expressed as D ψ ; x is a parameter The decoder of , μ is the mean of the latent variable, log(σ 2 ) is the log variance of the latent variable;

[0057] The proposed VAE model is optimized by the loss function L, as shown in formula (10):

[0058] L reap =L r +L kl +ηL cls (10)

[0059] Among them, the first term L r To reconstruct the peak frame set The reconstruction loss is , and the MSE loss is used for reconstruction, as shown in formula (11), where N is the number of frames, is the input frame of the i-th micro-expression sequence, is the peak frame of the i-th micro-expression sequence; the second term L kl is the Kullback-Leibler divergence, as shown in formula (12), which uses the reconstruction mean μ and the reconstruction variance σ 2 To smooth the constrained reconstruction process;

[0060]

[0061] L cls is the final classification loss and η is the scaling factor.

[0062] Preferably, according to the present invention, in step E, a pulse time domain matrix module for joint screening of long and short time domains is constructed, that is, the features of the long time domain and the short time domain are spliced ​​and input into the pulse neuron, the pulse neuron transmits information in the form of pulses, the state of each pulse neuron is represented by its membrane potential, and when the membrane potential exceeds a certain threshold, the neuron emits a pulse; wherein the pulse neuron is defined as formula (13) and formula (14):

[0063]

[0064] In formula (13) and formula (14), t and i represent the tth residual network constructed in step C for modeling the long-interval temporal micro-expression motion characteristics and the asymmetric autoencoder network constructed in step D for modeling the short-interval temporal micro-expression continuous frames. th time step and the i th layer; and Represents two time domains; represents the rate of change of input current I(t) with time t, τ syn represents the time constant, τ syn I(t) represents the decay speed of the input current I(t) controlled by the time constant. represents the weight between the i-th neuron and the j-th neuron in the n-th layer of the neural network, represents the rate of change of membrane potential V(t) with time t, τ mem Represents the time constant of the membrane potential, which is used to control the decay rate of the membrane potential; Formula (13) represents the characteristics of the two time domains through the weight Transform along time step t;

[0065] In formula (14): V(t) is the corresponding membrane potential, which is equal to the accumulation of different micro-expression features at different time steps; therefore, Represents the i-th th The pulse signal of the layer; when the membrane potential V(t) reaches the pulse threshold δ, the pulse neuron will output a pulse signal, that is, the pulse value 1 in formula (15). When it does not reach the threshold, the output is 0, which indicates that this position holds a relatively more critical micro-expression feature; formula (15) is as follows:

[0066]

[0067] Through the processing of the pulse time domain, the information in the two time domains will be expanded in a new time domain in the form of pulses; the size of the time domain is determined by the time step t of the pulse neuron;

[0068] Then, the time domain pulse matrix M TS(Temporal Spiking Matrix, TSM) is spliced ​​according to formula (16) to match the size of the original spliced ​​feature map, where M motion The value of the long-term information of micro-expression expanded in the new time domain, M set The value of the short-term information of micro-expression expanded in the new time domain; cat(·) represents the concatenation operation, and C represents the number of channels; TSM at this time represents the areas of the entire feature map that need to be highlighted in the two time domain spaces and the redundant areas that need to be ignored;

[0069]

[0070] Finally, the feature map Y from the long time domain M and the short-term feature map Y S After splicing, the time domain pulse matrix M TS Perform element-by-element multiplication to complete the selective integration of micro-expression features at different time scales, as shown in formula (17):

[0071] Y F =cat(Y M , Y S )·M TS (17)

[0072] Through the time domain pulse matrix M TS After filtering and protruding the features, the unified key solution vector is obtained And continue through the residual block R Γ Continue forward propagation to obtain the representation for the final micro-expression recognition, as shown in formula (18):

[0073]

[0074] Among them, Fc n Represents a fully connected layer, used to mix Y F channels and align the dimensions to the residual block R Γ , the last item As the basis of recognition, subject to the classification loss function (19), the final unified representation is The network is optimized with cross entropy loss, where Fc n represents the fully connected layer, represents the sum of all categories c, C represents the total number of categories, when the current sample belongs to the cth category, y true,c is 1, otherwise it is 0;

[0075]

[0076] An adaptive function is constructed by adaptively calculating the pulse threshold δ according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of long and short time domains under different samples and different network scales.

[0077] Preferably, according to the present invention, in step F, an adaptive function is constructed based on the pulse threshold δ adaptively calculated according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of the long and short time domains under different samples and different network scales; including:

[0078] First, the mutual information entropy of the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames is derived using formulas (20) and (21), which represent the amount of shared information of long-term and short-term distributions. Formula (20) represents the joint feature map at time point d obtained by discretizing (digitizing) these two feature maps at time point c in the continuous time domain; p(y S ,y M ) is the feature y extracted from the residual network modeling the long-interval temporal micro-expression motion characteristics constructed in step C s and the feature y extracted from the asymmetric autoencoder network constructed in step D to model continuous frames of short-interval temporal micro-expressions M The joint probability distribution of is, in formula (21), Indicates the calculation of mutual information entropy for the discretized feature map;

[0079]

[0080] Among them, the long-term spatial micro-expression features and the short-term spatial micro-expression features p(y S ) and p(y M ) are respectively the marginal probability distributions of the residual network features for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network features for modeling the short-interval time-domain micro-expression continuous frames constructed by step D;

[0081] From formula (21), if we want to calculate the mutual information between the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames, we need to first calculate p(y S ,y M ) and its s ,y M ;

[0082] For the joint probability distribution, it is necessary to construct a contingency table C[i,j], as shown in formula (22):

[0083] C[y S =i,y M =j]=count(y S =i∩y M =j) (22)

[0084] For two discrete variables y M and S ,y M and S It refers to the discretized long-term spatial micro-expression features and short-term spatial micro-expression features. The contingency table C[i,j] represents the variable y M In state i and variable y S The joint frequency of occurrence in state j;

[0085] According to this, the joint probability distribution p(y S ,y M ) and marginal probability distribution p(y S ), p(y M ) are obtained by formula (23) and formula (24) respectively:

[0086]

[0087] Where N is the total number of observations in the data;

[0088] Micro-expression features with the same shape at different time scales are distinguished based on the vector v, which is defined by formula (25) as follows:

[0089]

[0090] In formula (25), Indicates that the mutual information between the residual network feature map for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network feature map for modeling the short-interval time-domain micro-expression continuous frames constructed by step D is calculated at a discrete time step d;

[0091] Subsequently, based on the vector v, a variant of the Sigmoid function is used as a conversion function to convert the obtained mutual information entropy and obtain the adaptively variable threshold required by the spike neuron, as shown in formula (26):

[0092]

[0093] In formula (26), k represents the scaling factor, which is used to limit the adaptive threshold The upper and lower bounds of The change of will occur in the range [k,2k]; represents adaptive threshold, α and β are parameters used to control the threshold change rate and center point;

[0094] Further preferably, to analyze the influence of α, first, let the σ(x) function replace the Sigmoid function, as shown in formula (27):

[0095]

[0096] After that, the first-order derivative of the σ(x) function is taken, as shown in formula (28), and its derivative is:

[0097]

[0098] Finally, the first-order derivative of the σ(x) function σ(x|α,β) adjusted by α and β is expressed as shown in formula (29):

[0099]

[0100] According to formula (29), the derivative of the function σ(x|α,β) is proportional to α. Therefore, as α increases, the slope of the Sigmoid function at any point increases, causing the proposed adaptive threshold function to become steeper;

[0101] Analyze the influence of β on the adaptive threshold function; consider the adjusted Sigmoid function σ(x|α,β), whose value at x=β is shown in formula (30):

[0102] σ(α×(β-β))=σ(0)=0.5 (30)

[0103] It shows that β actually controls the phase shift of the adaptive threshold function; Equation (31) is the maximum value of the adaptive threshold function;

[0104]

[0105] By setting the adaptive threshold Substitute the threshold δ back into formula (14) to complete the mapping of cross-temporal micro-expression features in the new temporal space formed by spiking neurons. By repeating this process for each sample, the key distributions with different focus points are extracted using multivariate temporal space. Used for micro-expression recognition.

[0106] According to the preferred embodiment of the present invention, the facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained for modeling the long-interval time-domain micro-expression motion characteristics, and the micro-expression frame set is input. An asymmetric autoencoder network for modeling continuous frames of micro-expressions in short time intervals to achieve micro-expression classification and recognition, including:

[0107] First, the micro-expression video to be recognized is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation; at the same time, the micro-expression video is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation. by Periodically sampling a set of micro-expression frames adjacent to the peak frame;

[0108] Afterwards, the motion features are first extracted through the residual network modeled with the long-interval temporal micro-expression motion characteristics according to step C to obtain the long-interval temporal micro-expression motion feature map; then the micro-expression frame set is converted into Input the asymmetric variational autoencoder, first pass it through the L-encoder including the pooling layer and the LSTM network to obtain the short-term micro-expression motion feature map, then splice the two feature maps according to step E to map the two feature maps to the new time domain space, calculate the mutual information entropy of the two feature maps according to step F to obtain the threshold for each position on the spliced ​​feature map, and then substitute the threshold back into step E to obtain the time domain pulse matrix, multiply the time domain pulse matrix and the spliced ​​feature map to obtain the final feature, and send it to the residual block for micro-expression recognition and classification.

[0109] The beneficial effects of the present invention are:

[0110] 1. In order to overcome the problem that the instantaneous characteristics of micro-expressions are difficult to capture, the present invention proposes an asymmetric variational autoencoder to perform unsupervised modeling of the short-term feature distribution of micro-expressions in a generative manner.

[0111] 2. In order to ensure the effectiveness of feature extraction by the model, the present invention proposes a time domain pulse matrix, uses pulse neurons to construct a new time domain space, and fuses and screens micro-expression features of two different tenses.

[0112] 3. To increase the model's ability to adaptively select features of different data sets, different samples, and different scales, the present invention designs an adaptive threshold function, which adopts an activation threshold based on the mutual information entropy of the feature graph for the proposed time-domain pulse matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 Schematic diagram of 68 key feature points of the face of the present invention;

[0114] Figure 2 A schematic diagram of a process of a micro-expression recognition method based on multi-temporal information entropy across time domains according to the present invention;

[0115] Figure 3 Schematic diagram of a residual network for modeling long-interval temporal micro-expression motion characteristics;

[0116] Figure 4 Schematic diagram of micro-expression movement characteristics in long-interval time domain and short-interval time domain;

[0117] Figure 5 Schematic diagram of an asymmetric autoencoder network for modeling consecutive frames of short-interval temporal micro-expressions;

[0118] Figure 6 Schematic diagram of transposed convolution;

[0119] Figure 7 It is a schematic diagram of the time domain pulse matrix based on the binary time domain mutual information;

[0120] Figure 8 Experimental results analysis 1-Schematic diagram of confusion matrix of CASMEII dataset;

[0121] Fig. 9 Experimental results analysis 2-Schematic diagram of the confusion matrix of the SAMM dataset;

[0122] Fig.10 Experimental results analysis 3-Schematic diagram of the confusion matrix of the SMIC dataset. DETAILED DESCRIPTION

[0123] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0124] Example 1

[0125] A micro-expression recognition method based on multi-temporal information entropy across time domains, such as Figure 2 As shown, including:

[0126] A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment;

[0127] B. Pick out the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference calculation on the peak frame and the starting frame to obtain the difference of facial key frames. by Periodically sample the micro-expression frames adjacent to the peak frame

[0128] C. Construct a residual network for modeling micro-expression motion characteristics in the long-interval time domain. Using the facial key frame differences obtained in step B, the subtlety of micro-expression changes is amplified in the long-term domain, and subtle and representative motion pattern features are extracted and modeled.

[0129] D. Construct an asymmetric autoencoder network that models continuous frames of micro-expressions in short intervals, and input a micro-expression frame set. Modeling continuous and short-term complex changes between frames, reconstructing peak frames, encoding continuous and subtle data in extremely short time scales, and enhancing its ability to encode time dimension information;

[0130] E. Construct a pulse time domain matrix module for joint screening of long and short time domains, map the features extracted from micro-expressions in the long time domain and the features extracted in the short time domain to the new time domain space, and then perform unified feature modeling for the final recognition task;

[0131] F. According to the adaptive calculation pulse threshold δ of the feature map mutual information, an adaptive function is constructed to adaptively generate new time domain pulse signals for micro-expression representations in long and short time domains under different samples and different network scales;

[0132] G. Model training, using cross entropy loss, peak frame reconstruction loss, KL divergence loss, and classification loss to iteratively update the residual network for modeling the long-interval time-domain micro-expression motion characteristics and the asymmetric autoencoder network for modeling the short-interval time-domain micro-expression continuous frames constructed in steps C and D;

[0133] H. The facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained to model the long-interval time domain micro-expression motion characteristics, and the input micro-expression frame set An asymmetric autoencoder network is used to model continuous frames of micro-expressions in the short-interval time domain to achieve micro-expression classification and recognition.

[0134] Example 2

[0135] The difference between the micro-expression recognition method based on multi-temporal information entropy and cross-time domain described in Example 1 is that:

[0136] In step A, the micro-expression video is preprocessed, including:

[0137] 1) Obtain video frame sequence: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames and save them in chronological order;

[0138] 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames;

[0139] 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and a rectangular frame is used for segmentation.

[0140] In step A, face alignment is followed by face correction. This includes: adjusting certain key feature points among the 68 key feature points detected according to the characteristics of the selected micro-expression data set (including race, gender, shooting angle, etc.). For example, a line can be drawn between the key feature point 37 marking the left corner of the left eye and the key feature point 46 marking the right corner of the right eye. This line may have an angle α with the standard horizontal line. Through this angle, the segmented face can be rotated so that the line between the key feature points 37 and 46 of the face is parallel to the standard horizontal line, thereby achieving calibration of the face posture.

[0141] In step B, the facial keyframe differences Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is shown in formula (1):

[0142]

[0143] At the same time, within a certain time range, the micro-expression frame set adjacent to the peak frame is periodically sampled. As shown in formula (2):

[0144]

[0145] In formula (2), represents the peak frame x σ Before frame, represents the peak frame x σ After frame, and Represents micro-expression frame sets Left and right borders of the middle frame (micro-expression frame set In a certain time frame by periodically sampled).

[0146] In step C, a residual network for modeling micro-expression motion characteristics in the long interval time domain is constructed, that is, a neural network is constructed with Resnet as the skeleton to extract micro-expression features in the long time domain, and the peak frame x is compared. σ The difference between the frame x1 and the starting frame x1 highlights the subtle facial expressions. The difference of the facial key frames obtained in step B is used to amplify the subtlety of micro-expression changes in the long-term domain, and the subtle and representative motion pattern features are extracted and modeled; it is expressed as formula (3):

[0147]

[0148] Among them, the residual block R(·) is represented by the parameter Γ;

[0149] By magnifying the subtlety of micro-expression changes in the long-term domain, the subtle and representative movement pattern feature Y is extracted M , H, W, C represent the length, width, and number of channels of the current feature respectively, and N represents the number of residual blocks.

[0150] In step D, compared with the motion pattern information between the start frame and the peak frame, the motion pattern between two adjacent frames is not significant, such as Figure 4 As shown in , there is almost no effective information representation, and it is difficult to become an effective representation. Figure 5 As shown in the figure, an asymmetric autoencoder network is constructed to model continuous frames of micro-expressions in short intervals, that is, a variational autoencoder (VAE) is used to model continuous and short-term complex changes in a frame set, including an encoder and a decoder; it is worth noting that since the task is to extract subtle key representations between consecutive frames, VAE should pay more attention to the feature distribution constructed in the encoder, and the quality of the image generated by the decoder does not directly affect the classification result. In view of this feature, the structure of the encoder is redesigned. The L-encoder, that is, the basic unit of the encoder, includes LSTM and a maximum pooling layer (MaxPooling) to enhance its encoding ability for time dimension information. The structure of the decoder is consistent with the decoder (multiple convolutional layers) in the traditional VAE. Input a set of micro-expression frames Modeling continuous and short-term complex changes between frames, reconstructing peak frames, encoding continuous and subtle data in extremely short time scales, and enhancing the ability to encode time dimension information;

[0151] In a variational autoencoder, is the input of the L-encoder, as shown in formula (4), is the reconstruction target of the decoder, as shown in formula (5);

[0152]

[0153] In formula (4), x σ-△φ represents the peak frame x σ Before Frame, x σ+△φ represents the peak frame x σ After Frame, x σ-φ and x σ+φ Represents micro-expression frame sets The left and right borders of the midframe ( In a certain time frame by Periodically sampled); in formula (5), x σ is the peak frame of the current micro-expression sequence. For each sampled micro-expression frame xσ±△φ , both expect to generate the peak frame x of the current micro-expression sequence σ ;

[0154] The micro-expression frame set excludes the consecutive frames containing the peak frames with the largest micro-expression information, and by Periodically sample frames adjacent to the peak frame. The frame set only contains the peak frame and its replica. The goal of the network is to reconstruct the peak frame so that the encoder can encode continuous and subtle data at a very short time scale. The process of L-encoder performing downsampling and aggregating features representing continuous time changes is shown in formulas (6) to (8):

[0155]

[0156]

[0157] in, represents the encoder E with parameter φ, including LSTM and max pooling layers, and the input is a set of frames of continuous micro-expressions Y S Represents the output of each layer of the encoder, Y1 S;(H,W,C) Represents the features of the encoder's first layer output, with a size of H×W×C, Represents the features of the encoder's Nth layer output, with a size of Indicates that given input The approximate posterior distribution of the latent variable Z under the condition of , where the latent variable Z is the implicit representation learned from the input data by the encoder, represents the output of each layer after the L-encoder, Indicated in the parameter The mean of the latent variable Z under the condition, Indicated in the parameter The variance of the latent variable Z under the condition;

[0158] The construction of the decoder follows the construction method in the traditional VAE model, such as Figure 6 As shown, transposed convolution is used for upsampling; the reconstruction target of the decoder is a set of continuous input frames. The missing peak frame x σ , the whole process is shown in formula (9):

[0159] χ=D ψ (μ, log(σ 2 )) (9)

[0160] In formula (9), the decoding layer is composed of parameters ψ, expressed as D ψ ; x is a parameter The decoder of , μ is the mean of the latent variable, log(σ 2 ) is the log variance of the latent variable;

[0161] The proposed VAE model is optimized by the loss function L, as shown in formula (10):

[0162] L reap =L r +L kl +ηL cls (10)

[0163] Among them, the first term L r To reconstruct the peak frame set The reconstruction loss is , and the MSE loss is used for reconstruction, as shown in formula (11), where N is the number of frames, is the input frame of the i-th micro-expression sequence, is the peak frame of the i-th micro-expression sequence; the second term L kl is the Kullback-Leibler divergence, as shown in formula (12), which uses the reconstruction mean μ and the reconstruction variance σ 2 To smooth the constrained reconstruction process;

[0164]

[0165]

[0166] If the loss function L reap As shown in Figure 2, considering that the ultimate goal of the algorithm is the micro-expression recognition task, a third term L is added to the loss function of the VAE model. cls and a scaling factor η,L cls is the final classification loss, and η is the scaling factor. The purpose is to add certain constraints to the model to serve the classification task, rather than taking reconstruction and generation as the main goals.

[0167] In step E, a pulse time domain matrix module is constructed for joint screening of long and short time domains, that is, the features of the long time domain and the short time domain are spliced ​​and input into the pulse neuron. The pulse neuron transmits information in the form of pulses. The state of each pulse neuron is represented by its membrane potential. When the membrane potential exceeds a certain threshold, the neuron emits a pulse. The pulse neuron is defined as formula (13) and formula (14):

[0168]

[0169] In formula (13) and formula (14), t and i represent the tth residual network constructed in step C for modeling the long-interval temporal micro-expression motion characteristics and the asymmetric autoencoder network constructed in step D for modeling the short-interval temporal micro-expression continuous frames. th time step and the i th layer; and Represents two time domains; represents the rate of change of input current I(t) with time t, τ syn represents the time constant, τ syn I(t) represents the decay speed of the input current I(t) controlled by the time constant. represents the weight between the i-th neuron and the j-th neuron in the n-th layer of the neural network, represents the rate of change of membrane potential V(t) with time t, τ mem Represents the time constant of the membrane potential, which is used to control the decay rate of the membrane potential; Formula (13) represents the characteristics of the two time domains through the weight Transform along time step t;

[0170] In formula (14): V(t) is the corresponding membrane potential, which is equal to the accumulation of different micro-expression features at different time steps; therefore, Represents the i-th th The pulse signal of the layer; when the membrane potential V(t) reaches the pulse threshold δ, the pulse neuron will output a pulse signal, that is, the pulse value 1 in formula (15). When it does not reach the threshold, the output is 0, which indicates that this position holds a relatively more critical micro-expression feature; formula (15) is as follows:

[0171]

[0172] Through the processing of the pulse time domain, the information in the two time domains will be expanded in a new time domain in the form of pulses; the size of the time domain is determined by the time step t of the pulse neuron;

[0173] Then, the time domain pulse matrix M TS (Temporal Spiking Matrix, TSM) is spliced ​​according to formula (16) to match the size of the original spliced ​​feature map, where M motion The value of the long-term information of micro-expression expanded in the new time domain, M set The value of the short-term information of micro-expression expanded in the new time domain; cat(·) represents the concatenation operation, and C represents the number of channels; TSM at this time represents the areas of the entire feature map that need to be highlighted in the two time domain spaces and the redundant areas that need to be ignored;

[0174]

[0175] Finally, the feature map Y from the long time domain M and the short-term feature map Y S After splicing, the time domain pulse matrix M TS Perform element-by-element multiplication to complete the selective integration of micro-expression features at different time scales, as shown in formula (17):

[0176] Y F =cat(Y M , Y S )·M TS (17)

[0177] Through the time domain pulse matrix M TS After filtering and protruding the features, the unified key solution vector is obtained And continue through the residual block R Γ Continue forward propagation to obtain the representation for the final micro-expression recognition, as shown in formula (18):

[0178]

[0179] Among them, Fc n Represents a fully connected layer, used to mix Y F channels and align the dimensions to the residual block R Γ , the last item As the basis of recognition, subject to the classification loss function (19), the final unified representation is The network is optimized with cross entropy loss, where Fc n represents the fully connected layer, represents the sum of all categories c, C represents the total number of categories, when the current sample belongs to the cth category, y true,c is 1, otherwise it is 0;

[0180]

[0181] An adaptive function is constructed by adaptively calculating the pulse threshold δ according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of long and short time domains under different samples and different network scales.

[0182] How to set the threshold δ is crucial for the time domain pulse matrix. In step F, the pulse threshold δ is adaptively calculated based on the mutual information of the feature map to construct an adaptive function, which adaptively generates a new time domain pulse signal for the micro-expression representation of long and short time domains under different samples and different network scales. The generation process of the adaptive threshold function is as follows: Figure 7 shown.

[0183] In step F, an adaptive function is constructed based on the pulse threshold δ adaptively calculated according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of long and short time domains under different samples and different network scales; including:

[0184] First, the mutual information entropy of the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames is derived using formulas (20) and (21), which represent the amount of shared information of long-term and short-term distributions. Formula (20) represents the joint feature map at time point d obtained by discretizing (digitizing) these two feature maps at time point c in the continuous time domain; p(y S ,y M ) is the feature y extracted from the residual network modeling the long-interval temporal micro-expression motion characteristics constructed in step C s and the feature y extracted from the asymmetric autoencoder network constructed in step D to model continuous frames of short-interval temporal micro-expressions M The joint probability distribution of is, in formula (21), Indicates the calculation of mutual information entropy for the discretized feature map;

[0185]

[0186] Among them, the long-term spatial micro-expression features and the short-term spatial micro-expression features p(y S ) and p(y M ) are respectively the marginal probability distributions of the residual network features for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network features for modeling the short-interval time-domain micro-expression continuous frames constructed by step D;

[0187] From formula (21), if we want to calculate the mutual information between the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames, we need to first calculate p(y S ,y M ) and its s ,y M ;

[0188] For the joint probability distribution, it is necessary to construct a contingency table C[i,j], as shown in formula (22):

[0189] C[y S =i,yM=j]=count(y S =i∩y M =j) (22)

[0190] For two discrete variables y M and S ,y M and S It refers to the discretized long-term spatial micro-expression features and short-term spatial micro-expression features. The contingency table C[i,j] represents the variable y M In state i and variable y S The joint frequency of occurrence in state j;

[0191] According to this, the joint probability distribution p(y S ,y M ) and marginal probability distribution p(y S ), p(y M ) are obtained by formula (23) and formula (24) respectively:

[0192]

[0193] where N is the total number of observations in the data (i.e., the sum of all counts in the matrix C[·]);

[0194] It is worth noting that represents the diagonal elements in formula (21), which means The shared information of micro-expression features on the same network layer in different time domains is quantified, which indicates the degree of information overlap in specific locations and whether they capture representative or independent micro-expression representations. Therefore, micro-expression features with the same shape at different time scales are distinguished based on the vector v, which is defined by formula (25) as follows:

[0195]

[0196] In formula (25), Indicates that the mutual information between the residual network feature map for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network feature map for modeling the short-interval time-domain micro-expression continuous frames constructed by step D is calculated at a discrete time step d;

[0197] Subsequently, based on the vector v, a variant of the Sigmoid function is used as a conversion function to convert the obtained mutual information entropy and obtain the adaptively variable threshold required by the spike neuron, as shown in formula (26):

[0198]

[0199] In formula (26), k represents the scaling factor, which is used to limit the adaptive threshold The upper and lower bounds of The change of will occur in the range [k,2k]; represents the adaptive threshold, α and β are parameters used to control the threshold change rate and center point; the experimental part will analyze the final impact of different α and β values ​​on the recognition task. Here is a theoretical analysis of the impact of α and β on the actual threshold.

[0200] To analyze the impact of α, first, replace the Sigmoid function with the σ(x) function, as shown in formula (27):

[0201]

[0202] After that, the first-order derivative of the σ(x) function is taken, as shown in formula (28), and its derivative is:

[0203]

[0204] Finally, the first-order derivative of the σ(x) function σ(x|α,β) adjusted by α and β is expressed as shown in formula (29):

[0205]

[0206] According to formula (29), the derivative of the function σ(x|α,β) is proportional to α. Therefore, as α increases, the slope of the Sigmoid function at any point increases, causing the proposed adaptive threshold function to become steeper;

[0207] Analyze the influence of β on the adaptive threshold function; consider the adjusted Sigmoid function σ(x|α,β), whose value at x=β is shown in formula (30):

[0208] σ(α×(β-β))=σ(0)=0.5 (30)

[0209] This means that when the input x is equal to β, the output of the Sigmoid function is exactly k / 2 (here 0.5). As x increases or decreases, the output of the Sigmoid function will increase or decrease accordingly, indicating that β actually controls the phase shift of the adaptive threshold function (that is, the output is the position of the k / 2 point); Equation (31) is the maximum value of the adaptive threshold function;

[0210]

[0211] When the micro-expression features of the two time domains produce lower mutual information entropy, the sigmoid output is close to 0, and the threshold Set it close to k. When a higher entropy is generated, the output is close to 1, and the threshold Close to 2k. When the representations in the two time domains of the input cannot exceed this threshold in the given pulse time domain, the features at the corresponding positions are considered redundant. The larger the entropy, the greater the amount of shared information. The higher it is, the more information is filtered out.

[0212] By setting the adaptive threshold Substitute the threshold δ back into formula (14) to complete the mapping of cross-temporal micro-expression features in the new temporal space formed by spiking neurons. By repeating this process for each sample, the key distributions with different focus points are extracted using multivariate temporal space. Used for micro-expression recognition.

[0213] The residual network for modeling the long-interval time-domain micro-expression motion characteristics constructed in step C is optimized using cross entropy loss, and the asymmetric autoencoder network for modeling the short-interval time-domain micro-expression continuous frames constructed in step D is optimized using peak frame reconstruction loss, KL divergence loss, and classification loss. The parameters of the two networks are continuously updated through training, and the residual network features of the long-interval time-domain micro-expression motion characteristics obtained in step C and the asymmetric autoencoder network features for modeling the short-interval time-domain micro-expression continuous frames obtained in step D are updated. Then, the two feature maps are spliced ​​according to step E to map the two feature maps to a new time domain space, and the mutual information entropy values ​​of the two feature maps are calculated according to step F to obtain the threshold for each position on the spliced ​​feature map, and then the threshold is substituted back into step E to obtain a time domain pulse matrix, and the time domain pulse matrix and the spliced ​​feature map are multiplied to obtain the final feature, which is sent to the residual block for final recognition; this process is repeated multiple times to reach the number of training times, and the network is continuously optimized to obtain the optimal model.

[0214] The facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained to model the long-interval time-domain micro-expression motion characteristics, and the input micro-expression frame set An asymmetric autoencoder network for modeling continuous frames of micro-expressions in short time intervals to achieve micro-expression classification and recognition, including:

[0215] First, the micro-expression video to be recognized is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation; at the same time, the micro-expression video is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation. by Periodically sample the micro-expression frames adjacent to the peak frame

[0216] Afterwards, the motion features are first extracted through the residual network modeled with the long-interval temporal micro-expression motion characteristics according to step C to obtain the long-interval temporal micro-expression motion feature map; then the micro-expression frame set is converted into Input the asymmetric variational autoencoder, first pass it through the L-encoder including the pooling layer and the LSTM network to obtain the short-term micro-expression motion feature map, then splice the two feature maps according to step E to map the two feature maps to the new time domain space, calculate the mutual information entropy of the two feature maps according to step F to obtain the threshold for each position on the spliced ​​feature map, and then substitute the threshold back into step E to obtain the time domain pulse matrix, multiply the time domain pulse matrix and the spliced ​​feature map to obtain the final feature, and send it to the residual block for micro-expression recognition and classification.

[0217] In this embodiment, the original videos of CASMEⅡ micro-expression database released by Fu Xiaolan's team of Institute of Psychology, Chinese Academy of Sciences, SAMM database released by Davison's team of Manchester Metropolitan University, UK, and SMIC database released by the research team of University of Oulu, Finland were tested for micro-expression recognition. While comparing with the classic micro-expression recognition algorithm, it was also compared with the SOTA model proposed in recent years. Table 1 shows the test results.

[0218] Table 1

[0219]

[0220] In Table 1, Pre-P: preprocessing method; E: Euler image enlargement; R: region of interest; T: time domain normalization; Pre-T: pre-training dataset.

[0221] In Table 1, the LBP-TOP method comes from the literature Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions [J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29 (6): 915-928.

[0222] FDM comes from the literature Xu F, Zhang J, Wang J Z. Micro-expression identification and categorization using a facial dynamics map [J]. IEEE Transactions on AffectiveComputing, 2017, 8(2): 254-267.

[0223] MDMO is from the literature: Liu Y J, Zhang J K, Yan W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2015, 7(4): 299 - 310.

[0224] TSCNN is from the literature: Song B, Li K, Zong Y, et al. Recognizing spontaneous micro-expression using a three-stream convolutional neural network[J]. IEEE Access, 2019, 7: 184537 - 184551.

[0225] STSTNet is from the literature: Liong S T, Gan Y S, See J, et al. Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition[C] / / 2019 14th IEEE international conference on automatic face&gesture recognition (FG 2019). IEEE, 2019: 1 - 5.

[0226] DIKD is from the literature: Sun B, Cao S, Li D, et al. Dynamic micro-expression recognition using knowledge distillation[J]. IEEE Transactions on Affective Computing, 2020, 13(2): 1037 - 1043.

[0227] DSTAN is from the literature "Wang Y, Huang Y, Liu C, et al. Micro expression recognition via dual-stream spatiotemporal attention network[J]. Journal of Healthcare Engineering, 2021, 2021."

[0228] MiMaNet is from the literature "Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30."

[0229] GEME is from the literature "Nie X, Takalkar M A, Duan M, et al. GEME: Dual-stream multi-task GEnder-based micro-expression recognition[J]. Neurocomputing, 2021, 427:13 - 28."

[0230] AMAN is from the literature "Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[C] / / International Conference on Learning Representations. 2020."

[0231] ME-PLAN is from the literature "Zhao S, Tang H, Liu S, et al. ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition[J]. Neural Networks, 2022, 153:427 - 443."

[0232] SelfME comes from the literature Maass W.Networks of spiking neurons: the third generation of neural network models[J].Neural networks,1997,10(9):1659-1671.

[0233] u-BERT comes from the literature Liu YH, Wang

[0234] BDCNN comes from the literature Khor HQ, See J, Phan RCW, et al. Enriched long-termrecurrent convolutional network for facial micro-expression recognition [C] / / 2018 13th IEEE international conference on automatic face&gesture recognition (FG 2018). IEEE, 2018: 667-674.

[0235] According to Table 1, taking the CASMEII dataset as an example, the proposed algorithm achieved a recognition accuracy of 94.9%, an F1-Score of 92.9%, and a UAR of 92.9%. Compared with the u-BERT algorithm, which currently has the best performance, the proposed algorithm has improved by about 2% in all available indicators. It is also worth noting that the proposed algorithm does not perform any preprocessing or model pre-training on the input frames. Most of the existing leading algorithms are pre-trained on large image datasets. Compared with the current SOTA research BDCNN, which also does not require any feature preprocessing or model pre-training, the proposed algorithm leads it by more than 5% in all indicators. In order to fully illustrate the effectiveness of the proposed method, this experiment plotted the confusion matrix of the proposed model in the three-category task in the CASME II, SAMM, and SMIC datasets, as shown below. Figure 8 , Fig. 9 , Fig.10As shown. In the three-classification task, the proposed cross-temporal micro-expression recognition model achieved excellent recognition results on all three datasets. The classification accuracy of all micro-expressions in the three datasets used exceeded 50%, and the recognition accuracy of positive micro-expressions reached 84.8% and 88.2% on the CASME II and SMIC datasets, respectively. It is worth noting that even when recognizing micro-expressions with a shortage of training samples, such as the surprise expression of CASME II (only 25 sample video clips), a recognition accuracy of 100% was still achieved. And the recognition accuracy of negative micro-expressions with a large number of samples in the three datasets exceeded 85% on all three datasets.

Claims

1. A micro-expression recognition method based on multi-temporal information entropy across time domains, characterized in that: include: A. Preprocess the micro-expression video, including: obtaining video frame sequence, face detection and positioning, and face alignment; B. Pick out the peak frame and the starting frame from the video sequence preprocessed in step A, and perform pixel difference calculation on the peak frame and the starting frame to obtain the difference of facial key frames, and sample the micro-expression frame set adjacent to the peak frame. C. Construct a residual network for modeling micro-expression motion characteristics in the long-interval time domain. Use the facial key frame differences obtained in step B to amplify the subtlety of micro-expression changes in the long-term domain, extract subtle and representative motion pattern features, and model them. D. Construct an asymmetric autoencoder network that models continuous frames of micro-expressions in short intervals, and input a micro-expression frame set. Reconstruct peak frames, encode continuous and subtle data in extremely short time scales, and enhance its ability to encode time dimension information; E. Construct a pulse time domain matrix module for joint screening of long and short time domains, map the features extracted from micro-expressions in the long time domain and the features extracted in the short time domain to a new time domain space, and perform unified feature modeling; F. According to the adaptive calculation pulse threshold δ of the feature map mutual information, an adaptive function is constructed to adaptively generate new time domain pulse signals for micro-expression representations in long and short time domains under different samples and different network scales; G. Model training, using cross entropy loss, peak frame reconstruction loss, KL divergence loss, and classification loss to iteratively update the residual network for modeling the long-interval time-domain micro-expression motion characteristics and the asymmetric autoencoder network for modeling the short-interval time-domain micro-expression continuous frames constructed in steps C and D; H. The facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained to model the long-interval time domain micro-expression motion characteristics, and the input micro-expression frame set An asymmetric autoencoder network is used to model continuous frames of micro-expressions in the short-interval time domain to achieve micro-expression classification and recognition.

2. According to claim 1, a method for micro-expression recognition based on multi-temporal information entropy across time domains is characterized in that: In step A, the micro-expression video is pre-processed, including: 1) Obtain video frame sequence: Perform frame processing on the video samples in the micro-expression dataset to obtain continuous micro-expression video frames and save them in chronological order; 2) Face detection and positioning: Use the Dlib visual library to detect and locate the face in the separated micro-expression video frames, and obtain the distance between the face and the image boundary in the detected micro-expression video frames; 3) Face alignment: Using the distance between the face and the image boundary in the micro-expression video frame obtained in step 2), the Dlib visual library is used to determine the 68 key feature points of the face in the micro-expression video frame, and a rectangular frame is used for segmentation.

3. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step A, face alignment is followed by face correction.

4. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step B, the facial keyframe differences Defined as the peak frame X σ The difference in pixels between the starting frame X1 and the starting frame X1 is as shown in formula (1): At the same time, within a certain time range, the micro-expression frame set adjacent to the peak frame is periodically sampled. As shown in formula (2): In formula (2), represents the peak frame x σ Before frame, represents the peak frame x σ After frame, and Represents micro-expression frame sets The left and right borders of the midframe.

5. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step C, a residual network is constructed to model the long-interval time domain micro-expression motion characteristics, that is, a neural network is constructed with Resnet as the skeleton to extract the micro-expression features in the long-time domain, and the peak frame x is compared. σ The difference between the starting frame x1 and the starting frame x1 highlights the subtle facial expressions. The difference between the facial key frames obtained in step B is used to amplify the subtlety of micro-expression changes in the long-term domain, and the subtle and representative motion pattern features are extracted and modeled; It is expressed as formula (3): Among them, the residual block R(·) is represented by the parameter Γ; By magnifying the subtlety of micro-expression changes in the long-term domain, the subtle and representative movement pattern feature Y is extracted M , H, W, C represent the length, width, and number of channels of the current feature respectively, and N represents the number of residual blocks.

6. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step D, an asymmetric autoencoder network is constructed to model continuous frames of micro-expressions in short intervals, that is, a variational autoencoder is used to model continuous and short-term complex changes in a frame set, including an encoder and a decoder; the L-encoder, i.e., the basic unit of the encoder, includes an LSTM and a maximum pooling layer; the micro-expression frame set is input Modeling continuous and short-term complex changes between frames, reconstructing peak frames, encoding continuous and subtle data in extremely short time scales, and enhancing the ability to encode time dimension information; In a variational autoencoder, is the input of the L-encoder, as shown in formula (4), is the reconstruction target of the decoder, as shown in formula (5); In formula (4), represents the peak frame x σ Before frame, represents the peak frame x σ After frame, and Represents micro-expression frame sets The left and right borders of the middle frame; in formula (5), x σ is the peak frame of the current micro-expression sequence. For each sampled micro-expression frame All of them expect to generate the peak frame x of the current micro-expression sequence σ ; The process of L-encoder performing downsampling and aggregating features representing continuous time changes is shown in formulas (6) to (8): in, represents the encoder E with parameter φ, including LSTM and max pooling layers, and the input is a set of frames of continuous micro-expressions Y S Represents the output of each layer of the encoder, Y1 S;(H,W,C) Represents the features of the encoder's first layer output, with a size of H×W×C, Represents the features of the encoder's Nth layer output, with a size of Indicates that given input The approximate posterior distribution of the latent variable Z under the condition of , where the latent variable Z is the implicit representation learned from the input data by the encoder, represents the output of each layer after the L-encoder, Indicated in the parameter The mean of the latent variable Z under the condition, Indicated in the parameter The variance of the latent variable Z under the condition; The decoder uses transposed convolution to perform upsampling operations; the decoder's reconstruction target is a set of consecutive input frames. The missing peak frame x σ , the whole process is shown in formula (9): x=D Ψ (μ, log(σ) 2 )) (9) In formula (9), the decoding layer is composed of parameters ψ, expressed as D ψ ; χ is a parameter The decoder of , μ is the mean of the latent variable, log(σ 2 ) is the log variance of the latent variable; The proposed VAE model is optimized by the loss function L, as shown in formula (10): L reap =L r +L kl +ηL cls (10) Among them, the first term L r To reconstruct the peak frame set The reconstruction loss is , and the MSE loss is used for reconstruction, as shown in formula (11), where N is the number of frames, is the input frame of the i-th micro-expression sequence, is the peak frame of the i-th micro-expression sequence; the second term L kl is the Kullback-Leibler divergence, as shown in formula (12), which uses the reconstruction mean μ and the reconstruction variance σ 2 To smooth the constrained reconstruction process; L cls is the final classification loss and η is the scaling factor.

7. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step E, a pulse time domain matrix module is constructed for joint screening of long and short time domains, that is, the features of the long time domain and the short time domain are spliced ​​and input into the pulse neuron. The pulse neuron transmits information in the form of pulses. The state of each pulse neuron is represented by its membrane potential. When the membrane potential exceeds a certain threshold, the neuron emits a pulse. The pulse neuron is defined as formula (13) and formula (14): In formula (13) and formula (14), t and i represent the tth residual network constructed in step C for modeling the long-interval temporal micro-expression motion characteristics and the asymmetric autoencoder network constructed in step D for modeling the short-interval temporal micro-expression continuous frames. th time step and the i th layer; and Represents two time domains; represents the rate of change of input current I(t) with time t, τ syn represents the time constant, τ syn I(t) represents the decay speed of the input current I(t) controlled by the time constant. represents the weight between the i-th neuron and the j-th neuron in the n-th layer of the neural network, represents the rate of change of membrane potential V(t) with time t, τ mem Represents the time constant of the membrane potential, which is used to control the decay rate of the membrane potential; Formula (13) represents the characteristics of the two time domains through the weight Transform along time step t; In formula (14): V(t) is the corresponding membrane potential, which is equal to the accumulation of different micro-expression features at different time steps; therefore, Represents the i-th th The pulse signal of the layer; when the membrane potential V(t) reaches the pulse threshold δ, the pulse neuron will output a pulse signal, that is, the pulse value 1 in formula (15). When it does not reach the threshold, the output is 0, which indicates that this position holds a relatively more critical micro-expression feature; formula (15) is as follows: Through the processing of the pulse time domain, the information in the two time domains will be expanded in a new time domain in the form of pulses; the size of the time domain is determined by the time step t of the pulse neuron; Then, the time domain pulse matrix M TS (Temporal Spiking Matrix, TSM) is spliced ​​according to formula (16) to match the size of the original spliced ​​feature map, where M motion The value of the long-term information of micro-expression expanded in the new time domain, M set The value of the short-term information of micro-expression expanded in the new time domain; cat(·) represents the concatenation operation, and C represents the number of channels; TSM at this time represents the areas of the entire feature map that need to be highlighted in the two time domain spaces and the redundant areas that need to be ignored; Finally, the feature map Y from the long time domain M and the short-term feature map Y S After splicing, the time domain pulse matrix M TS Perform element-by-element multiplication to complete the selective integration of micro-expression features at different time scales, as shown in formula (17): Y F =cat(Y M ,Y S )·M TS (17) Through the time domain pulse matrix M TS After filtering and protruding the features, the unified key solution vector is obtained And continue through the residual block R Γ Continue forward propagation to obtain the representation for the final micro-expression recognition, as shown in formula (18): Among them, Fc n Represents a fully connected layer, used to mix Y F channels and align the dimensions to the residual block R Γ , the last item As the basis of recognition, subject to the classification loss function (19), the final unified representation is The network is optimized with cross entropy loss, where Fc n represents the fully connected layer, represents the sum of all categories c, C represents the total number of categories, when the current sample belongs to the cth category, y true,c is 1, otherwise it is 0; An adaptive function is constructed by adaptively calculating the pulse threshold δ according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of long and short time domains under different samples and different network scales.

8. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 1, characterized in that: In step F, an adaptive function is constructed based on the pulse threshold δ adaptively calculated according to the mutual information of the feature graph, and a new time domain pulse signal is adaptively generated for the micro-expression representation of long and short time domains under different samples and different network scales; including: First, the mutual information entropy of the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames is derived using formulas (20) and (21), which represent the amount of shared information between long-term and short-term distributions. Formula (20) represents the joint feature map at time point d obtained by discretizing these two feature maps at time point c in the continuous time domain; p(y S ,y M ) is the feature y extracted from the residual network modeling the long-interval temporal micro-expression motion characteristics constructed in step C s and the feature y extracted from the asymmetric autoencoder network constructed in step D to model continuous frames of short-interval temporal micro-expressions M The joint probability distribution of is, in formula (21), Indicates the calculation of mutual information entropy for the discretized feature map; Among them, the long-term spatial micro-expression features and the short-term spatial micro-expression features p(y S ) and p(y M ) are respectively the marginal probability distributions of the residual network features for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network features for modeling the short-interval time-domain micro-expression continuous frames constructed by step D; From formula (21), if we want to calculate the mutual information between the residual network feature map constructed by step C for modeling the long-interval time domain micro-expression motion characteristics and the asymmetric autoencoder network feature map constructed by step D for modeling the short-interval time domain micro-expression continuous frames, we need to first calculate p(y S ,y M ) and its s ,y M ; For the joint probability distribution, it is necessary to construct a contingency table C[i,j], as shown in formula (22): C[y S =i,y M =j]=count(y S =i∩y M =j) (22) For two discrete variables y M and S ,y M and S It refers to the discretized long-term spatial micro-expression features and short-term spatial micro-expression features. The contingency table C[i,j] represents the variable y M In state i and variable y S The joint frequency of occurrence in state j; According to this, the joint probability distribution p(y S ,y M ) and marginal probability distribution p(y S ), p(y M ) are obtained by formula (23) and formula (24) respectively: Where N is the total number of observations in the data; Micro-expression features with the same shape at different time scales are distinguished based on the vector v, which is defined by formula (25) as follows: In formula (25), Indicates that the mutual information between the residual network feature map for modeling the long-interval time-domain micro-expression motion characteristics constructed by step C and the asymmetric autoencoder network feature map for modeling the short-interval time-domain micro-expression continuous frames constructed by step D is calculated at a discrete time step d; Subsequently, based on the vector v, a variant of the Sigmoid function is used as a conversion function to convert the obtained mutual information entropy and obtain the adaptively variable threshold required by the spike neuron, as shown in formula (26): In formula (26), k represents the scaling factor, which is used to limit the adaptive threshold The upper and lower bounds of The change of will occur in the range [k,2k]; represents adaptive threshold, α and β are parameters used to control the threshold change rate and center point.

9. The method for micro-expression recognition based on multi-temporal information entropy across time domain according to claim 8, characterized in that: To analyze the impact of α, first, replace the Sigmoid function with the σ(x) function, as shown in formula (27): After that, the first-order derivative of the σ(x) function is taken, as shown in formula (28), and its derivative is: Finally, the first-order derivative of the σ(x) function σ(x|α,β) adjusted by α and β is expressed as shown in formula (29): According to formula (29), the derivative of the function σ(x|α,β) is proportional to α. Therefore, as α increases, the slope of the Sigmoid function at any point increases, causing the proposed adaptive threshold function to become steeper; Analyze the influence of β on the adaptive threshold function; consider the adjusted Sigmoid function σ(x|α,β), whose value at x=β is shown in formula (30): σ(α×(β-β))=σ(0)=0.5 (30) It shows that β actually controls the phase shift of the adaptive threshold function; Equation (31) is the maximum value of the adaptive threshold function; By setting the adaptive threshold Substitute the threshold δ back into formula (14) to complete the mapping of cross-temporal micro-expression features in the new temporal space formed by spiking neurons. By repeating this process for each sample, the key distributions with different focus points are extracted using multivariate temporal space. Used for micro-expression recognition.

10. A method for cross-temporal micro-expression recognition based on multi-temporal information entropy according to any one of claims 1-9, characterized in that: The facial key frame differences obtained after preprocessing the micro-expression video to be recognized are input into the residual network trained to model the long-interval time-domain micro-expression motion characteristics, and the input micro-expression frame set An asymmetric autoencoder network for modeling continuous frames of micro-expressions in short time intervals to achieve micro-expression classification and recognition, including: First, the micro-expression video to be recognized is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation; at the same time, the micro-expression video is pre-processed and marked as peak frame and start frame, and the motion features are obtained after pixel difference operation. by Periodically sampling a set of micro-expression frames adjacent to the peak frame; Afterwards, the motion features are first extracted through the residual network modeled with the long-interval temporal micro-expression motion characteristics according to step C to obtain the long-interval temporal micro-expression motion feature map; then the micro-expression frame set is converted into Input the asymmetric variational autoencoder, first pass it through the L-encoder including the pooling layer and the LSTM network to obtain the short-term micro-expression motion feature map, then splice the two feature maps according to step E to map the two feature maps to the new time domain space, calculate the mutual information entropy of the two feature maps according to step F to obtain the threshold for each position on the spliced ​​feature map, and then substitute the threshold back into step E to obtain the time domain pulse matrix, multiply the time domain pulse matrix and the spliced ​​feature map to obtain the final feature, and send it to the residual block for micro-expression recognition and classification.

Citation Information

Patent Citations

  • Micro-expression recognition method based on active migration learning

    CN108629314A

  • Camouflage face detection method based on micro-expression recognition

    CN110084152A

  • Micro-expression recognition method and system based on optical flow and RGB modal contrast learning

    CN113139479A

  • System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform

    US20180204111A1

Cited By

  • Ultra-precision full-field displacement measurement method and system based on convolution variational auto-encoder

    CN120890374A