Micro-expression recognition method and device based on amplified knowledge retention

Abstract

The application discloses a micro-expression recognition method and device based on enlarged knowledge retention, and the method comprises the following steps: acquiring macro-expression and micro-expression image sequence samples and performing pretreatment; extracting the last frame and the frame with an index in front from the macro-expression image sequence samples to form a strong-weak expression pair; constructing a strong-expression double-flow convolutional neural network, extracting static spatial features and dynamic spatial features of the strong-expression samples as training samples to perform training; taking the trained strong-expression network as a pre-training network, constructing a strong-weak expression double-flow convolutional neural network, and training by using the strong-weak expression pair; taking the trained strong-weak expression network as a pre-training network, constructing a weak-micro-expression double-flow convolutional neural network, and training by using the weak expression and the middle frame of the micro-expression image sequence; and inputting a micro-expression video to be recognized into the trained weak-micro-expression double-flow convolutional neural network to recognize the micro-expression category. The application has higher accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to image processing technology, and in particular to a micro-expression recognition method and device based on magnification knowledge preservation. BACKGROUND

[0002] Micro-expression is a kind of spontaneous and short facial expression produced by human when trying to hide their true emotions. Its subconscious and uncontrollable nature indicates that micro-expression can reveal the true psychological state of human, thus playing an important role in practical application scenarios such as criminal investigation, social security, and clinical medicine. Therefore, it is of great significance to accurately recognize micro-expression.

[0003] Compared with macro-expression, micro-expression has the characteristics of short duration and low facial muscle movement intensity, so it is very difficult to extract and distinguish the movement characteristics of micro-expression. To solve this problem, researchers introduce magnification technology to enhance micro-expression facial movement, so that significant features can be extracted when recognizing micro-expression, just like macro-expression, and the difficulty of distinguishing is reduced. Traditional manual magnification technology mainly focuses on the physical level, such as Eulerian Motion Magnification (EMM), Global Lagrangian Motion Magnification (GLMM), and deep learning filters are also applied to magnification, such as Learning-based Motion Magnification (LMM), etc. These methods have been proven to be able to improve the recognition performance by magnifying the movement intensity of micro-expression. However, most of the current magnification technologies are limited to the preprocessing stage, and no learning knowledge is introduced in the subsequent main expression feature extraction and discrimination stage. Although magnification in the preprocessing stage can enhance the movement intensity of micro-expression to a certain extent, it also inevitably increases noise and blur, resulting in reduced recognition performance. On the other hand, if the magnification knowledge can be preserved in the micro-expression feature extraction and discrimination stage, i.e., the weak features can be implicitly magnified and distinguished based on learning, then the micro-expression can be effectively recognized without introducing too much noise and blur. Therefore, the micro-expression recognition based on magnification knowledge preservation in the prior art still needs to be studied. SUMMARY

[0004] The present application relates to image processing technology, and in particular to a micro-expression recognition method and device based on magnification knowledge preservation.

[0005] Technical scheme: The micro-expression recognition method based on magnification knowledge preservation provided by the present application comprises:

[0006] (1) Obtain macro and micro expression image sequence samples and pre-process them, which are converted from several macro and micro expression videos;

[0007] (2) Extract the last frame from the macro expression image sequence samples as a strong expression sample, and extract the frame with an index in the front as a weak expression sample, to form a strong-weak expression pair;

[0008] (3) Construct a strong expression double-flow convolutional neural network, and extract the static spatial features and dynamic spatial features of the strong expression sample as training samples for training;

[0009] (4) Take the trained strong expression network as a pre-training network, and construct a strong-weak expression double-flow convolutional neural network;

[0010] (5) Take the strong-weak expression pair as training samples to train the strong-weak expression double-flow convolutional neural network;

[0011] (6) Take the trained strong-weak expression network as a pre-training network, and construct a weak-micro expression double-flow convolutional neural network;

[0012] (7) Pair the weak expression sample with the middle frame of the micro expression image sequence to form a weak-micro expression pair and take it as a training sample to train the weak-micro expression double-flow convolutional neural network;

[0013] (8) Extract the static spatial features and dynamic spatial features of the micro expression video to be identified and input them into the trained weak-micro expression double-flow convolutional neural network to identify the micro expression category.

[0014] Further, the pre-processing in step (1) includes face detection and face region cutting.

[0015] Further, step (2) specifically includes:

[0016] (2-1) Extract the last frame of each macro expression image sequence sample as a strong expression sample;

[0017] (2-2) Extract the frame with an index in the front of each macro expression image sequence sample as a weak expression sample:

[0018]

[0019]

[0020] In the formula, x a represents the a-th macro expression image sequence sample, represents the last frame of the image sequence sample, and T represents the index number of the last frame, denotes the index of the leading frame under the image sequence sample, t denotes the index number of the index leading frame;

[0021] (2-3) combining the extracted strong and weak expression sample groups to form a strong-weak expression pair

[0022] Further, step (3) specifically comprises:

[0023] (3-1) performing grayscale processing on the strong expression sample , and copying and stacking the obtained grayscale picture into a 3-channel grayscale picture as a static spatial feature;

[0024] (3-2) obtaining the starting frame of the macro-expression image sequence sample

[0025] (3-3) calculating the horizontal optical flow field, the vertical optical flow field and the optical strain intensity between the starting frame and the strong expression sample respectively, and stacking them into a 3-channel optical flow map as a dynamic temporal feature;

[0026] (3-4) establishing a strong expression double-flow convolutional neural network, the strong expression double-flow convolutional neural network comprising a static feature convolutional branch, a dynamic feature convolutional branch and a fusion layer for averaging and weighting the output values of the two branches, and the loss function used is:

[0027]

[0028] wherein, n c is the number of expression categories, R c is the actual probability distribution corresponding to the true label, z c and θ c respectively represent the cth output neuron and its weight;

[0029] (3-5) inputting the static spatial feature and the dynamic temporal feature into the strong expression double-flow convolutional neural network for training, and using a stochastic gradient descent algorithm during training.

[0030] Further, the static feature convolutional branch and the dynamic feature convolutional branch have the same structure, both being Resnet-18 networks.

[0031] Further, step (4) specifically comprises:

[0032] (4-1) constructing a strong-weak expression double-flow convolutional neural network having the same structure as the strong expression double-flow convolutional neural network;

[0033] (4-2) initializing the parameter values of the strong-weak expression dual-stream convolutional neural network as the parameter values of the trained strong expression dual-stream convolutional neural network.

[0034] Further, the step (5) specifically comprises:

[0035] (5-1) extracting the static spatial features and dynamic temporal features of each weak expression sample in the strong-weak expression pair;

[0036] (5-2) first inputting the static spatial features and dynamic temporal features of the strong expression sample in the strong-weak expression pair into the strong-weak expression dual-stream convolutional neural network to obtain a strong expression output, then inputting the static spatial features and dynamic temporal features of the weak expression sample into the strong-weak expression dual-stream convolutional neural network to obtain a weak expression output, and finally inputting the static spatial features and dynamic temporal features of the strong expression sample into the trained strong expression dual-stream convolutional neural network to obtain a pre-training output;

[0037] (5-3) constructing a loss function based on the weak expression output, the strong expression output, the true label and the pre-training output to train the parameter of the strong-weak expression dual-stream convolutional neural network, wherein the loss function is specifically:

[0038]

[0039]

[0040]

[0041]

[0042] wherein, is the total loss, is the cross-entropy loss between the weak expression output and the label, is the cross-entropy loss between the strong expression output and the pre-training output, and λ is the weight of the loss function, n c is the number of expression categories, R c is the actual probability distribution corresponding to the true label, respectively represent the cth output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak expression sample, respectively represent the cth output neuron and its weight of the strong expression dual-stream convolutional neural network for the strong expression sample, P softmax represents the class probability distribution recognized by the strong expression dual-stream convolutional neural network for the strong expression sample, respectively represent the cth output neuron and its weight of the strong-weak dual-stream convolutional neural network for the strong expression sample.

[0043] Further, the step (6) specifically comprises:

[0044] (6-1) constructing a weak-micro-expression dual-stream convolutional neural network with the same structure as the strong-weak-expression dual-stream convolutional neural network;

[0045] (6-2) initializing the parameter values of the weak-micro-expression dual-stream convolutional neural network as the parameter values of the trained strong-weak-expression dual-stream convolutional neural network.

[0046] Further, step (7) specifically comprises:

[0047] (7-1) obtaining the middle frame of the micro-expression image sequence sample as the weak expression sample in the strong-weak expression pair to form a weak-micro-expression pair

[0048] (7-2) extracting the static spatial features and dynamic temporal features of the micro-expression sample ;

[0049] (7-3) first inputting the static spatial features and dynamic temporal features of the weak expression sample of the weak-micro-expression pair into the weak-micro-expression dual-stream convolutional neural network to obtain a weak expression output, then inputting the static spatial features and dynamic temporal features of the micro-expression sample into the weak-micro-expression dual-stream convolutional neural network to obtain a micro-expression output, and finally inputting the static spatial features and dynamic temporal features of the weak expression sample into the trained strong-weak-expression dual-stream convolutional neural network to obtain a pre-training output;

[0050] (7-4) constructing a loss function based on the micro-expression output, the weak expression output, the true label, and the pre-training output to train the parameters of the weak-micro-expression dual-stream convolutional neural network, and the loss function is specifically:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] In the formula, is the total loss, γ and μ are loss function weights, is the cross-entropy loss between the micro-expression output and the label, is the cross-entropy loss between the weak expression output and the pre-training output, is the maximum mean difference between the weak expression features and the micro-expression features; n cR is the number of expression categories, c is the actual probability distribution corresponding to the real label, respectively represent the cth output neuron and its weight of the weak-micro dual-stream convolutional neural network for the micro-expression sample, respectively represent the cth output neuron and its weight of the weak-micro dual-stream convolutional neural network for the weak-expression sample, respectively represent the cth output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, Q softmax represents the category probability distribution recognized by the strong-weak dual-stream convolutional neural network for the weak-expression sample, x i and y j respectively represent the sample features from the weak-expression domain X and the micro-expression domain Y, and n and m respectively represent the number of samples in the weak-expression domain X and the micro-expression domain Y, represents the inner product in the reproducing Hilbert space H, and φ(·) represents a mapping function for projecting the sample feature data into the reproducing Hilbert space H.

[0057] The micro-expression recognition device based on amplified knowledge retention provided by the present application comprises a processor and a computer program stored on a memory and capable of running on the processor, and the processor implements the above method when executing the program.

[0058] Advantages: Compared with the prior art, the present application has the following significant advantages: the present application adopts the idea of amplified knowledge retention, adapts the weak-expression discriminant knowledge from the strong-expression discriminant knowledge, and further transfers and retains the weak-expression discriminant knowledge to the process of adapting the micro-expression discriminant knowledge, so as to implicitly amplify, extract and map the weak features in the micro-expression, thereby realizing high-accuracy micro-expression amplified recognition without introducing too much noise and ambiguity. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a flowchart of one embodiment of the micro-expression recognition method based on amplified knowledge retention provided by the present application;

[0060] Figure 2 is a schematic diagram of a dual-stream convolutional neural network. DETAILED DESCRIPTION

[0061] The present embodiment provides a micro-expression recognition method based on amplified knowledge retention, as shown in the following formula (1), which comprises the following steps: Figure 1

[0062] (1) Obtain macro and micro-expression image sequence samples and perform preprocessing, wherein the macro and micro-expression image sequence samples are converted from a plurality of macro and micro-expression videos.

[0063] The preprocessing comprises face registration and face region cutting.​

[0064] (2) Extract the last frame of each macro-expression image sequence sample as a strong expression sample, and extract the frame with an index close to the front as a weak expression sample, to form a strong-weak expression pair.

[0065] This step specifically includes:

[0066] (2-1) Extract the last frame of each macro-expression image sequence sample as a strong expression sample;

[0067] (2-2) Extract the frame with an index close to the front of each macro-expression image sequence sample as a weak expression sample:

[0068]

[0069]

[0070] In the formula, x a represents the a-th macro-expression image sequence sample, represents the last frame of the image sequence sample, and T represents the index number of the last frame, represents the frame with an index close to the front of the image sequence sample, and t represents the index number of the frame with an index close to the front;

[0071] (2-3) Combine the extracted strong and weak expression samples to form a strong-weak expression pair

[0072] The strong-weak expression pair constructed comes from the same sequence sample under the same database, and the motion features between the two have high similarity, only the intensity difference exists, therefore, the strong expression discrimination knowledge can be used to adapt to the weak expression discrimination, and the learning knowledge of the weak motion features can be reserved and transmitted.

[0073] (3) Construct a strong expression double-flow convolutional neural network, the structure of which is as shown in Figure 2 , and extract the static spatial features and dynamic spatial features of the strong expression samples as training samples for training, to learn the strong expression discrimination knowledge.

[0074] This step specifically includes:

[0075] (3-1) Perform grayscale processing on the strong expression sample , and copy and stack the obtained grayscale picture into a 3-channel grayscale picture as a static spatial feature;

[0076] (3-2) Obtain the starting frame of the macro-expression image sequence sample

[0077] (3-3) Calculate the starting frame With strong expression samples The horizontal optical flow field, vertical optical flow field, and optical strain intensity between them are stacked into a 3-channel optical flow map as a dynamic temporal feature;

[0078] (3-4) Establish a strong facial expression dual-stream convolutional neural network, which includes a static feature convolutional branch, a dynamic feature convolutional branch, and a fusion layer that weights and fuses the output values ​​of the two branches by averaging the values. The static feature convolutional branch and the dynamic feature convolutional branch have the same structure, both being ResNet-18 networks, and the loss function used is... for:

[0079]

[0080] In the formula, n c R represents the number of emoji categories. c z represents the actual probability distribution corresponding to the true label. c θ c These represent the c-th output neuron and its weights, respectively.

[0081] (3-5) Input the static spatial features and dynamic temporal features into the strong expression dual-stream convolutional neural network for training, and use the stochastic gradient descent algorithm during training.

[0082] (4) Using the trained strong expression network as the pre-trained network, construct a strong-weak expression dual-stream convolutional neural network.

[0083] This step specifically includes:

[0084] (4-1) Construct a strong-weak expression dual-stream convolutional neural network with the same structure as the strong expression dual-stream convolutional neural network;

[0085] (4-2) Initialize the parameter values ​​of the strong-weak expression dual-stream convolutional neural network to the parameter values ​​of the trained strong expression dual-stream convolutional neural network, so as to maintain and inherit the learned strong expression discrimination knowledge.

[0086] (5) Use strong and weak expression pairs as training samples to train a strong-weak expression dual-stream convolutional neural network, so as to learn amplification discrimination knowledge that is adapted to weak expressions from the strong expression discrimination.

[0087] Specifically, it includes:

[0088] (5-1) Extract the static spatial features and dynamic temporal features of each weak expression sample in the strong and weak expression pair; the extraction method is the same as that in steps (3-1) to (3-3);

[0089] (5-2) first input the static spatial features and dynamic temporal features of the strong expression sample of the strong-weak expression pair into the strong-weak expression double-flow convolutional neural network to obtain a strong expression output, then input the static spatial features and dynamic temporal features of the weak expression sample into the strong-weak expression double-flow convolutional neural network to obtain a weak expression output, and finally input the static spatial features and dynamic temporal features of the strong expression sample into the trained strong expression double-flow convolutional neural network to obtain a pre-training output;

[0090] (5-3) based on the weak expression output, the strong expression output, the true label and the pre-training output, a loss function is constructed to train the parameters of the strong-weak expression double-flow convolutional neural network, wherein the loss function is specifically:

[0091]

[0092]

[0093]

[0094]

[0095] In the formula, is the total loss, is the cross-entropy loss between the weak expression output and the label, is the cross-entropy loss between the strong expression output and the pre-training output, and λ is the weight of the loss function, n c is the number of expression categories, R c is the actual probability distribution corresponding to the true label, respectively represent the cth output neuron and its weight of the strong-weak double-flow convolutional neural network for the weak expression sample, respectively represent the cth output neuron and its weight of the strong expression double-flow convolutional neural network for the strong expression sample, P softmax represents the class probability distribution of the strong expression sample recognized by the strong expression double-flow convolutional neural network, respectively represent the cth output neuron and its weight of the strong-weak double-flow convolutional neural network for the strong expression sample.

[0096] By constructing the cross-entropy loss between the strong expression output and the pre-training network output, the network can maintain the strong expression feature extraction and discrimination knowledge learned by the pre-training network to a certain extent, and avoid its pollution in the weak expression learning process; at the same time, the cross-entropy loss between the weak expression output and the label is constructed to guide the weak expression classification according to the specific category, and the knowledge is continuously learned and amplified from the retained strong expression discrimination, so as to adapt to the excellent weak expression discrimination.

[0097] (6) Taking the trained strong-weak expression network as a pre-training network, a weak-micro-expression dual-stream convolutional neural network is constructed.

[0098] Specifically comprising:

[0099] (6-1) A weak-micro-expression dual-stream convolutional neural network with the same structure as the strong-weak expression dual-stream convolutional neural network is constructed.

[0100] (6-2) The parameter values of the weak-micro-expression dual-stream convolutional neural network are initialized as the parameter values of the trained strong-weak expression dual-stream convolutional neural network, so as to maintain and inherit the learned strong and weak expression amplification discrimination knowledge.

[0101] (7) The weak expression sample is paired with the intermediate frame of the micro-expression image sequence to form a weak-micro-expression pair and is used as a training sample to train the weak-micro-expression dual-stream convolutional neural network, so as to learn the amplification discrimination knowledge of the micro-expression from the amplification discrimination retained by the weak expression.

[0102] Specifically comprising:

[0103] (7-1) The intermediate frame of the micro-expression image sequence sample is obtained As a micro-expression sample, the weak expression sample in the strong-weak expression pair is combined to form a weak-micro-expression pair

[0104] (7-2) The static spatial features and dynamic temporal features of the micro-expression sample are extracted;

[0105] (7-3) The static spatial features and dynamic temporal features of the weak expression sample of the weak-micro-expression pair are first input into the weak-micro-expression dual-stream convolutional neural network to obtain a weak expression output, then the static spatial features and dynamic temporal features of the micro-expression sample are input into the weak-micro-expression dual-stream convolutional neural network to obtain a micro-expression output, and finally the static spatial features and dynamic temporal features of the weak expression sample are input into the trained strong-weak expression dual-stream convolutional neural network to obtain a pre-training output;

[0106] (7-4) A loss function is constructed based on the micro-expression output, the weak expression output, the true label and the pre-training output, so as to train the parameters of the weak-micro-expression dual-stream convolutional neural network, and the loss function is specifically:

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] wherein, is the total loss, γ, μ are loss function weights, is the cross-entropy loss between the micro-expression output and the label, is the cross-entropy loss between the weak-expression output and the pre-trained output, is the maximum mean difference between the weak-expression feature and the micro-expression feature; n c is the number of expression categories, R c is the actual probability distribution corresponding to the true label, respectively represent the c-th output neuron and its weight of the weak-micro dual-stream convolutional neural network for the micro-expression sample, respectively represent the c-th output neuron and its weight of the weak-micro dual-stream convolutional neural network for the weak-expression sample, respectively represent the c-th output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, Q softmax represents the class probability distribution identified by the strong-weak dual-stream convolutional neural network for the weak-expression sample, x i and y j respectively represent the sample features from the weak-expression domain X and the micro-expression domain Y, n and m respectively represent the number of samples in the weak-expression domain X and the micro-expression domain Y, represents the inner product in the reproducing Hilbert space H, φ(·) represents a mapping function that projects sample feature data into the reproducing Hilbert space H.

[0113] By constructing the above maximum mean difference loss function between the weak-expression feature and the micro-expression feature, the feature distribution of the two domains can be aligned, so that the network learns the domain-invariant expression feature, thereby more accurately completing the transfer of the magnified knowledge between the weak and micro-expression feature discrimination; at the same time, the cross-entropy loss between the weak-expression output and the pre-trained network output is constructed, which can enable the network to retain the weak-expression feature extraction and discrimination knowledge learned by the pre-trained network to some extent, avoid its pollution in the micro-expression learning process, and contain the magnified knowledge retained from the strong expression discriminativeness; in addition, the cross-entropy loss between the micro-expression output and the label is constructed to guide the actual class, constantly learn the magnified knowledge from the retained weak-expression discriminativeness, implicitly amplify the weak motion feature, and thus adapt to excellent micro-expression discriminativeness.

[0114] (8) Extracting the static spatial features and dynamic spatial features of the to-be-identified micro-expression video, inputting them into the trained weak-micro-expression dual-stream convolutional neural network, and identifying the micro-expression category.

[0115] The embodiment also provides a micro-expression recognition device based on amplified knowledge retention, comprising a processor and a computer program stored on a memory and capable of running on the processor, and the processor implements the above method when executing the program.

[0116] To verify the effectiveness of the present application, micro-expression recognition is performed between the CAME2 micro-expression database and the HS sub-database of the SMIC database, and the macro-expression database is selected as the CK+ database.

[0117] The following processing is performed on the expressions of the CASME2 database: the categories with too few sample numbers are omitted, and only 5 main emotional categories in the original 7 categories are selected to complete the 5-category recognition task, which are others, disgust, happiness, surprise and repression. The SMIC database itself has negative, positive and surprise categories, and the three-class recognition task is completed. To enable micro-expression discrimination to learn amplified knowledge from macro-expression discrimination, the macro-expression emotional categories and the micro-expression emotional categories need to be consistent, so the following processing is performed on the expressions of the CK+ database: when performing the CASME2 expression recognition task, the fear and sadness categories in the CK+ are removed, and the anger and contempt categories are merged into the others category, and the remaining emotions remain unchanged; when performing the SMIC expression recognition task, the happiness is divided into the positive category, the negative emotions such as anger are divided into the negative category, and the surprise emotion remains unchanged. The expression category merging is shown in Tables 1 and 2:

[0118] Table 1

[0119] Expression category CK+ CASME2 Others Anger, contempt Others Disgust Disgust Disgust Happiness Happiness Happiness Surprise Surprise Surprise Repression - Repression

[0120] Table 2

[0121]

[0122] The verification results and comparison with other latest methods are shown in Table 3:

[0123] Table 3

[0124] Method Year CASME2 SMIC-HS LBP-SIP 2014 66.40 62.80 DSSN 2019 70.78 63.41 TSCNN-I 2019 74.05 72.74 MicroNet 2020 75.60 76.80 LFM 2020 73.98 71.34 GEME 2021 75.20 64.63 GACNN 2021 81.30 N / A AMAN 2022 75.40 79.87 The method 2022 82.93 80.49

[0125] In Table 3, N / A represents no relevant record.

[0126] The experimental results show that the micro-expression recognition method based on the present application achieves a high micro-expression recognition rate.

[0127] The above merely provides the preferred embodiment of the present application, and cannot allude the protection scope of the present application, thus the equivalent changes made by the claims of the present application still belong to the scope of the present application.

Claims

1. A micro-expression recognition method based on amplified knowledge retention, characterized by The method comprises: (1) obtaining macro and micro expression image sequence samples and preprocessing, the macro and micro expression image sequence samples are converted from a plurality of macro and micro expression videos; (2) extracting the last frame from the macro expression image sequence sample as a strong expression sample, extracting the index frame as a weak expression sample, and forming a strong and weak expression pair; (3) constructing a strong expression double-flow convolutional neural network, and extracting the static spatial features and dynamic spatial features of the strong expression sample as training samples for training; (4) using the trained strong expression network as a pre-training network, constructing a strong-weak expression double-flow convolutional neural network; (5) training the strong-weak expression double-flow convolutional neural network with the strong and weak expression pair as training samples; (6) using the trained strong-weak expression network as a pre-training network, constructing a weak-micro expression double-flow convolutional neural network; (7) pairing the weak expression sample with the middle frame of the micro expression image sequence to form a weak-micro expression pair and training the weak-micro expression double-flow convolutional neural network with the weak-micro expression pair as training samples; (8) extracting the static spatial features and dynamic spatial features of the to-be-identified micro expression video and inputting them into the trained weak-micro expression double-flow convolutional neural network to identify the micro expression category; Step (3) specifically comprises: (3-1) For the strong expression samples Perform grayscale processing, copy and stack the resulting grayscale images to form a 3-channel grayscale image, which is used as a static spatial feature; (3-2) acquiring a starting frame of the macro-expression image sequence sample ; (3-3) Calculate the starting frame respectively The horizontal and vertical optical flow fields and the optical strain intensity between the strong expression samples are calculated and stacked into a 3-channel optical flow map as dynamic temporal features; (3-4) a strong expression double-flow convolutional neural network is established, the strong expression double-flow convolutional neural network comprises a static feature convolution branch, a dynamic feature convolution branch and a fusion layer for averaging and weighting the output values of the two branches, and the loss function used is: is: , In the formula, For the number of emoji categories, This represents the actual probability distribution corresponding to the true labels. , They represent the first Each output neuron and its weights; (3-5) inputting the static spatial features and dynamic time features into the strong expression double-flow convolutional neural network for training, and using a stochastic gradient descent algorithm during training.

2. The micro-expression recognition method based on amplified knowledge retention of claim 1, wherein: The preprocessing in step (1) comprises face detection and face region cutting.

3. The micro-expression recognition method based on amplified knowledge retention of claim 1, wherein: Step (2) specifically comprises: (2-1) Extracting the last frame of each macro-expression image sequence sample as a strong expression sample ; (2-2) Extract the index top frame of each macro-expression image sequence sample as a weak expression sample: , , In the formula, represents the first macro-expression image sequence sample, represents the last frame in the image sequence sample, represents the index number of the last frame, represents the index preceding frame in the image sequence sample, and t represents the index number of the index preceding frame. (2-3) Combining the extracted strong and weak expression sample sets to form strong-weak expression pairs .

4. The micro-expression recognition method based on amplified knowledge retention of claim 1, wherein: The static feature convolution branch and the dynamic feature convolution branch have the same structure, which is a Resnet-18 network.

5. The micro-expression recognition method based on knowledge retention of amplification according to claim 1, characterized in that: Step (4) specifically comprises: (4-1) constructing a strong-weak expression double-flow convolutional neural network with the same structure as the strong expression double-flow convolutional neural network; (4-2) initializing the parameter values of the strong-weak expression double-flow convolutional neural network as the parameter values of the trained strong expression double-flow convolutional neural network.

6. The micro-expression recognition method based on knowledge retention of amplification according to claim 1, characterized in that: Step (5) specifically comprises: (5-1) extracting the static spatial features and dynamic time features of each weak expression sample in the strong and weak expression pair; (5-2) inputting the static spatial features and dynamic time features of the strong expression sample of the strong and weak expression pair into the strong-weak expression double-flow convolutional neural network to obtain a strong expression output, then inputting the static spatial features and dynamic time features of the weak expression sample into the strong-weak expression double-flow convolutional neural network to obtain a weak expression output, and finally inputting the static spatial features and dynamic time features of the strong expression sample into the trained strong expression double-flow convolutional neural network to obtain a pre-training output; (5-3) constructing a loss function based on the weak expression output, the strong expression output, the true label and the pre-training output to train the parameters of the strong-weak expression double-flow convolutional neural network, wherein the loss function is specifically: , , , , In the formula, For the total loss, The cross-entropy loss is between weak expression output and label. The cross-entropy loss is used to compare the strong facial expression output with the pre-trained output. for The weights of the loss function, For the number of emoji categories, This represents the actual probability distribution corresponding to the true labels. , These represent the weak expression samples in the strong-weak two-stream convolutional neural network. Each output neuron and its weights , These represent the strong expression samples in the strong expression two-stream convolutional neural network. Each output neuron and its weights This represents the probability distribution of categories identified by a strong facial expression sample under a strong facial expression dual-stream convolutional neural network. , These represent the strong expression samples in the strong-weak two-stream convolutional neural network. Each output neuron and its weights.

7. The micro-expression recognition method based on knowledge retention of amplification according to claim 1, characterized in that: Step (6) specifically comprises: (6-1) constructing a weak-micro expression double-flow convolutional neural network with the same structure as the strong-weak expression double-flow convolutional neural network; (6-2) initializing the parameter values of the weak-micro expression double-flow convolutional neural network as the parameter values of the trained strong-weak expression double-flow convolutional neural network.

8. The micro-expression recognition method based on knowledge retention of amplification according to claim 1, characterized in that: Step (7) specifically comprises: (7-1) obtaining a middle frame of the micro-expression image sequence sample As a micro-expression sample, the weak expression sample in the strong-weak expression pair Combined to form a weak micro-expression pair ; (7-2) Extracting micro-expression samples static spatial features and dynamic temporal features; (7-3) first input the static spatial features and dynamic temporal features of the weak expression sample of the weak micro-expression pair into the weak-micro-expression dual-flow convolutional neural network to obtain a weak expression output, then input the static spatial features and dynamic temporal features of the micro-expression sample into the weak-micro-expression dual-flow convolutional neural network to obtain a micro-expression output, and finally input the static spatial features and dynamic temporal features of the weak expression sample into the trained strong-weak expression dual-flow convolutional neural network to obtain a pre-training output; (7-4) a loss function is constructed based on the micro-expression output, the weak expression output, the true label and the pre-training output to train the parameters of the weak-micro-expression dual-flow convolutional neural network, and the loss function is specifically: , , , , , wherein, is the total loss, , is the loss function weight, is the cross-entropy loss between the micro-expression output and the label, is the cross-entropy loss between the weak-expression output and the pre-trained output, is the maximum mean difference between the weak-expression feature and the micro-expression feature; is the number of expression classes, is the actual probability distribution corresponding to the true label, respectively represent the i-th output neuron and its weight of the weak-micro dual-stream convolutional neural network for the micro-expression sample, respectively represent the i-th output neuron and its weight of the weak-micro dual-stream convolutional neural network for the weak-expression sample, respectively represent the i-th output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, respectively represent the i-th output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, respectively represent the i-th output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, respectively represent the i-th output neuron and its weight of the strong-weak dual-stream convolutional neural network for the weak-expression sample, represents the class probability distribution recognized by the strong-weak dual-stream convolutional neural network for the weak-expression sample, and denote the sample features from the weak expression domain and the micro-expression domain respectively, and denote the number of samples in the weak expression domain and the micro-expression domain respectively, denotes the inner product on the reproducing Hilbert space (·) denotes the mapping function that projects the sample feature data into the reproducing Hilbert space .​ 9.A micro-expression recognition device based on amplified knowledge retention, comprising a processor and a computer program stored on a memory and executable on the processor, characterized in that: The processor implements the method in any one of claims 1-8 when executing the program.

Images