Micro-expression emotion recognition method based on over-parameterization

Abstract

The invention relates to a micro-expression emotion recognition method based on over-parameterization, and the method specifically comprises the steps: S1, preprocessing: obtaining a micro-expression video, carrying out the face detection of each frame in the video, and carrying out the preprocessing operation of face alignment and cutting; s2, feature extraction: calculating optical flow images of the face images of the peak frame and the initial frame; s3, inputting the optical flow graph obtained by feature extraction into a convolutional neural network model for training; s4, performing structural re-parameterization processing on the trained multi-branch convolutional neural network model; and S5, inputting a to-be-detected micro-expression video into the single-branch convolutional neural network model after structure re-parameterization for reasoning, and identifying the micro-expression. In the training stage, model parameters are improved, and the expression ability of the model is improved, so that the micro-expression discrimination precision is improved, and the reasoning speed is optimized; in the training stage, parameters are reduced in a structure re-parameterization mode, and model reasoning is accelerated while precision is reserved.

Description

Technical Field

[0001] The present invention relates to the field of micro-expression recognition, and in particular to a micro-expression emotion recognition method based on over-parameterization. Background Art

[0002] Facial expressions are a way to show a person's emotional state without verbal communication. However, a person's facial expressions do not always accurately reflect their emotions, as they may deliberately suppress facial expressions that they deem inappropriate. Microexpressions occur when a person tries to hide his or her true emotions and last very briefly. Since microexpressions can reveal a person's true emotions, they can be applied in many fields, such as depression treatment, lie detection, and criminal interrogation. Compared with normal facial expressions, microexpressions usually last very short, ranging from 1 / 25 second to 1 / 5 second. In addition to their short duration, microexpressions have another notable characteristic: low intensity and small range. Due to these three characteristics, microexpressions are difficult to recognize with the naked eye.

[0003] In recent years, with the rapid development of deep learning, convolutional neural networks have been able to automatically extract features for classification end-to-end. Micro-expression recognition methods based on deep learning have also received more and more attention and achieved increasingly excellent performance. In the process of deep learning development, although the scale of network structure continues to expand, the growth of data set size is relatively limited. According to the theory of traditional machine learning, over-parameterization, that is, models with extremely many parameters, are more likely to lead to overfitting. However, experimental results show that over-parameterized models can achieve better performance. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a micro-expression emotion recognition method based on over-parameterization. This method uses over-parameterization technology to expand branches horizontally during training, increase network parameters, and enhance the expressive ability of the network, thereby improving training accuracy. At the same time, the network model is designed based on structural re-parameterization, and features of feature maps at various scales are obtained through multiple branches. During inference, multiple branches are compressed into a single branch through structural re-parameterization, thereby improving the model training speed while retaining accuracy.

[0005] The technical solution adopted by the present invention is a micro-expression emotion recognition method based on over-parameterization, comprising the following steps:

[0006] S1. Preprocessing: obtaining a micro-expression video, performing face detection on each frame in the video to obtain a face image, and preprocessing the face image to obtain a preprocessed face image;

[0007] S2, feature extraction: extract the peak frame and the starting frame in the preprocessed face image, and calculate the optical flow image according to the starting frame and the peak frame, and the expression is: Where I0 and I1 represent the peak frame and the starting frame in the face image, respectively; x is a pixel of I0; u(x) represents the optical flow field (u, v); u represents the horizontal optical flow and v represents the vertical optical flow; I0(x)-I1(x+u(x)) is the grayscale difference between I0 and I1, also known as the data constraint term; and λ represents the weight parameter; represents the motion regularization constraint;

[0008] S3, constructing a multi-branch convolutional neural network model, the multi-branch convolutional neural network model includes a first branch, a second branch, a third branch, a weighting module and an activation function module, the first branch is a 1×1 convolution, the second branch is a 3×3 convolution, and the third branch is an identity mapping;

[0009] S4, inputting the optical flow image into the multi-branch convolutional neural network model for training to obtain a trained multi-branch convolutional neural network model;

[0010] S5, performing structural reparameterization processing on the trained multi-branch convolutional neural network model to obtain a processed single-branch convolutional neural network model;

[0011] S6. Input the micro-expression video to be detected into the re-parameterized single-branch convolutional neural network model for inference to recognize the micro-expression.

[0012] Preferably, in step S1, the specific process of preprocessing the face image is: aligning the face image and cropping it into images of the same size.

[0013] Preferably, in step S4, the specific process of inputting the optical flow image into the multi-branch convolutional neural network model for training comprises the following steps:

[0014] S4.1, inputting the optical flow image into the first branch, the second branch and the third branch respectively for feature extraction;

[0015] S4.2, the extracted features of each branch are input into the weighting module for weighting, and the weighting module outputs the aggregation result;

[0016] S4.3, inputting the aggregation result into an activation function module for nonlinear operation;

[0017] S4.4, take the next optical flow image and return to step S4.1 for cyclic training until a trained multi-branch convolutional neural network model is obtained.

[0018] Preferably, in step S5, the specific process of re-parameterizing the trained multi-branch convolutional neural network model comprises the following steps:

[0019] S5.1. Introduce an n×n convolutional layer parameter matrix, and weight the batch normalization parameters in each branch of the trained multi-branch convolutional neural network model to each item in the convolutional layer parameter matrix, so that batch normalization is integrated into the convolutional layer parameter matrix;

[0020] S5.2. Add the convolutional layer parameter matrices on all branches point by point, so that the multi-branch convolutional layers are merged into a single branch, and a multi-branch convolutional neural network model is obtained after re-parameterization.

[0021] The beneficial effects of the present invention are: using over-parameterization technology, the model parameters are increased by horizontally expanding multiple branches, the expression ability of the network model is improved, and thus the model accuracy is improved; the network model is processed by structural re-parameterization, and a more complex multi-branch convolutional neural network is used to enhance the expression ability of the network model. In order to avoid slow reasoning speed and large memory usage caused by complex multiple branches, the multi-branch model is merged into a single branch to accelerate the reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a process of a micro-expression emotion recognition method based on over-parameterization according to the present invention;

[0023] Figure 2 It is a structural schematic diagram of a multi-branch network module in an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of re-parameterization in an embodiment of the present invention; Figure 4 Schematic diagram of the change of test loss with training iterations when k subjects are used as samples for training in the experiment of the present invention. DETAILED DESCRIPTION

[0025] The invention will be further described below with reference to the accompanying drawings and in combination with specific implementations, so that those skilled in the art can implement the invention with reference to the description. The protection scope of the invention is not limited to the specific implementations.

[0026] The present invention relates to a micro-expression emotion recognition method based on over-parameterization, such as Figure 1 As shown, the following steps are included:

[0027] S1. Preprocessing: Get micro-expression videos, perform face detection on each frame in the video, perform face alignment and cropping preprocessing operations on them, and obtain preprocessed face images; the micro-expression samples use the MEGC2019 mixed data set as the training set, with a total of 409 micro-expression sequences, including 109 positive expressions, 83 surprised expressions, and 250 negative expressions; perform face detection and positioning on these sequences, and align and crop the face areas of all frame images according to the key facial feature points;

[0028] S2. Feature extraction: Take the starting frame and peak frame of the aligned micro-expression sequence images and use the TV-L1 optical flow method to obtain the optical flow map and input it into the network.

[0029] Optical flow refers to the instantaneous speed of pixel movement of a moving object in space on the imaging plane. It is a method that uses the changes in pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the pixels of the previous frame and the current frame, thereby calculating the motion information of the pixels between adjacent frames. It can extract optical flow information from continuous image sequences in micro-expression datasets and effectively describe the movement of facial muscles when micro-expressions occur. The TV-L1 optical flow method improves the original optical flow method and improves its robustness to noise. The formula of the TV-L1 optical flow algorithm can be expressed as:

[0030]

[0031] Where I0 and I1 represent two frames in the image sequence, x is a pixel of I0, u(x) represents the optical flow field (u, v), u represents the horizontal optical flow, v represents the vertical optical flow, I0(x)-I1(x+u(x)) is the grayscale difference between I0 and I1, also known as the data constraint term, and λ is the weight parameter of this term; is the motion regularization constraint.

[0032] S3, input the optical flow map obtained by feature extraction into a multi-branch convolutional neural network model for training, and obtain a trained multi-branch convolutional neural network model; the multi-branch convolutional neural network includes three branches, which are composed of 3×3 convolution, 1×1 convolution and identity mapping. The intermediate feature map is input into the three branches respectively to obtain the output feature map, and the three output feature maps are added point by point to obtain the final feature map.

[0033] The multi-branch convolutional neural network module is improved on the basis of RepVGG. RepVGG is a simple and powerful convolutional neural network architecture, similar to the VGG network, which stacks the network architecture through 3×3 convolutions. The VGG network is designed with a multi-branch network architecture. Different branches can learn to focus on learning features of different levels or types, which is conducive to the output of richer feature representations and increases the network's ability to capture complex patterns in the data. By processing different types of features in parallel, the multi-branch network can understand the input data in more detail, thereby improving the accuracy of classification or prediction. Therefore, a multi-branch network can provide a richer and more flexible way to learn and represent data, thereby improving performance while also enhancing the generalization ability and application scope of the model.

[0034] Inspired by ResNet, it uses residual connections with identity mapping to model the information flow as y=x+f(x), which allows point-by-point addition while ensuring that the dimensions of x and f(x) are consistent. When the dimensions of the two are inconsistent, it is necessary to modify their dimensions through 1×1 convolution, that is, modeling as y=g(x)+f(x). In order to make the multi-branch information richer, the network model uses three branches: 3×3 convolution, 1×1 convolution, and identity mapping, to model y=x+g(x)+f(x).

[0035] The traditional convolutional layer includes three parts: convolution, batch normalization, and activation function. In the design of multi-branch network models, in order to ensure that the results of multiple branches can be finally aggregated, the output size dimensions of multiple branches must be kept consistent. Since the batch normalization parameters required by different branches are slightly different, each branch includes convolution and batch normalization parts. After all branches obtain the results, all the results are added point by point, and the output feature map of the module is obtained through the activation function.

[0036] With the development of deep learning, networks with deeper and larger models have been proposed, and large models have achieved better and better results. However, the amount of data that matches the model size has not increased significantly. Studies have also shown that over-parameterized models, that is, models with extremely large parameters, including convolutional neural networks and Transformer models, can maintain a good level of test error and generalization error even when the data scale is relatively small. Therefore, increasing parameters can also achieve a relatively good model.

[0037] In order to make the model over-parameterized while avoiding overly complex network design, the multi-branch network model is over-parameterized by horizontally expanding branches, mainly for 3×3 convolution, and horizontally expanding multiple 3×3 convolutions, thereby greatly improving the parameters of the network model to achieve better network performance.

[0038] S4. Reparameterize the trained multi-branch convolutional neural network model to obtain a processed convolutional neural network model.

[0039] The parameters of the convolutional layer in the above multi-branch network model are merged into the convolutional layer parameters of a single branch through the method of structural reparameterization, thereby reducing the increase in inference time and memory, while retaining all the parameters and performance of the original multi-branch model.

[0040] Although the above-mentioned multi-branch can obtain richer feature representation and thus obtain better performance, even when the FLOP of the multi-branch architecture network is lower than that of the single-branch network, its running speed will still be lower than that of the single-branch network due to its need to frequently access memory. At the same time, since the multi-branch network occupies a high amount of memory, the result of each branch needs to be retained until the addition or aggregation operation, which significantly increases the memory usage and puts forward memory requirements for the inference device. In order to accelerate the inference speed and reduce the memory usage while ensuring accuracy, the structural reparameterization method is used to merge the parameters of multiple branches into one branch, so that the more consistent feature representation performance and higher accuracy of the multiple branches are retained.

[0041] For a convolutional layer parameter matrix with D convolution kernels of size H×W, the size of the input feature map is U×V, and the number of channels is C, then the input feature map can be expressed as I∈R U×V×C , the convolutional layer parameter matrix can be expressed as F∈R H×W×D , the output is O∈R R ×T×D ; The feature map output by the jth convolution kernel in the convolution layer parameter matrix is:

[0042]

[0043] Where, “·” represents a two-dimensional convolution operation, M :,:,k represents the kth channel of the input feature map, Represents the kth channel of the jth convolution kernel in the convolutional layer.

[0044] Since its introduction, batch normalization has been widely used in the field of deep learning to accelerate the convergence speed of the model and improve the generalization ability of the model. The batch normalization layer is often followed by a linear scaling transformation to enhance its ability. Therefore, the batch normalization parameters in each branch of the trained multi-branch convolutional neural network model are weighted to each item in the convolutional layer parameter matrix, so that batch normalization is integrated into the convolutional layer parameter matrix. Then the above equation is transformed into:

[0045]

[0046] Among them, μ j and σ jdenote the accumulated mean and standard deviation in batch normalization, γ j and β j They represent the scaling factor and offset factor learned by the batch normalization layer, respectively. In this way, after training, the parameters of the convolutional layer and batch normalization layer on each branch can be fused into one convolutional layer.

[0047] The two-dimensional convolution operation is additive, that is:

[0048]

[0049] Among them, I represents the input feature map, K (1) and K (2) Represent two convolution kernels respectively. Represents the addition of the corresponding position elements on the two convolution kernel parameter matrices. Therefore, the corresponding position elements on the convolution layer parameter matrices on multiple branches can be added to obtain a new convolution layer parameter matrix, thereby merging the convolution layers on multiple branches into a single-branch convolution layer.

[0050] Finally, the parameters of multiple branches can be fused into a single convolutional layer with the following parameter matrix, which is expressed as:

[0051]

[0052] The above structural reparameterization process is also applicable to the fusion of convolution branches with different convolution kernel sizes and the fusion of the same identity mapping branch. The identity mapping branch can be regarded as a 1×1 convolution with the identity matrix as the kernel. For the fusion of 1×1 convolution and 3×3 convolution, the multi-branch convolution can be fused into a single branch by adding the 1×1 convolution kernel to the center point of the 3×3 convolution kernel parameter matrix.

[0053] The following experiments are conducted to verify the effectiveness of the micro-expression emotion recognition method based on over-parameterization of the present invention.

[0054] Since the number of micro-expression data sets is small, in order to ensure that the model can achieve good recognition accuracy with small data, the Leave One Subject Out (LOSO) experiment is used. For a sample set containing k subjects, the Leave One Subject Out (LOSO) method is iterated k times. In each iteration, all samples of one subject are selected as the test set, and the remaining samples are used for training and validation. After ensuring that each subject is used as a test set once, the average of all iteration results is calculated.

[0055] Stochastic Gradient Descent (SGD) is a commonly used optimization algorithm, mainly used for training machine learning and deep learning models. It minimizes the loss function by continuously adjusting model parameters (such as the weights and biases of the neural network). Stochastic gradient descent calculates the gradient based on a mini-batch or a single sample, and uses this gradient to update the model parameters. Momentum can be used to accelerate the process of stochastic gradient descent. By introducing a momentum term, the exponentially weighted average of the previous gradients is accumulated to reduce the oscillation phenomenon in training. Weight decay is a regularization technique used to prevent the model from overfitting. It adds a penalty term related to the size of the parameter to the loss function so that the model parameters will not become too large. The learning rate determines the step size of each parameter update. A learning rate that is too large will cause the training process to be unstable or even unable to converge; a learning rate that is too small will cause the convergence speed to be too slow. Through these methods and parameter settings, the model can be helped to converge to the optimal solution better and faster.

[0056] The model uses SGD with momentum, momentum is set to 0.9, and weight decay is set to 10. -4 , the learning rate is set to 0.1. For LOSO evaluation, each fold is trained for 30 epochs.

[0057] Depend on Figure 4 It can be seen that even for different subjects as test sets, the test loss gradually decreases with the training iterations. As for the impact of whether to use structural reparameterization on the inference speed, when structural reparameterization is not used, the inference speed is 250ms, and after the model structure is compressed by structural reparameterization, the inference speed is halved to 125ms.

[0058] Experiments have shown that by adding branches and thus significantly improving the model parameters, the model can have better expressive capabilities for micro-expression features, thereby improving the recognition accuracy of micro-expressions. The introduction of structural reparameterization can compress the network model while ensuring performance, making its model occupancy lower and the inference speed faster.

[0059] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0060] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A micro-expression emotion recognition method based on over-parameterization, characterized by: The following steps are involved: S1. Preprocessing: obtaining a micro-expression video, performing face detection on each frame in the video to obtain a face image, and preprocessing the face image to obtain a preprocessed face image; S2, feature extraction: extract the peak frame and the starting frame in the preprocessed face image, and calculate the optical flow image according to the starting frame and the peak frame, and the expression is: Where I0 and I1 represent the peak frame and the starting frame in the face image, respectively; x represents a pixel of I0; u(x) represents the optical flow field (u, v); u represents the horizontal optical flow; v represents the vertical optical flow; I0(x)-I1(x+u(x)) represents the grayscale difference between I0 and I1; and λ represents the weight parameter; represents the motion regularization constraint; S3, constructing a multi-branch convolutional neural network model, the multi-branch convolutional neural network model includes a first branch, a second branch, a third branch, a weighting module and an activation function module, the first branch is a 1×1 convolution, the second branch is a 3×3 convolution, and the third branch is an identity mapping; S4, inputting the optical flow image into the multi-branch convolutional neural network model for training to obtain a trained multi-branch convolutional neural network model; S5, performing structural reparameterization processing on the trained multi-branch convolutional neural network model to obtain a processed single-branch convolutional neural network model; S6. Input the micro-expression video to be detected into the re-parameterized single-branch convolutional neural network model for inference to recognize the micro-expression.

2. The micro-expression emotion recognition method based on over-parameterization according to claim 1, characterized in that: In step S1, the specific process of preprocessing the face image is: aligning the face image and cropping it into images of the same size.

3. The micro-expression emotion recognition method based on over-parameterization according to claim 2, characterized in that: In step S4, the specific process of inputting the optical flow image into the multi-branch convolutional neural network model for training includes the following steps: S4.1, inputting the optical flow image into the first branch, the second branch and the third branch respectively for feature extraction; S4.2, the features extracted from each branch are input into the weighting module for weighting, and the weighting module outputs the aggregation result; S4.3, inputting the aggregation result into an activation function module for nonlinear operation; S4.4, take the next optical flow image and return to step S4.1 for cyclic training until a trained multi-branch convolutional neural network model is obtained.

4. The micro-expression emotion recognition method based on over-parameterization according to claim 3, characterized in that: In step S5, the specific process of re-parameterizing the trained multi-branch convolutional neural network model includes the following steps: S5.

1. Introduce a convolutional layer parameter matrix of size H×W, and weight the batch normalization parameters in each branch of the trained multi-branch convolutional neural network model to each item of the convolutional layer parameter matrix, so that batch normalization is integrated into the convolutional layer parameter matrix. The specific expression is: Among them, O :,:,j represents the feature map of the jth convolution kernel output in the convolution layer parameter matrix, I represents the input feature map, F represents the convolution layer parameter matrix, "·" represents a two-dimensional convolution operation, M :,:,k represents the kth channel of the input feature map, represents the kth channel of the jth convolution kernel in the convolution layer, μ j and σ j denote the accumulated mean and standard deviation in batch normalization, γ j and β j They represent the scaling factor and offset factor learned by the batch normalization layer respectively; S5.

2. Add the convolutional layer parameter matrices on all branches point by point, so that the multi-branch convolutional layer is fused into a single branch, and the multi-branch convolutional neural network model after re-parameterization is obtained. The specific expression is: Among them, W1, W2 and W3 all represent the convolutional layers in the convolutional layer parameter matrix; b represents the accumulation of offset parameters learned by the batch normalization layer.

Images