Lightweight facial expression recognition framework construction method based on joint model compression

Abstract

The present application relates to the technical field of facial expression recognition, and discloses a lightweight facial expression recognition framework construction method based on joint model compression, comprising the following steps: S1, a data preprocessing stage, which is to convert original image data into a more suitable form for model training and evaluation, CK+, FER2013 dataset is adopted and divided into two parts of training set and test set; S2, a network self-distillation stage, VGG-16 model is selected as the initial model for knowledge distillation, and a sample matching regularization term is used in this stage to minimize the KL divergence between the target probability distribution and the original model prediction result. The present application designs a feasible framework for transplanting facial recognition algorithm to terminal, realizes the dynamic combination between several model compression technologies, gives full play to their respective advantages, reduces the memory space size and the calculation resource consumption of the facial expression recognition model under the condition of ensuring the model accuracy, and effectively improves the operation efficiency of the model on the device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of facial expression recognition, and more particularly to a lightweight facial expression recognition framework construction method based on joint model compression. BACKGROUND

[0002] In existing facial expression recognition algorithms, convolutional neural networks can utilize the feature extraction capability of convolutional layers and update neural network weights through a backpropagation algorithm to minimize the difference between predicted output and true output. Some commonly used convolutional neural network models have achieved good results in facial expression recognition tasks.

[0003] Complex structure models of facial expression recognition methods can usually improve the learning performance of deep learning tasks, but also bring expensive memory space and computational resource consumption due to the huge number of model parameters, making it difficult to deploy facial expression recognition technology on hardware devices. In particular, in the classification task of expression features, multiple layers of convolutional neural networks are needed to process high-dimensional image data.

[0004] In the process of lightweight facial expression recognition model, the present application uses model pruning, quantization and knowledge distillation and other model compression techniques to optimize the facial expression recognition model. In the model pruning stage, the existing technology often selects the following two methods for processing weight parameters: the first method is to pre-set a threshold value through multiple experimental results, and discard the weight parameters below the threshold value in the subsequent training process, and the second method is to introduce an L1 regularization loss function to punish the scaling coefficients of some channels, so that they tend to 0. However, L1 regularization is equally penalized to all parameters, and cannot distinguish the importance of model fitting. In addition, the traditional knowledge distillation method only focuses on the output of the entire model, ignoring the differences in different category prediction results, and is no longer suitable for facial expression recognition tasks. SUMMARY

[0005] The technical problem solved by the present application is to provide a lightweight facial expression recognition framework construction method based on joint model compression, which can solve the problem of complex model results of existing facial expression recognition algorithms and difficulty in deploying on hardware devices.

[0006] To solve the above technical problems, according to one aspect of the present application, more specifically, a lightweight facial expression recognition framework construction method based on joint model compression, comprising the following steps:

[0007] S1, data preprocessing stage, which is to convert the original image data into a more suitable form for model training and evaluation, adopt CK+, FER2013 dataset, and divide it into training set and test set two parts;

[0008] S2, a network self-distillation stage, a VGG-16 model is selected as an initial model for knowledge distillation, a sample matching regularization term is used to minimize the KL divergence between the target probability distribution and the prediction result of the original model to keep the prediction distribution consistency of the same class sample;

[0009] S3, a channel coefficient optimization stage, first, an array is set to store the sum of L1 penalty terms, then, the absolute values of all channel coefficients of the current layer are scaled and translated to obtain L1 penalty terms, and the sum of the L1 penalty terms is stored in the array, finally, the data in the array is sorted in descending order, and the threshold for channel coefficient pruning is determined;

[0010] S4, a pruning and quantization stage, according to the threshold obtained in the last stage and the pre-set pruning ratio, the model is pruned, that is, the channel coefficients less than the threshold are set to 0, and the corresponding connection is deleted to generate a pruning model, finally, the pruning model is quantized to "2".

[0011] Further, in S1, for the training set image, it is cropped to a size of 44x44 aspect ratio, and a random mirror operation is applied; for the test set image, the four corners and the center are cut, and the cut image is mirror operated to obtain 10 different images.

[0012] Further, in S2, a standard cross-entropy loss function is introduced to predict different samples, and the sum of the two loss functions is used as the loss function of the whole network self-distillation stage.

[0013] Further, in S4, since pruning may cause the model accuracy to decrease, the pruning model needs to be retrained to reach a convergence state, and sparse regularization is not needed during retraining.

[0014] Further, the method further comprises a step of verifying the model performance, 10 images cropped and randomly mirrored in the test data are input into the model for inference, and the average of the output probabilities is calculated, wherein the classification with the maximum probability is the corresponding expression.

[0015] The beneficial effects of the lightweight facial expression recognition framework construction method based on joint model compression are as follows: the traditional knowledge distillation method only focuses on the output of the whole model and ignores the different category prediction results, and therefore is not sensitive to the facial expression image recognition task. The self-distillation method based on sample matching regularization used in the present application predicts different samples of a single network, that is, the prediction of the self model is used to distill itself, which considers the difference between the prediction distributions of different samples under the same label, can significantly improve the generalization ability and calibration performance of the model, and therefore improves the performance of facial expression recognition.

[0016] In determining the pruning coefficient of the channel, the application adopts a non-convex L1 regularization function, which is an improvement over the traditional L1 regularization method. By transforming each parameter of the L1 norm penalty term, the original convex function is converted into a non-convex function. Compared with the traditional L1 regularization, the non-convex form of the constraint is more flexible and can better handle the correlation between network weight parameters. Therefore, the non-convex L1 regularization method can better reflect the importance of different parameters to the model performance and can more accurately control the penalty degree when selecting parameters. In order to reduce the storage difficulty of the model on the terminal device, the application quantizes the network parameters to low bits after the pruning stage.

[0017] In summary, the application designs a feasible framework for transplanting the face recognition algorithm to the terminal, realizes the dynamic combination of several model compression technologies, and plays their respective advantages. Under the condition of ensuring the model accuracy, the memory space size occupied by the face expression recognition model and the calculation resource consumption are reduced, and the running efficiency of the model on the device is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The application will be further described in detail below in combination with the drawings and specific implementation methods.

[0019] Figure 1 It is a method flow diagram. DETAILED DESCRIPTION

[0020] The application will be further described in detail below in combination with the drawings and specific implementation methods.

[0021] Example 1

[0022] According to an aspect of the application, as shown in Figure 1 A lightweight face expression recognition framework construction method based on joint model compression is provided, including the following stages:

[0023] The data preprocessing stage is to convert the original image data into a more suitable form for model training and evaluation. The CK+ and FER2013 data sets are used and divided into training set and test set. For the training set images, they are cropped to a size of 44x44 aspect ratio and random mirror operation is applied. For the test set images, the four corners and the center are cut, and the cut images are mirror operated to obtain 10 different images.

[0024] In the network self-distillation stage, the VGG-16 model is selected as the initial model for knowledge distillation. In this stage, the sample matching regularization term is used to minimize the KL divergence between the target probability distribution and the prediction result of the original model, so as to keep the prediction distribution consistency of the same class samples. At the same time, the standard cross-entropy loss function is introduced to predict different samples, and the sum of the two loss functions is taken as the loss function of the whole network self-distillation stage.

[0025] In the channel coefficient optimization stage, an array is first set to store the sum of the L1 penalty terms. Then, the absolute values of all channel coefficients of the current layer are scaled and translated to obtain the L1 penalty terms, which are summed by channel and stored in the array. Finally, the data in the array is sorted in descending order, and the threshold for channel coefficient pruning is determined.

[0026] In the pruning and quantization stage, according to the threshold obtained in the last stage and the pre-set pruning ratio, the model is pruned, that is, the channel coefficients less than the threshold are set to 0, and the corresponding connections are deleted to generate a pruned model. Since pruning may cause the model accuracy to decrease, the pruned model needs to be retrained to reach a convergence state. During retraining, sparse regularization is not needed. Finally, the pruned model is quantized to a low bit, and the model weights are quantized to "2".

[0027] In the model performance verification stage, 10 images that have been cropped and randomly mirrored in the test data are sent to the model for inference, and the average of the output probabilities is calculated. The classification with the highest probability is the corresponding expression.

[0028] Embodiment Two

[0029] Based on the above embodiment, in the network self-distillation stage, samples are first selected from the data set that has been cut and mirrored The corresponding true label is And the softmax classifier is used to model a posterior predictive distribution defined as:

[0030]

[0031] Where exp(·) is the exponential function with base natural constant e, f i (x; θ) represents the logit of the i-th class defined by the parameter θ, and T represents the temperature transformation coefficient. Then, in order to ensure that the prediction distribution of the same class samples remains consistent when matching, a sample matching regularization loss function is defined here. The prediction result of each sample category is regarded as an independent "prediction head", and the KL divergence is calculated between the target probability distribution and the prediction result P(y|x; θ, T) of the original model.​

[0032]

[0033] In the above equation, x' is another randomly sampled input data corresponding to the input x. KL(·) represents the KL divergence, which is used to calculate the difference between the predicted distribution of a given input and the predicted distribution of another randomly input. is a fixed copy of the parameter θ, which avoids the gradient propagating through θ and causing the model to collapse. The self-distillation loss function is defined as follows:

[0034] L SKD = L CE (x, y; θ) + ψ · T 2 · L SKL (x, x'; θ, T)

[0035] where L CE (x, y; θ) is the standard cross-entropy loss function, ψ is a factor used to balance the sample matching regularization term and the cross-entropy loss, and T 2 is the square of the temperature coefficient T. Unlike traditional knowledge distillation, which matches the same sample predictions from two networks, the self-distillation of this method is to predict between different samples of a single network, i.e., to distill the prediction of the self model.

[0036] In the channel coefficient optimization phase, a non-convex L1 regularization function is introduced to process the channel scaling coefficients, making them approach 0. For the variable The formula of the non-convex L1 regularization is as follows:

[0037]

[0038] In the formula, the L1 penalty function in non-convex form is:

[0039] Combining the above equations, this method can be represented by the following loss function:

[0040] L total = L SKD + η × P a (x)

[0041] where L SKD and P a (x) are the self-distillation and non-convex L1 regularization loss functions, respectively. In the parameter optimization phase, the model's weights are constrained to quantized weights, thereby reducing quantization errors. η is a balancing factor that limits the influence of each loss function.

[0042] Of course, the above description is not a limitation of the present application, and the present application is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the spirit and scope of the present application also belong to the protection scope of the present application.

Claims

1. A method for constructing a lightweight facial expression recognition framework based on joint model compression, characterized in that, Includes the following steps: S1, Data Preprocessing Stage: The purpose is to transform the raw image data into a more suitable form for model training and evaluation. The CK+ and FER2013 datasets are used and divided into training and testing sets. S2. In the network self-distillation stage, the VGG-16 model is selected as the initial model for knowledge distillation. In this stage, a sample matching regularization term is used to minimize the KL divergence between the target probability distribution and the prediction results of the original model, so as to maintain the consistency of the prediction distribution of the same type of samples. S3, Channel Coefficient Optimization Stage: First, an array is set up to store the sum of L1 penalty terms. Then, the absolute values ​​of all channel coefficients in the current layer are scaled and translated to obtain the L1 penalty terms. These terms are then summed by channel and stored in the array. Finally, the data in the array is sorted in descending order, and the threshold for channel coefficient clipping is determined. In this stage, a non-convex L1 regularization function is introduced to process the channel scaling coefficients, making them approach 0. For variables... The formula for non-convex L1 regularization is as follows: In the formula, the non-convex L1 penalty function is: ; S4. In the pruning and quantization stage, based on the threshold obtained in the previous stage and the pre-set pruning ratio, the model is pruned. That is, the channel coefficients less than the threshold are set to 0, and the corresponding connections are deleted to generate a pruned model. Finally, the pruned model is quantized to low bit quantization, and the model weights are quantized to "2".

2. The method for constructing a lightweight facial expression recognition framework based on joint model compression according to claim 1, characterized in that: In step S1, for the training set images, they are cropped to a size with an aspect ratio of 44×44 and a random mirroring operation is applied; for the test set images, they are cut at their four corners and center, and the cut images are mirrored to obtain 10 different images.

3. The method for constructing a lightweight facial expression recognition framework based on joint model compression according to claim 1, characterized in that: In S2, a standard cross-entropy loss function is introduced to predict different samples, and the two loss functions are summed as the loss function for the entire network's self-distillation stage.

4. The method for constructing a lightweight facial expression recognition framework based on joint model compression according to claim 1, characterized in that: In S4, since pruning may cause a decrease in model accuracy, the pruned model needs to be retrained to reach a convergent state, and sparsity regularization is not required during retraining.

5. The method for constructing a lightweight facial expression recognition framework based on joint model compression according to claim 1, characterized in that: The method also includes a step to verify the model's performance. Ten images from the test data, which have been cropped and randomly mirrored, are fed into the model for inference, and the average output probability is calculated. The category with the highest probability is the corresponding expression.

Images