A dynamic expression recognition method and system based on multi-scale feature knowledge distillation

Abstract

The application discloses a dynamic expression recognition method based on multi-scale feature knowledge distillation, which comprises the following steps: preprocessing each video in a facial expression video library to generate facial image sequence samples and optical flow image sequence samples; constructing a teacher-student model in a knowledge distillation framework; pre-training the teacher model by using the optical flow image sequence samples; constructing a loss function of the teacher-student model, and training the student model by using the pre-trained teacher model to perform multi-scale feature knowledge distillation; and using the trained student model as a final expression recognition model to recognize the expression of a face in an input video. The method can improve the feature extraction capability of the student model, enhance the generalization capability and the identification capability of the model for dynamic expressions, and improve the accuracy and robustness of dynamic expression recognition by introducing multi-level knowledge distillation and transferring the dynamic expression features learned by the pre-trained teacher model from the optical flow image sequence to the student model.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and pattern recognition, and specifically relates to a dynamic facial expression recognition method and system based on multi-scale feature knowledge distillation. Background Technology

[0002] In daily communication, emotion serves as a crucial bridge for mutual understanding. The perception and understanding of emotions help people comprehend each other's behaviors and psychological states, playing a vital role in interpersonal interactions. With the continuous advancement of technologies such as artificial intelligence, the pursuit of more intelligent and human-computer interaction (HCI) experiences is attracting increasing attention. People have higher and higher expectations for machine intelligence, hoping that machines can perceive, understand, and even express emotions, achieving human-centered HCI interactions and better serving humanity. Facial expressions are a direct reflection of human psychological states, containing rich emotional information. Facial expression recognition, as a branch of HCI technology, has potential applications in service robots, driver fatigue detection, and other fields, and has received increasing attention in recent years.

[0003] With the development of deep learning, facial expression recognition algorithms based on deep convolutional neural networks (CNNs) have achieved good performance. However, while increasingly complex CNN models bring high performance, their huge number of parameters and computational demands consume a lot of storage and computing resources, making them difficult to deploy on mobile terminals or embedded devices with limited storage and computing resources. To save storage space and computational overhead for neural network models, researchers have proposed a deep neural network model optimization method based on knowledge distillation. By defining an appropriate distillation loss, knowledge distillation can often outperform traditional supervised training methods based on real labels, improving the model's inference accuracy. Compared with other deep neural network compression optimization methods, knowledge distillation has the advantages of not requiring complex modifications to the model structure, having a relatively simple implementation process, good generalization, and good model stability.

[0004] However, traditional knowledge distillation methods only consider the label predictions output by the teacher model as the "soft goal" for the student model's learning. They transfer knowledge to the student model by matching the output distributions of the student and teacher models, but ignore the richer hidden knowledge and information between different layers contained in the intermediate layers of the teacher model. Summary of the Invention

[0005] Purpose of the Invention: Addressing the issue that while optical flow information in facial image sequences can effectively reflect dynamic expressions, the computational complexity of optical flow is high, this invention aims to provide a dynamic expression recognition method based on multi-scale feature knowledge distillation. Knowledge distillation is performed between multiple feature levels of a teacher model (inputting optical flow image sequences) and a student model (inputting facial image sequences). By constructing a distillation loss function between multiple intermediate feature layers of the teacher and student models, the dynamic expression features learned from the optical flow image sequences and contained in the intermediate feature layers of the pre-trained teacher model are distilled into multiple feature maps of different scales in the student model. This allows the student model to extract dynamic expression features from facial image sequences without complex optical flow computation, and also learn dynamic expression features transferred from the teacher model. This enhances the model's generalization ability and its ability to identify dynamic expressions, thereby improving the accuracy and robustness of dynamic expression recognition.

[0006] Technical Solution: To achieve the above-mentioned objectives, this invention proposes a dynamic facial expression recognition method based on multi-scale feature knowledge distillation. This method includes the following steps:

[0007] S1: Preprocess each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples;

[0008] S2: Construct the teacher-student model in the knowledge distillation framework, where the student model and the teacher model use the same deep neural network structure;

[0009] S3: Pre-train the teacher model using optical flow image sequence samples;

[0010] S4: Construct the loss function of the teacher-student model. Using the pre-trained teacher model, input optical flow image sequence samples and face image sequence samples into the teacher model and student model of the teacher-student model respectively to train multi-scale feature knowledge distillation.

[0011] S5: Use the student model from the trained teacher-student model as the final expression recognition model to perform expression recognition on the faces in the input video.

[0012] Preferably, the specific steps in step S1 for preprocessing each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples are as follows:

[0013] (1.1) Segment each video in the facial expression video library according to the time sequence to generate an RGB image sequence of length K frames for each segment;

[0014] (1.2) Perform face detection on the RGB image sequence of length K frames, and crop and align the detected faces to obtain a face image sequence sample of length K frames and resolution H×W per frame, where H and W are the height and width of the face image, respectively.

[0015] (1.3) Calculate the optical flow between each two adjacent frames of a face image sequence of length K frames to generate an optical flow image sequence sample of length (K-1) frames with a resolution of H×W per frame;

[0016] (1.4) In a face image sequence of length K frames, remove the first frame face image to obtain a face image sequence sample of length (K-1) frames with a resolution of H×W for each frame.

[0017] Preferably, K is an integer, taking a value between 16 and 32.

[0018] Preferably, the student model and the teacher model in step S2 adopt the same deep neural network structure, which includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fully connected layer, and a classification layer connected in sequence.

[0019] Preferably, in step S3, the teacher model is pre-trained using optical flow image sequence samples. Specifically, the optical flow image sequence samples with a length of (K-1) frames and a resolution of H×W per frame obtained in step (1.3) are input into the teacher model. The cross-entropy loss function is used to classify and train the optical flow image sequence, and the parameters of the teacher model with the highest accuracy during the training process are frozen.

[0020] Preferably, the specific steps for constructing the loss function of the teacher-student model in step S4 are as follows:

[0021] (4.1) Construct a first feature layer distillation loss function between the first feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the first feature extraction modules of the teacher model and the student model be respectively... Where, subscript t represents the teacher model, subscript s represents the student model, superscript (1) represents the first feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (1) H represents the number of channels in the feature map extracted by the first feature extraction module. (1) and W (1) The height and width of the feature maps extracted by the first feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the first feature layer distillation loss function as:

[0022]

[0023] Among them, t (1) i,j,l For characteristic tensors One of the elements, s (1) i,j,l For characteristic tensors One of the elements;

[0024] (4.2) Construct a second feature layer distillation loss function between the second feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the second feature extraction modules of the teacher model and the student model be respectively... Wherein, the superscript (2) represents the second feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (2) H represents the number of channels in the feature map extracted by the second feature extraction module. (2) and W (2) The height and width of the feature maps extracted by the second feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the distillation loss function for the second feature layer as follows:

[0025]

[0026] in, For characteristic tensors One of the elements, For characteristic tensors One of the elements;

[0027] (4.3) Construct a third feature layer distillation loss function between the teacher model and the student model (inputting optical flow image sequence samples) and the student model (inputting face image sequence samples). Specifically, let the feature tensors output by the third feature extraction modules of the teacher and student models be respectively... Wherein, the superscript (3) represents the third feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (3) H represents the number of channels in the feature map extracted by the third feature extraction module. (3) and W (3) The height and width of the feature maps extracted by the third feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and The distillation loss function for the third feature layer is defined as follows:

[0028]

[0029] Among them, t (3) i,j,l For characteristic tensors One of the elements, s (3) i,j,l For characteristic tensors One of the elements;

[0030] (4.4) Construct a fourth feature layer distillation loss function between the fourth feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the fourth feature extraction modules of the teacher model and the student model be respectively... Wherein, the superscript (4) represents the fourth feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (4) H represents the number of channels in the feature map extracted by the fourth feature extraction module. (4) and W (4) The height and width of the feature maps extracted by the fourth feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the distillation loss function for the fourth feature layer as follows:

[0031]

[0032] Among them, t (4) i,j,l For characteristic tensors One of the elements, s (4) i,j,l For characteristic tensors One of the elements;

[0033] (4.5) Construct a fully connected layer distillation loss function between the fully connected layers of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, the logits feature vectors output by the last fully connected layer of the teacher model and the student model are defined as follows: and Where N is the number of expression categories, let v t,n v s,n Representing vectors v t v s If the logit of the nth class is given, then the predicted probability of an input sample belonging to the nth class is estimated by the softmax function:

[0034]

[0035]

[0036] Where exp(·) denotes an exponential function with base e, p t,n p s,n Let p represent the predicted probabilities of the teacher model and the student model for the input sample belonging to the nth category, respectively; for p t,n p s,n Introducing the distillation temperature factor T to calculate the predicted probability distribution of the softened class, i.e., letting

[0037]

[0038]

[0039] Here, the distillation temperature T is an integer. When T→∞, all categories have the same prediction probability. When T→0, the soft label becomes a one-hot encoded label, i.e., a hard label. The importance of each soft label is controlled by the distillation temperature T, so that knowledge distillation based on soft labels is performed between the teacher model and the student model to obtain the target function based on soft labels for the student model. The hidden knowledge contained in the soft labels obtained by the teacher model is used as supervision information to transfer knowledge from the teacher model to the student model.

[0040] The distillation loss function for fully connected layers is defined using the cross-entropy loss function:

[0041]

[0042] (4.6) The loss function for the classification layer of the teacher-student model uses the cross-entropy loss function, defined as:

[0043]

[0044] Among them, c n This represents the category label for a real facial expression. When using one-hot encoding, if the real facial expression belongs to the nth category, then c... n =1, otherwise c n =0, p n This represents the probability that the teacher-student model classification layer will predict a training sample as the nth class;

[0045] (4.7) The total loss function of the teacher-student model is defined as:

[0046] L=α1L1+α2L2+α3L3+α4L4+βL5+γL6

[0047] Among them, α1, α2, α3, α4, β, and γ are weight parameters, α1, α2, α3, α4, β, and γ are all positive numbers and α1+α2+α3+α4+β+γ=1.

[0048] Preferably, T is selected between 1 and 10.

[0049] The present invention also proposes a dynamic facial expression recognition system based on multi-scale feature knowledge distillation, comprising at least one computing device, wherein the computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded onto the processor, it implements any of the dynamic facial expression recognition methods based on multi-scale feature knowledge distillation as described above.

[0050] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the dynamic facial expression recognition methods based on multi-scale feature knowledge distillation.

[0051] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0052] (1) This invention provides a dynamic expression recognition method based on multi-scale feature knowledge distillation. Knowledge distillation is performed between multiple feature levels of a teacher model with optical flow image sequence as input and a student model with face image sequence as input. By constructing a distillation loss function between multiple intermediate feature layers of the teacher model and the student model, the feature maps of multiple different scales output by the student model are fitted to the feature maps of the corresponding scales output by the teacher model. The dynamic expression features learned from the optical flow image sequence and contained in the intermediate feature layers of the pre-trained teacher model are distilled into the feature maps of multiple different scales of the student model. This allows the student model to extract dynamic expression features from the face image sequence without complex optical flow calculations, and also learn dynamic expression features transferred from the teacher model. This improves the feature extraction ability of the student model, thereby enhancing the model's generalization ability and the ability to identify dynamic expressions, and improving the accuracy and robustness of dynamic expression recognition.

[0053] (2) This invention introduces multi-level knowledge distillation, using the dark knowledge contained in the feature maps of different scales and soft labels output by multiple intermediate feature layers of the teacher model as supervision information, to guide the training of the student model from different angles, and to transfer the dynamic expression features learned by the pre-trained teacher model from the optical flow image sequence to the student model, making the knowledge distillation more efficient.

[0054] (3) This invention uses optical flow image sequence samples to pre-train the teacher model and freezes the parameters of the pre-trained teacher model. Using the framework of the teacher-student model in knowledge distillation, it is possible to transfer effective dynamic expression features from optical flow image sequence (source domain) to face image sequence (target domain). This not only eliminates the resource overhead of the student model in optical flow calculation, but also enhances the feature extraction and generalization ability of the student model in the target domain, thereby improving the accuracy and robustness of dynamic expression recognition. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0056] Figure 1 A flowchart of a dynamic facial expression recognition method based on multi-scale feature knowledge distillation provided in this embodiment of the invention;

[0057] Figure 2 A structural diagram of the teacher-student model provided in an embodiment of the present invention;

[0058] Figure 3 A structural diagram of the teacher model provided in an embodiment of the present invention;

[0059] Figure 4 This is a structural diagram of the facial expression recognition model provided in an embodiment of the present invention. Detailed Implementation

[0060] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0061] like Figure 1 As shown in the figure, an embodiment of the present invention discloses a dynamic facial expression recognition method based on multi-scale feature knowledge distillation. This method includes the following steps: First, preprocessing each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples; then, constructing a teacher-student model within the knowledge distillation framework; second, pre-training the teacher model using the optical flow image sequence samples; next, constructing the loss function of the teacher-student model, and using the pre-trained teacher model to train the student model through multi-scale feature knowledge distillation; finally, using the student model from the trained teacher-student model as the final facial expression recognition model to perform facial expression recognition on the input video. In practical applications, the specific steps include:

[0062] (1) Preprocess each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples.

[0063] This embodiment uses the RAVDESS facial expression video library. In practice, other facial expression video libraries can also be used, or facial expression videos can be collected independently to create a facial expression video library with facial expression category labels. The RAVDESS facial expression video library used in this embodiment contains 1440 video samples, and the facial expression category in each video sample is labeled as angry, afraid, disgusted, happy, sad, surprised, furious, or neutral. The specific steps for preprocessing each video in the RAVDESS facial expression video library are as follows:

[0064] (1.1) Each video in the facial expression video library is segmented in chronological order to generate an RGB image sequence of length K frames, where K is an integer, and in this embodiment K = 17;

[0065] (1.2) Perform face detection on the RGB image sequence of length K frames, and crop and align the detected faces to obtain a face image sequence of length K frames with a resolution of H×W per frame, where H and W are the height and width of the face image, respectively. In this embodiment, H = 224 and W = 224.

[0066] (1.3) Calculate the optical flow between each two adjacent frames of a face image sequence of length K frames to generate an optical flow image sequence sample of length (K-1) frames with a resolution of H×W per frame;

[0067] (1.4) In a face image sequence of length K frames, remove the first frame face image to obtain a face image sequence sample of length (K-1) frames with a resolution of H×W for each frame.

[0068] (2) Construct the teacher-student model in the knowledge distillation framework.

[0069] The constructed teacher-student model uses the same deep neural network structure for both the student and teacher models, including sequentially connected first feature extraction modules, second feature extraction modules, third feature extraction modules, fourth feature extraction modules, fully connected layers, and classification layers, as shown below. Figure 2 As shown.

[0070] In this embodiment, the first feature extraction module, the second feature extraction module, the third feature extraction module, and the fourth feature extraction module adopt the first-stage module, the second-stage module, the third-stage module, and the fourth-stage module of the Swin Transformer network, respectively. In practical applications, the feature extraction modules can also adopt other deep neural network structures.

[0071] (3) The teacher model is pre-trained using optical flow image sequence samples.

[0072] like Figure 3 As shown, the optical flow image sequence samples with a length of (K-1) frames and a resolution of H×W per frame obtained in step (1.3) are input into the teacher model. The cross-entropy loss function is used to classify and train the optical flow image sequence, and the parameters of the teacher model with the highest accuracy during the training process are frozen.

[0073] (4) Construct the loss function for the teacher-student model. Using the pre-trained teacher model, train the student model using optical flow image sequence samples and face image sequence samples for multi-scale feature knowledge distillation. The specific steps are as follows:

[0074] (4.1) Construct a first feature layer distillation loss function between the first feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the first feature extraction modules of the teacher model and the student model be respectively... Where, subscript t represents the teacher model, subscript s represents the student model, superscript (1) represents the first feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (1) H represents the number of channels in the feature map extracted by the first feature extraction module. (1) and W (1) The height and width of the feature maps extracted by the first feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the first feature layer distillation loss function as:

[0075]

[0076] Among them, t (1) i,j,l For characteristic tensors One of the elements, s (1) i,j,l For characteristic tensors One of the elements. In this embodiment, C (1) =96,H (1) =56, W (1) =56.

[0077] (4.2) Construct a second feature layer distillation loss function between the second feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the second feature extraction modules of the teacher model and the student model be respectively... Wherein, the superscript (2) represents the second feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (2) H represents the number of channels in the feature map extracted by the second feature extraction module. (2) and W (2) The height and width of the feature maps extracted by the second feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the distillation loss function for the second feature layer as follows:

[0078]

[0079] Among them, t (2) i,j,l For characteristic tensors One of the elements, s (2) i,j,l For characteristic tensors One of the elements. In this embodiment, C (2) =192,H (2) =28, W (2) =28.

[0080] (4.3) Construct a third feature layer distillation loss function between the third feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the third feature extraction modules of the teacher model and the student model be respectively... Wherein, the superscript (3) represents the third feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (3) H represents the number of channels in the feature map extracted by the third feature extraction module. (3) and W (3) The height and width of the feature maps extracted by the third feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and The distillation loss function for the third feature layer is defined as follows:

[0081]

[0082] Among them, t (3) i,j,l For characteristic tensors One of the elements, s (3) i,j,l For characteristic tensors One of the elements. In this embodiment, C (3) =384, H (3) =14, W (3) =14.

[0083] (4.4) Construct a fourth feature layer distillation loss function between the fourth feature extraction modules of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, let the feature tensors output by the fourth feature extraction modules of the teacher model and the student model be respectively... Wherein, the superscript (4) represents the fourth feature extraction module, (K-1) is the number of frames in the optical flow image sequence or face image sequence, and C (4) H represents the number of channels in the feature map extracted by the fourth feature extraction module. (4) and W (4) The height and width of the feature maps extracted by the fourth feature extraction module are respectively subjected to global average pooling along the channel dimension to obtain feature tensors. and Define the distillation loss function for the fourth feature layer as follows:

[0084]

[0085] Among them, t (4) i,j,l For characteristic tensors One of the elements, s (4) i,j,l For characteristic tensors One of the elements. In this embodiment, C (4) =768, H (4) =7, W (4) =7.

[0086] (4.5) Construct a fully connected layer distillation loss function between the fully connected layers of the teacher model (input optical flow image sequence samples) and the student model (input face image sequence samples). Specifically, the logits feature vectors output by the last fully connected layer of the teacher model and the student model are defined as follows: and Where N is the number of expression categories, let v t,n v s,n Representing vectors v t v s If the logit of the nth class is given, then the predicted probability of an input sample belonging to the nth class is estimated by the softmax function:

[0087]

[0088]

[0089] Where exp(·) denotes an exponential function with base e, p t,n p s,n Let p represent the predicted probabilities of the teacher model and the student model for the input sample belonging to the nth category, respectively; for p t,n p s,n A distillation temperature factor T is introduced to calculate the predicted probability distribution of the softened class, i.e., let...

[0090]

[0091]

[0092] In this model, the distillation temperature T is an integer selected between 1 and 10. Higher distillation temperatures T produce a smoother distribution of predicted class probabilities. When T→∞, all classes have the same predicted probability. When T→0, the soft labels become one-hot encoded labels, i.e., hard labels. The distillation temperature T is used to control the importance of each soft label, thereby performing soft label-based knowledge distillation between the teacher model and the student model to obtain the soft label-based objective function for the student model. The hidden knowledge contained in the soft labels obtained by the teacher model is used as supervision information to transfer knowledge from the teacher model to the student model. In this embodiment, T=6 and N=8.

[0093] The distillation loss function for fully connected layers uses the cross-entropy loss function, defined as:

[0094]

[0095] (4.6) The loss function for the classification layer of the teacher-student model uses the cross-entropy loss function, defined as:

[0096]

[0097] Among them, c n This represents the category label for a real facial expression. When using one-hot encoding, if the real facial expression belongs to the nth category, then c... n =1, otherwise c n =0, p n This represents the probability that the teacher-student model classification layer will predict a training sample as the nth class.

[0098] (4.7) The total loss function of the teacher-student model is defined as:

[0099] L=α1L1+α2L2+α3L3+α4L4+βL5+γL6

[0100] Among them, α1, α2, α3, α4, β, and γ are weight parameters, α1, α2, α3, α4, β, and γ are all positive numbers, and α1+α2+α3+α4+β+γ=1. In this embodiment, α1=0.1, α2=0.1, α3=0.1, α4=0.1, β=0.3, and γ=0.3.

[0101] (5) Use the student model from the trained teacher-student model as the final facial expression recognition model, such as... Figure 4 As shown, facial expressions are recognized in the input video, and the expressions are divided into one of eight categories: angry, afraid, disgusted, happy, sad, surprised, furious, and neutral.

[0102] To verify the effectiveness of the method in this embodiment, we used the publicly available RAVDESS dataset to evaluate the performance of the proposed method, selecting 1152 videos as the training set and 288 videos as the test set.

[0103] Without introducing knowledge distillation methods, when using a dynamic facial expression recognition model based on the Swin Transformer network, the model's recognition accuracy is 55.56% when trained with optical flow image sequence samples, and 60.42% when trained with face image sequence samples.

[0104] After introducing the multi-scale feature knowledge distillation method, in order to verify the distillation effect on feature layers of different scales, we conducted ablation experiments on the RAVDESS dataset. The experimental results are shown in Table 1.

[0105] Table 1. Distillation effect on feature layers of different scales

[0106]

[0107]

[0108] Therefore, it is evident that the accuracy of facial expression recognition is significantly improved after introducing the knowledge distillation method. Table 2 shows that distillation at each scale of the feature layer plays a positive role, with varying effects at different scales. Higher-level feature layers generally exhibit better distillation results. Therefore, in practical applications, we can optimize the values ​​of the weight parameters α1, α2, α3, α4, β, and γ to further improve the accuracy of facial expression recognition.

[0109] Based on the same inventive concept, this invention discloses a dynamic facial expression recognition system based on multi-scale feature knowledge distillation, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the dynamic facial expression recognition method based on multi-scale feature knowledge distillation.

[0110] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. All technical solutions falling within the scope of the present invention's concept are within the protection scope of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic expression recognition method based on multi-scale feature knowledge distillation, characterized in that, The method comprises the following steps: S1: preprocessing each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples; S2: constructing a teacher-student model in a knowledge distillation framework, wherein the student model and the teacher model adopt the same deep neural network structure; S3: pre-training the teacher model using the optical flow image sequence samples; S4: constructing a loss function of the teacher-student model, and training the student model using the pre-trained teacher model, the optical flow image sequence samples and the facial image sequence samples for multi-scale feature knowledge distillation; S5: using the student model in the trained teacher-student model as the final expression recognition model to recognize the expression of the face in the input video; The student model and the teacher model in the step S2 adopt the same deep neural network structure, and comprise sequentially connected first, second, third and fourth feature extraction modules, a full connection layer and a classification layer; The specific steps of constructing the loss function of the teacher-student model in the step S4 are as follows: (4.1) Construct a first feature layer distillation loss function between the first feature extraction module of the teacher model inputting the optical flow image sequence sample and the first feature extraction module of the student model inputting the face image sequence sample. The specific method is: let the feature tensors output by the first feature extraction module of the teacher model and the student model be Where subscript t represents the teacher model, subscript s represents the student model, superscript (1) represents the first feature extraction module, (K-1) is the frame number of the optical flow image sequence or the face image sequence, C (1) is the channel number of the feature map extracted by the first feature extraction module, H (1) and W (1) are the height and width of the feature map extracted by the first feature extraction module respectively, and the global average pooling operation is performed on the channel dimension to obtain the feature tensors and The first feature layer distillation loss function is defined as: where t (1) i,j,l is an element of the feature tensor (1) i,j,l is an element of the feature tensor ​​ (4.2) Construct a second feature layer distillation loss function between the second feature extraction module of the teacher model inputting the optical flow image sequence sample and the second feature extraction module of the student model inputting the face image sequence sample. The specific method is: let the feature tensors output by the second feature extraction modules of the teacher model and the student model be Where the superscript (2) represents the second feature extraction module, (K-1) is the frame number of the optical flow image sequence or the face image sequence, C (2) is the channel number of the feature map extracted by the second feature extraction module, H (2) and W (2) are the height and width of the feature map extracted by the second feature extraction module, respectively, and the global average pooling operation is performed on the channel dimension to obtain the feature tensors and The second feature layer distillation loss function is defined as: where t (2) i,j,l is one element of the feature tensor (2) i,j,l is one element of the feature tensor ​​ (4.3) Constructing a third feature layer distillation loss function between the third feature extraction module of the teacher model inputting the optical flow image sequence sample and the third feature extraction module of the student model inputting the face image sequence sample, the specific method is: let the feature tensors output by the third feature extraction module of the teacher model and the student model be Where the superscript (3) represents the third feature extraction module, (K-1) is the frame number of the optical flow image sequence or the face image sequence, C (3) is the channel number of the feature map extracted by the third feature extraction module, H (3) and W (3) are the height and width of the feature map extracted by the third feature extraction module respectively, and the global average pooling operation is performed on the channel dimension to obtain the feature tensors and The third feature layer distillation loss function is defined as: where t (3) i,j,l is an element of the feature tensor (3) i,j,l is an element of the feature tensor ​​ (4.4) A fourth feature layer distillation loss function is constructed between the fourth feature extraction module of the teacher model inputting the optical flow image sequence sample and the fourth feature extraction module of the student model inputting the face image sequence sample. The specific method is: let the feature tensors output by the fourth feature extraction modules of the teacher model and the student model be wherein the superscript (4) represents the fourth feature extraction module, (K-1) is the frame number of the optical flow image sequence or the face image sequence, C (4) is the channel number of the feature map extracted by the fourth feature extraction module, H (4) and W (4) are the height and width of the feature map extracted by the fourth feature extraction module, respectively, and the global average pooling operation is performed on the channel dimension to obtain the feature tensors and The fourth feature layer distillation loss function is defined as: where t (4) i,j,l is an element of the feature tensor s (4) i,j,l is an element of the feature tensor ​ (4.5) Constructing a full connection layer distillation loss function between the full connection layer of the teacher model inputting the light flow image sequence sample and the student model inputting the human face image sequence sample, the specific method is: defining the logits feature vectors output by the last full connection layer of the teacher model and the student model as and Wherein N is the number of expression categories, let v t,n , v s,n Respectively represent the logit of the n-th category in the vector v t , v s , then the prediction probability of the input sample belonging to the n-th category is estimated by the softmax function: where exp(·) denotes the exponential function with base of the natural constant e, p t,n , p s,n denote the prediction probability of the teacher model and the student model, respectively, that the input sample belongs to the nth class; and p t,n , p s,n A distillation temperature factor T is introduced to calculate the softened class prediction probability distribution, i.e., let Wherein, the distillation temperature T is an integer, when T→∞, all classes have the same prediction probability, when T→0, the soft label becomes a one-hot encoded label, i.e. a hard label; the importance of each soft label is controlled by the distillation temperature T, so as to perform soft label-based knowledge distillation between the teacher model and the student model, obtain a soft label-based target function of the student model, and use the implicit knowledge contained in the soft label obtained by the teacher model as supervision information to migrate the knowledge from the teacher model to the student model; The full connection layer distillation loss function is defined by using a cross-entropy loss function as: (4.6) The classification layer loss function of the teacher-student model is defined by using a cross-entropy loss function as: wherein c n represents a real expression category label, when one-hot encoding is adopted, if the real expression belongs to the nth category, c n = 1, otherwise c n = 0, p n represents the probability that the teacher-student model classification layer predicts the training sample as the nth category. (4.7) The total loss function of the teacher-student model is defined as: L=α1L1+α2L2+α3L3+α4L4+βL5+γL6 Wherein, α1, α2, α3, α4, β, γ are weight parameters, α1, α2, α3, α4, β, γ are positive numbers and α1+α2+α3+α4+β+γ=1.

2. The dynamic expression recognition method based on knowledge distillation according to claim 1, characterized in that, The specific steps of preprocessing each video in the facial expression video library to generate facial image sequence samples and optical flow image sequence samples in the step S1 are as follows: (1.1) segmenting and intercepting each video in the facial expression video library according to time sequence to generate an RGB image sequence with a length of K frames; (1.2) performing face detection on the RGB image sequence with a length of K frames, and cropping and aligning the detected face to obtain a facial image sequence sample with a length of K frames and a resolution of H×W per frame, wherein H and W are the height and width of the facial image respectively; (1.3) calculating the optical flow between each adjacent two frames of the facial image sequence with a length of K frames to generate an optical flow image sequence sample with a length of (K-1) frames and a resolution of H×W per frame; (1.4) removing the first frame of face images in the face image sequence of K frames to obtain a face image sequence sample of (K-1) frames, each frame having a resolution of HxW.

3. The dynamic expression recognition method based on knowledge distillation according to claim 2, characterized in that, wherein, K is an integer, and is selected from 16 to 32.

4. The dynamic expression recognition method based on multi-scale feature knowledge distillation according to claim 1, characterized in that, In the step S3, the teacher model is pre-trained using the optical flow image sequence sample, and the specific method is as follows: the optical flow image sequence sample of (K-1) frames, each frame having a resolution of HxW, obtained in the step (1.3) is input into the teacher model, a cross-entropy loss function is used to classify and train the optical flow image sequence, and the teacher model parameters with the highest accuracy in the training process are frozen.

5. The method of claim 1, wherein the method is a dynamic expression recognition method based on multi-scale feature knowledge distillation. T is selected from 1 to 10.

6. A dynamic expression recognition system based on multi-scale feature knowledge distillation, characterized in that, The computer program is loaded into the processor to implement the dynamic expression recognition method based on multi-scale feature knowledge distillation according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer program is loaded into the processor to implement the dynamic expression recognition method based on multi-scale feature knowledge distillation according to any one of claims 1-5.

Images