A method, apparatus, device, and storage medium for recognizing dynamic facial expressions.

Abstract

This invention discloses a method, apparatus, device, and storage medium for recognizing dynamic facial expressions. The method includes: acquiring a video of an expression of a face to be recognized; extracting several facial image frames from the video; inputting all the facial image frames into a preset dynamic facial expression recognition model, so that the model performs expression recognition on each facial image frame and outputs the expression category of the face to be recognized; wherein, during training of the dynamic facial expression recognition model, a feature map extraction sub-network extracts features from several sample facial image frames of the training samples to generate feature maps for each sample facial image frame; a first network branch fuses the feature maps and inputs them into a fully connected layer; a second network branch fits the facial key point coordinates of each sample facial image frame based on the feature maps. This invention can improve the accuracy of dynamic facial expression recognition.

Description

Technical Field

[0001] This invention relates to the field of facial image recognition, and more particularly to a method, apparatus, device, and storage medium for recognizing dynamic facial expressions. Background Technology

[0002] Facial dynamic expressions generally include the following nine categories: happiness, sadness, anger, surprise, fear, disgust (blinking), nodding, and shaking. Currently, facial dynamic expression recognition technology has been widely used in scenarios such as face liveness detection, face replacement, 3D virtual human generation, and emotion analysis.

[0003] Existing dynamic expression recognition methods generally employ two types: The first type is based on video action and convolutional neural networks. This method typically involves first extracting optical flow information from facial video frames using a CNN network, then extracting surface information from still facial images using another CNN network, and finally fusing the output features of the two networks to obtain the classified expression. The second type is based on facial key points. This method typically involves pre-determining transformation rules for key points of various expressions. During recognition, multiple facial images are continuously input. First, a convolutional neural network is used to fit the facial key points, analyze the positional changes of each key point, and then complete the dynamic expression recognition based on the preset key point transformation rules.

[0004] However, both of the above methods have certain drawbacks, resulting in low accuracy and a tendency for errors in facial dynamic recognition. The first method does not utilize facial key points and fails to capture subtle facial expression features. While the second method does utilize facial key points, it is susceptible to the influence of key point detection and transformation specifications. These transformation specifications are difficult to cover, and the transformation thresholds of key points are easily affected by environmental factors such as age, gender, lighting, and occlusion. Therefore, we urgently need a method for recognizing dynamic facial expressions to improve the accuracy of facial dynamic recognition. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for recognizing dynamic facial expressions, in order to solve the technical problem of low accuracy of dynamic facial recognition results in the prior art.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for recognizing dynamic facial expressions, comprising:

[0007] Acquire a video of the facial expression of the person to be identified, and extract several facial image frames based on the video of the facial expression;

[0008] All the face image frames are input into a preset dynamic facial expression recognition model, so that the dynamic facial expression recognition model can perform expression recognition on each face image frame and output the expression category of the face to be recognized;

[0009] The facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction sub-network shared by the first network branch and the second network branch;

[0010] When training the temporal convolutional neural network model, the feature map extraction subnetwork extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame.

[0011] The first network branch fuses the feature maps and inputs them into the fully connected layer. Then, it calculates the loss for dynamic expression classification according to the first loss function and updates the network parameters according to the error backpropagation algorithm.

[0012] The second network branch fits the facial key point coordinates of each sample face image frame according to each feature map, then calculates the loss of the key point coordinate fitting according to the second loss function, and updates the network parameters according to the error backpropagation algorithm.

[0013] As a preferred embodiment, the step of extracting several facial image frames from the expression video includes:

[0014] Based on the facial expression video, locate the facial image region in the facial expression video that contains the face to be identified;

[0015] A set of face images of the face to be identified is obtained based on the face image region, and then several face image frames are extracted from the face image set.

[0016] As a preferred embodiment, the training samples include:

[0017] Videos showing various facial expressions of different people in different external environments and personal states; the expressions include: happiness, sadness, anger, surprise, fear, disgust, blinking, nodding, and shaking head.

[0018] As a preferred embodiment, before the feature map extraction subnetwork extracts features from several sample face image frames of the training samples, the method further includes:

[0019] Based on several sample face image frames from the training samples, facial key points are annotated for each sample face image frame.

[0020] In the process of facial landmark annotation, the first sample face image frame is annotated with facial landmarks. Each subsequent sample face image frame is annotated with facial landmarks based on the previous annotated sample face image frame, and the facial landmarks are adjusted to complete the annotation.

[0021] As a preferred embodiment, the first loss function includes: a cross-entropy loss function;

[0022] The second loss function includes: the squared error loss function;

[0023] The loss for fitting keypoint coordinates is calculated using the following formula:

[0024]

[0025] Where N is the number of training samples, and M is the number of face image frames extracted from each expression video, (x i,j,k ,y i,j,k Let be the coordinates of the k-th keypoint labeled in the j-th frame of the face image of the i-th training sample. The coordinates of the k-th keypoint corresponding to the j-th frame of the face image of the i-th training sample predicted by the second network branch.

[0026] Based on the above embodiments, another embodiment of the present invention provides a facial dynamic expression recognition device, characterized in that it includes: a facial image frame extraction module, a facial dynamic expression recognition module, and a model training module;

[0027] The face image frame extraction module is used to acquire the expression video of the face to be identified, and extract several face image frames based on the expression video.

[0028] The facial dynamic expression recognition module is used to input all the facial image frames into a preset facial dynamic expression recognition model, so that the facial dynamic expression recognition model can perform expression recognition on each facial image frame and output the expression category of the face to be recognized; wherein, the facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction subnetwork shared by the first network branch and the second network branch;

[0029] The model training module is used to train the temporal convolutional neural network model. During training, the feature map extraction sub-network extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame. The first network branch fuses the feature maps and inputs them into a fully connected layer, then calculates the loss for dynamic expression classification based on a first loss function, and updates the network parameters according to the backpropagation algorithm. The second network branch fits the facial keypoint coordinates of each sample face image frame based on the feature maps, then calculates the loss for keypoint coordinate fitting based on a second loss function, and updates the network parameters according to the backpropagation algorithm.

[0030] As a preferred embodiment, the facial dynamic expression recognition device further includes: a facial key point annotation module;

[0031] The facial landmark annotation module is used to annotate facial landmarks for each of the several sample facial image frames in the training samples. Specifically, when annotating facial landmarks, the first sample facial image frame is annotated first, and each subsequent sample facial image frame is annotated based on the previous annotated sample facial image frame, with the facial landmarks adjusted to complete the annotation.

[0032] Based on the above embodiments, another embodiment of the present invention provides a facial dynamic expression recognition device, the device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the facial dynamic expression recognition method described in the above embodiments of the invention.

[0033] Based on the above embodiments, another embodiment of the present invention provides a storage medium, the storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the facial dynamic expression recognition method described in the above embodiments of the invention.

[0034] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0035] This invention uses a pre-defined dynamic facial expression recognition model to perform expression recognition on each facial image frame in a video of the face to be identified, thereby obtaining the expression category of the face to be identified. Unlike existing technologies that first fit facial key point localization information and then use facial key point transformation rules to identify dynamic facial expressions, this invention's dynamic facial expression recognition model uses two network branches and a shared feature map extraction subnetwork to simultaneously achieve dynamic facial expression recognition and facial key point localization. Because the two branch networks share the feature map extraction subnetwork, the loss in facial key point coordinate fitting affects dynamic facial expression classification, allowing facial key point localization information to influence dynamic expression recognition. Similarly, the loss in dynamic facial expression classification also affects facial key point coordinate fitting. This dynamically unifies facial key point localization information and dynamic facial expression recognition, avoiding the need to manually formulate facial key point transformation rules to match and identify dynamic facial expressions, thus improving the accuracy of dynamic facial expression recognition. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a method for recognizing dynamic facial expressions according to an embodiment of the present invention.

[0037] Figure 2 This is a network framework diagram of the temporal convolutional neural network model of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a facial dynamic expression recognition device provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1

[0041] Please refer to Figure 1 The following is a flowchart illustrating a method for recognizing dynamic facial expressions according to an embodiment of the present invention, comprising the following specific steps:

[0042] S1. Obtain the facial expression video of the person to be identified, and extract several facial image frames based on the facial expression video.

[0043] Preferably, the step of extracting several face image frames from the expression video includes: locating a face image region in the expression video that contains the face to be identified; obtaining a set of face images of the face to be identified based on the face image region; and then extracting several face image frames from the set of face images.

[0044] For the acquired facial expression video of the person to be identified, the facial image region containing the face to be identified is first located in the facial expression video using a preset facial detection model. A set of facial images of the face to be identified is then extracted from the facial image region, and several facial image frames are extracted from the set of facial images. All of the facial image frames are used as input to the facial dynamic expression recognition model.

[0045] S2. Input all the face image frames into a preset face dynamic expression recognition model, so that the face dynamic expression recognition model can perform expression recognition on each face image frame and output the expression category of the face to be recognized;

[0046] The facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction sub-network shared by the first network branch and the second network branch;

[0047] When training the temporal convolutional neural network model, the feature map extraction subnetwork extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame.

[0048] The first network branch fuses the feature maps and inputs them into the fully connected layer. Then, it calculates the loss for dynamic expression classification according to the first loss function and updates the network parameters according to the error backpropagation algorithm.

[0049] The second network branch fits the facial key point coordinates of each sample face image frame according to each feature map, then calculates the loss of the key point coordinate fitting according to the second loss function, and updates the network parameters according to the error backpropagation algorithm.

[0050] Preferably, the training samples include: several facial expression videos of different people in different external environments and in different states; wherein the expressions include: happiness, sadness, anger, surprise, fear, disgust, blinking, nodding and shaking.

[0051] The facial dynamic expression recognition model is a temporal convolutional neural network model. When training this model, the training samples are several video clips of different faces displaying various facial expressions under different external environments and personal states. In one specific embodiment, video clips of nine facial expressions (happiness, sadness, anger, surprise, fear, disgust, blinking, nodding, and shaking head) are collected from different faces under different lighting, postures, and age conditions. Six facial image frames are extracted from each of these collected video clips according to certain rules, and all these facial image frames are used as training samples for the temporal convolutional neural network model.

[0052] Preferably, before the feature map extraction subnetwork extracts features from several sample face image frames of the training samples, the method further includes: annotating facial key points for each sample face image frame based on several sample face image frames of the training samples; wherein, when annotating facial key points, the first sample face image frame is annotated first, and each subsequent sample face image frame is annotated based on the previous annotated sample face image frame, and the facial key points are adjusted to complete the facial key point annotation.

[0053] After obtaining the training samples of the model, in the data annotation stage of the training samples, the facial key point data information corresponding to each face image frame is annotated. In the process of annotating the facial key point data, in a specific embodiment, for 6 face image frames of each expression, the first face image frame is first annotated with facial key points, and the data information of each of the 128 key points of the face is annotated. Each subsequent face image frame is adjusted based on the previous face image frame after the annotation is completed, and the annotation of the facial key point data information is completed.

[0054] This method of sample data annotation, except for the first sample face image frame, adjusts each subsequent sample face image frame based on the previously annotated sample face image frame. It makes full use of the temporal information between different face image frames in dynamic expression videos, and the features of the previous and subsequent frames influence each other. Finally, during model training, the model integrates the features of 6 face image frames by connecting the feature maps of each face image frame, making it easier to capture expression changes.

[0055] After completing the training sample labeling, please refer to the following during the model training phase: Figure 2 This is a network framework diagram of the temporal convolutional neural network model of the present invention, which includes a first network branch, a second network branch, and a feature map extraction subnetwork shared by the first network branch and the second network branch. In a specific embodiment, such as... Figure 2In the temporal convolutional neural network model shown, the first half is a shared feature map extraction sub-network. This sub-network extracts features from 6 sample face image frames of each expression video, generating feature maps corresponding to each sample face image frame. The second half consists of a first network branch and a second network branch. The first network branch fuses the features of each feature map by superposition. After fusion, the fused features are input into a fully connected layer (FC). The first loss function (softmax loss, cross-entropy loss function) is then used to calculate the loss for dynamic expression classification, and the network parameters are updated using the backpropagation algorithm. Simultaneously, the second network branch fits the facial keypoint coordinates of each sample face image frame based on the feature map corresponding to each sample face image frame. The second loss function (L2 loss, squared error loss function) is then used to calculate the loss for keypoint coordinate fitting, and the network parameters are updated using the backpropagation algorithm until the temporal convolutional neural network model converges as a whole.

[0056] Unlike existing technologies that first fit facial landmark localization information and then use facial landmark transformation rules to identify facial dynamic expressions, this invention performs facial dynamic expression recognition through a first network branch and facial landmark localization through a second network branch during model training. Therefore, facial dynamic expression recognition and facial landmark localization are achieved simultaneously during model training. Furthermore, since the two branches share a feature map extraction sub-network, the loss of facial landmark coordinate fitting in the second network branch affects the parameter calculation of the feature map extraction sub-network, thereby affecting the facial dynamic expression classification in the first network branch. This allows facial landmark localization information to influence dynamic expression recognition, and similarly, the loss of facial dynamic expression classification also affects facial landmark coordinate fitting. This dynamically unifies facial landmark localization information and facial dynamic expression recognition, avoiding the need to manually formulate facial landmark transformation rules to match and identify facial dynamic expressions. This improves the algorithm's robustness to factors such as lighting, age, and gender, and ultimately enhances the accuracy of facial dynamic expression recognition.

[0057] Preferably, the first loss function includes a cross-entropy loss function; the second loss function includes a squared error loss function; and the loss for keypoint coordinate fitting is calculated using the following formula:

[0058]

[0059] Where N is the number of training samples, and M is the number of face image frames extracted from each expression video, (x i,j,k ,y i,j,k Let be the coordinates of the k-th keypoint labeled in the j-th frame of the face image of the i-th training sample. The coordinates of the k-th keypoint corresponding to the j-th frame of the face image of the i-th training sample predicted by the second network branch.

[0060] In the temporal convolutional neural network model, the first loss function of the first network branch is the softmax loss function, which is a combination of the softmax function and the cross-entropy loss function, specifically the cross-entropy loss function, used to measure the loss of dynamic expression classification in the first network branch. The second loss function of the second network branch is the L2 loss function, which is the squared error loss function, used to measure the loss of keypoint coordinate fitting in the second network branch. The calculation formula for the second loss function is as follows:

[0061]

[0062] In another specific embodiment, the facial dynamic expression recognition method can be applied in the field of liveness detection. Dynamic expression recognition is used to determine whether the user is real. For the input video data, 6 frames of images are extracted at equal time intervals, and each image is resized to 224×224 as the input of the model. After model inference, the expression category is output. By judging whether the expression is consistent with the required expression, the liveness is detected.

[0063] Therefore, this invention provides a method for recognizing dynamic facial expressions. The dynamic facial expression recognition model of this method uses two network branches and a feature map extraction subnetwork shared by the two network branches to simultaneously realize dynamic facial expression recognition and facial key point localization. Since the two branch networks share the feature map extraction subnetwork, the loss of facial key point coordinate fitting will affect the classification of dynamic facial expressions, allowing facial key point localization information to influence the recognition of dynamic expressions. Similarly, the loss of dynamic facial expression classification will also affect the fitting of facial key point coordinates. This achieves dynamic unification of facial key point localization information and dynamic facial expression recognition, avoiding the method of matching and recognizing dynamic facial expressions by manually formulating rules for changes in facial key points, thus improving the accuracy of dynamic facial expression recognition.

[0064] Example 2

[0065] Please refer to Figure 3 This is a schematic diagram of the structure of a facial dynamic expression recognition device according to an embodiment of the present invention. The device includes: a facial image frame extraction module, a facial dynamic expression recognition module, and a model training module.

[0066] The face image frame extraction module is used to acquire the expression video of the face to be identified, and extract several face image frames based on the expression video.

[0067] The facial dynamic expression recognition module is used to input all the facial image frames into a preset facial dynamic expression recognition model, so that the facial dynamic expression recognition model can perform expression recognition on each facial image frame and output the expression category of the face to be recognized; wherein, the facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction subnetwork shared by the first network branch and the second network branch;

[0068] The model training module is used to train the temporal convolutional neural network model. During training, the feature map extraction sub-network extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame. The first network branch fuses the feature maps and inputs them into a fully connected layer, then calculates the loss for dynamic expression classification based on a first loss function, and updates the network parameters according to the backpropagation algorithm. The second network branch fits the facial keypoint coordinates of each sample face image frame based on the feature maps, then calculates the loss for keypoint coordinate fitting based on a second loss function, and updates the network parameters according to the backpropagation algorithm.

[0069] Preferably, the facial dynamic expression recognition device further includes: a facial key point annotation module;

[0070] The facial landmark annotation module is used to annotate facial landmarks for each of the several sample facial image frames in the training samples. Specifically, when annotating facial landmarks, the first sample facial image frame is annotated first, and each subsequent sample facial image frame is annotated based on the previous annotated sample facial image frame, with the facial landmarks adjusted to complete the annotation.

[0071] Example 3

[0072] Accordingly, this invention provides a facial dynamic expression recognition device, the device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the facial dynamic expression recognition method described in the above-described invention.

[0073] Example 4

[0074] Accordingly, embodiments of the present invention provide a storage medium, the storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the facial dynamic expression recognition method described in the above embodiments of the invention.

[0075] In summary, the present invention provides a facial dynamic expression recognition device, equipment, and storage medium, which can achieve dynamic unification of facial key point positioning information and facial dynamic expression recognition, avoiding the method of matching and recognizing facial dynamic expressions by manually formulating rules for changes in facial key points, and improving the accuracy of facial dynamic expression recognition.

[0076] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0077] Those skilled in the art will clearly understand that, for convenience and simplicity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0078] The device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The device may include, but is not limited to, processors and memory.

[0079] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the device, connecting various parts of the device via various interfaces and lines.

[0080] The memory can be used to store the computer program. The processor implements various functions of the device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0081] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0082] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for recognizing dynamic facial expressions, characterized in that, include: Acquire a video of the facial expression of the person to be identified, and extract several facial image frames based on the video of the facial expression; All the face image frames are input into a preset dynamic facial expression recognition model, so that the dynamic facial expression recognition model can perform expression recognition on each face image frame and output the expression category of the face to be recognized; The facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction sub-network shared by the first network branch and the second network branch; When training the temporal convolutional neural network model, the feature map extraction subnetwork extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame. The first network branch fuses the feature maps and inputs them into the fully connected layer. Then, it calculates the loss for dynamic expression classification based on the first loss function and updates the network parameters according to the backpropagation algorithm. The second network branch fits the facial key point coordinates of each sample face image frame according to each feature map, then calculates the loss of key point coordinate fitting according to the second loss function, and updates the network parameters according to the error backpropagation algorithm. Wherein, the first loss function includes: cross-entropy loss function; The second loss function includes: the squared error loss function; The loss for fitting keypoint coordinates is calculated using the following formula: Where N is the number of training samples, and M is the number of face image frames extracted from each expression video. , Let K be the coordinates of the k-th keypoint labeled in the j-th frame of the face image of the i-th training sample. , The coordinates of the k-th keypoint corresponding to the j-th frame of the face image of the i-th training sample predicted by the second network branch.

2. The method for recognizing dynamic facial expressions as described in claim 1, characterized in that, The step of extracting several facial image frames from the facial expression video includes: Based on the facial expression video, locate the facial image region in the facial expression video that contains the face to be identified; A set of face images of the face to be identified is obtained based on the face image region, and then several face image frames are extracted from the face image set.

3. The method for recognizing dynamic facial expressions as described in claim 1, characterized in that, The training samples include: Videos showing various facial expressions of different people in different external environments and personal states; the expressions include: happiness, sadness, anger, surprise, fear, disgust, blinking, nodding, and shaking head.

4. The method for recognizing dynamic facial expressions as described in claim 1, characterized in that, Before the feature map extraction subnetwork extracts features from several sample face image frames of the training samples, the following is also included: Based on several sample face image frames from the training samples, facial key points are annotated for each sample face image frame. In the process of facial landmark annotation, the first sample face image frame is annotated with facial landmarks. Each subsequent sample face image frame is annotated with facial landmarks based on the previous annotated sample face image frame, and the facial landmarks are adjusted to complete the annotation.

5. A facial dynamic expression recognition device, characterized in that, include: The system includes a face image frame extraction module, a face dynamic expression recognition module, and a model training module. The face image frame extraction module is used to acquire the expression video of the face to be identified, and extract several face image frames based on the expression video. The facial dynamic expression recognition module is used to input all the facial image frames into a preset facial dynamic expression recognition model, so that the facial dynamic expression recognition model can perform expression recognition on each facial image frame and output the expression category of the face to be recognized; wherein, the facial dynamic expression recognition model is a temporal convolutional neural network model; the temporal convolutional neural network model includes: a first network branch, a second network branch, and a feature map extraction subnetwork shared by the first network branch and the second network branch; The model training module is used to train the temporal convolutional neural network model. During training, the feature map extraction sub-network extracts features from several sample face image frames of the training samples to generate feature maps for each sample face image frame. The first network branch fuses the feature maps and inputs them into a fully connected layer, then calculates the loss for dynamic expression classification based on a first loss function, and updates the network parameters according to the backpropagation algorithm. The second network branch fits the facial keypoint coordinates of each sample face image frame based on the feature maps, then calculates the loss for keypoint coordinate fitting based on a second loss function, and updates the network parameters according to the backpropagation algorithm. Wherein, the first loss function includes: cross-entropy loss function; The second loss function includes: the squared error loss function; The loss for fitting keypoint coordinates is calculated using the following formula: Where N is the number of training samples, and M is the number of face image frames extracted from each expression video. , Let K be the coordinates of the k-th keypoint labeled in the j-th frame of the face image of the i-th training sample. , The coordinates of the k-th keypoint corresponding to the j-th frame of the face image of the i-th training sample predicted by the second network branch.

6. The facial dynamic expression recognition device as described in claim 5, characterized in that, Also includes: Facial landmark annotation module; The facial landmark annotation module is used to annotate facial landmarks for each of the several sample facial image frames in the training samples. Specifically, when annotating facial landmarks, the first sample facial image frame is annotated first, and each subsequent sample facial image frame is annotated based on the previous annotated sample facial image frame, with the facial landmarks adjusted to complete the annotation.

7. A facial dynamic expression recognition device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method for recognizing dynamic facial expressions as described in any one of claims 1 to 4.

8. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the facial dynamic expression recognition method as described in any one of claims 1 to 4.

Images