A dynamic expression recognition method, electronic device and computer-readable storage medium

Abstract

The present invention discloses a dynamic expression recognition method, an electronic device and a computer-readable storage medium, which are intended to use a self-supervised method to effectively extract facial representations with temporal consistency and spatial consistency in a video. The method steps are: sampling the input video, inputting the sampled video segments into an online learning module and a momentum learning module respectively, and performing data enhancement on the input video; in the momentum learning module and the online learning module, performing three-dimensional convolution on each input video segment to obtain a dense feature map with rich temporal and spatial semantics; the prior knowledge module combines the video label with the facial action unit as prior knowledge to obtain a prior knowledge feature vector; the prior knowledge feature vector is used as a facial feature query to perform feature fusion with the dense feature map, and finally a facial weighted feature map is obtained; self-supervised learning is performed; the output dynamic facial features with temporal and spatial consistency are input into a pre-trained dynamic emotion recognizer for emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a dynamic expression recognition method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, facial expression recognition technology based on deep learning has received widespread attention in various applications. The main goal of facial expression recognition is to automatically identify an individual's emotional state by analyzing facial movements and changes in facial images or video sequences. However, due to the dynamic and complex nature of facial expressions, recognition methods based only on static images are difficult to fully capture the temporal and spatial characteristics of facial expressions, especially when recognizing subtle emotional changes.

[0003] At present, most traditional facial expression recognition methods rely on a large amount of pre-annotated video data for supervised learning. However, the cost of annotating videos is high and data acquisition is difficult. At the same time, the subjectivity of manual annotation may also cause the representation learned by the model to be not robust enough. In addition, the image analysis of a single frame cannot capture the emotional change information in the temporal dimension, so these methods face many challenges in practical applications. How to make full use of unsupervised or self-supervised learning methods with limited annotated data to extract temporally and spatially consistent dynamic facial features from video data is an important direction of current research. Summary of the invention

[0004] In order to make up for the shortcomings of the prior art, the present invention provides a dynamic expression recognition method, an electronic device and a computer-readable storage medium. The recognition method adopts a combination of a self-supervised learning model and a 3D convolutional network, and introduces prior knowledge such as AU in the feature extraction process, and innovatively proposes a construction paradigm for spatiotemporal feature extraction and feature consistency in facial expression recognition.

[0005] The technical problem solved by the present invention can be achieved through the following specific technical solutions:

[0006] On the one hand, a method for dynamic expression recognition is provided, comprising the following steps:

[0007] Step 1: Sample the input video into n video segments with reference to the time step of expression changes, then input the sampled video segments into the online learning module and momentum learning module respectively, and perform multiple data enhancement strategies on the input video;

[0008] Step 2: In the momentum learning module and the online learning module, three-dimensional convolution is performed on each input video segment to obtain a dense feature map with spatiotemporal semantics;

[0009] Step 3: The prior knowledge module combines the video label with the facial action unit AU as prior knowledge to obtain a prior knowledge feature vector;

[0010] Step 4: The facial feature fusion module uses the prior knowledge feature vector as a facial feature query and performs feature fusion with the dense feature map to finally obtain a facial weighted feature map;

[0011] Step 5: Use the attention encoder to process the facial weighted feature map to obtain the attention mask, calculate the contrast loss between the attention masks obtained by the two encoders and the facial weighted feature map, and calculate the similarity loss between the facial weighted feature maps output by the two encoders to perform a self-supervised learning process;

[0012] Step 6: After the self-supervised learning converges, the output dynamic facial features with temporal and spatial consistency can be input into the pre-trained dynamic emotion recognizer for emotion recognition.

[0013] Furthermore, in step 1, the data enhancement strategy is to randomly select a data enhancement strategy from the picture space cropping and radiation color enhancement and use it before each video segment is input.

[0014] Furthermore, in step 2, during the training of the online learning module and the momentum learning module, it is assumed that the total number of video segments obtained from step 1 is , then each video segment is input into the online learning module as a query sample. The reference samples are input into the momentum learning module.

[0015] Furthermore, in step 2, the parameters of the momentum learning module are updated by performing exponential moving average on the parameters of the online learning module, and the formula is:

[0016] ,

[0017] in, Represents the parameters of the momentum learning module; Represents the parameters of the online learning module; Represents a momentum coefficient close to 1, with a value of 0.99.

[0018] Furthermore, three-dimensional convolution refers to convolution in the time dimension and space dimension. Assuming that the tensor of the input video is , whose dimensions are , respectively represent the frame number ,high ,width and number of input channels ; Let the convolution kernel be , whose size is , indicating the depth ,high ,width , Number of input channels and number of output channels , the formula is:

[0019] ,

[0020] in, is the output dense feature map with dimension ; Indicates the location of the output feature map; are the indices of the convolution kernel in depth, height, and width, respectively; is the input channel index; is the output channel index.

[0021] Furthermore, in step 3, the prior knowledge module uses AU as prior knowledge related to expression, combines the labels of the expression videos in the data set as the classification of facial actions in the video segments, forms various query statements for facial components, inputs the query statements into the CLIP text encoder, and obtains the prior knowledge feature vector.

[0022] Furthermore, the facial feature fusion module in step 4 uses the Transformer Decoder architecture to fuse the dense feature map with the prior knowledge feature vector, and its formula is:

[0023] ,

[0024] in, is the weighted feature map of the face; is the prior knowledge feature vector; Indicates that From 4D tensor Expand into ,in .

[0025] Furthermore, the attention encoder of step 5 includes two multi-layer perceptrons MLP, which process the facial weighted feature map to finally obtain the attention mask ;

[0026] The contrast loss is calculated using InfoNCE loss, making similar attention masks With the weighted facial feature map Close, dissimilar, far away, is used to supervise the attention encoder's perception of facial features, and its formula is:

[0027] ,

[0028] in, is the temperature parameter, which is used to control the sensitivity to similarity; represents the similarity measure, and its formula is:

[0029] ;

[0030] The similarity loss is calculated using MSE loss, and its formula is:

[0031] ,

[0032] in, and It is the output of the momentum learning module and the online learning module. A facial weighted feature map; is the total number of feature maps.

[0033] On the other hand, an electronic device is provided, comprising one or more processors and a memory, wherein the memory stores one or more programs, and the programs include instructions for executing a dynamic expression recognition method as described above.

[0034] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the programs include instructions for executing a dynamic expression recognition method as described above.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) The present invention adopts a self-supervised learning model. Through the synergy of the online learning module and the momentum learning module, it can learn robust facial representations from data without explicit negative sample comparison. This architecture reduces the dependence on a large amount of labeled data and effectively captures the core features of dynamic facial expressions through the momentum update strategy, thereby improving the robustness and generalization ability of the model.

[0037] (2) By processing the video segments through a three-dimensional convolutional network, the model can simultaneously extract dynamic expression features from the temporal and spatial dimensions and capture richer spatiotemporal semantic information. Combining the fusion of AU prior knowledge feature vectors and the dynamic mask generated by the attention encoder, the model can learn consistent facial representations at different spatiotemporal steps, thereby ensuring the continuity and integrity of the features.

[0038] (3) The present invention adopts the joint optimization of contrast loss and similarity loss, distinguishes different expression features through contrast loss, and ensures the high consistency of facial features through similarity loss. This optimization design effectively improves the accuracy of feature learning, so that the dynamic facial features output by the model can be more accurately input into the pre-trained emotion recognizer, improves the recognition effect and reduces the dependence on data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the process of the dynamic expression recognition method of the present invention;

[0040] Figure 2 A schematic diagram of the video data processing process of the present invention;

[0041] Figure 3 It is a structural diagram of the prior knowledge module of the present invention;

[0042] Figure 4 Schematic diagram of the training process of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Example 1

[0044] like Figure 1 As shown, a dynamic expression recognition method comprises the following steps:

[0045] Step 1: Sample the input video into n video segments based on the time step of expression changes, input the sampled video segments into the online learning module and momentum learning module respectively, and perform multiple data enhancement strategies on the input video.

[0046] The sampling process used in this embodiment needs to extract video segments with reference to the time step of expression changes. According to the previous study "Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms", the duration of facial expression changes is between 0.5s and 4s, and the video frame rate is generally 24 frames per second. Therefore, this sampling takes 2 frames as the step length and selects 32 frames as a video segment. A total of 4 videos are selected. .

[0047] The data enhancement strategy used in this embodiment refers to the paper "A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning". A data enhancement strategy is randomly selected from the picture space cropping and radiation color enhancement before each video segment is input. Figure 2 As shown in is the input video data after data augmentation.

[0048] Step 2: In the momentum learning module and the online learning module, three-dimensional convolution is performed on each input video segment to obtain a dense feature map with spatiotemporal semantics.

[0049] During the training of the online learning module and the momentum learning module, assume that the total number of video segments obtained from step 1 is , then each video segment is input into the online learning module as a query sample. The reference samples are input into the momentum learning module. The parameters of the momentum learning module are updated by performing exponential moving average on the parameters of the online learning module, and the formula is:

[0050] ,

[0051] in, Represents the parameters of the momentum learning module; Represents the parameters of the online learning module; Represents a momentum coefficient close to 1, with a value of 0.99. This update method means that the momentum network parameters are updated very slowly, and it will maintain a more stable feature representation during training; the online network will learn and adjust according to the output of the momentum network.

[0052] Three-dimensional convolution refers to convolution in the time dimension and space dimension. Assume that the tensor of the input video is , whose dimensions are , respectively represent the frame number ,high ,width and number of input channels ; Let the convolution kernel be , whose size is , indicating the depth ,high ,width , Number of input channels and number of output channels , the formula is:

[0053] ,

[0054] in, is the output dense feature map, with dimension ; Indicates the location of the output feature map; are the indices of the convolution kernel in depth, height, and width, respectively; is the input channel index; is the output channel index.

[0055] Step 3: The prior knowledge module combines the video label with the facial action unit AU as prior knowledge to obtain a prior knowledge feature vector.

[0056] like Figure 3 As shown in the figure, it is a schematic diagram of the structure of the prior knowledge module. The input is the emotion label text and the facial action unit description text, and the output is the prior knowledge feature vector The prior knowledge module uses AU as prior knowledge related to expression, and combines the labels of expression videos in the dataset as the final classification of facial actions in the video segment to form various query statements for facial components. The query statements are input into the CLIP text encoder to obtain the prior knowledge feature vector.

[0057] Step 4: The facial feature fusion module uses the prior knowledge feature vector as the facial feature query and performs feature fusion with the dense feature map to finally obtain a facial weighted feature map.

[0058] like Figure 4 As shown, it is a schematic diagram of the training process of the present invention, and the input is Figure 2 The facial representation shown , the output is the attention mask The facial feature fusion module of this application uses the Transformer Decoder architecture to fuse the prior knowledge feature vector as the facial feature query with the dense feature map, and finally obtains the facial weighted feature map, and its formula is:

[0059] ,

[0060] in, is the weighted feature map of the face; is the prior knowledge feature vector; Indicates that From 4D tensor Expand into ,in .

[0061] Step 5: Use the attention encoder to process the facial weighted feature map to obtain the attention mask, calculate the contrast loss between the attention masks obtained by the two encoders and the facial weighted feature map, and calculate the similarity loss between the facial weighted feature maps output by the two encoders to perform a self-supervised learning process.

[0062] The attention encoder consists of two multi-layer perceptrons (MLPs) that process the weighted facial feature map and finally obtain the attention mask. ; The contrast loss is calculated using InfoNCE loss, making similar attention masks With the weighted facial feature map Close, dissimilar, far away, is used to supervise the attention encoder's perception of facial features, and its formula is:

[0063] ,

[0064] in, is the temperature parameter, which is used to control the sensitivity to similarity; represents the similarity measure, and its formula is:

[0065] ;

[0066] In addition, the similarity loss is calculated using MSE loss, and its formula is:

[0067] ,

[0068] in, and It is the output of the momentum learning module and the online learning module. A facial weighted feature map; is the total number of feature maps.

[0069] Step 6: After the self-supervised learning converges, the output dynamic facial features with temporal and spatial consistency can be input into the pre-trained dynamic emotion recognizer for emotion recognition.

[0070] In summary, the method of the present invention samples the input video in time steps, divides it into multiple video segments, and performs multiple data enhancement processing in the online learning and momentum learning modules; then, a dense feature map with spatiotemporal semantics is extracted through a three-dimensional convolutional network, and a priori knowledge feature vector is generated by combining video labels and facial action unit (AU) information; the facial feature fusion module fuses these prior knowledge with the dense feature map to obtain a facial weighted feature map; further, an attention mask of dynamic facial features is generated through an attention encoder, and the spatiotemporal consistency and robustness of the features are optimized by combining contrast loss and similarity loss, thereby realizing self-supervised learning; finally, the dynamic facial features after training convergence can be used for efficient emotion recognition. Example 2

[0071] This embodiment proposes an electronic device, which includes one or more processors and a memory, wherein the memory stores one or more programs, and these programs include instructions for executing a dynamic expression recognition method as described in Example 1. Example 3

[0072] This embodiment provides a computer-readable storage medium, which contains one or more programs for execution by one or more processors of an electronic device. These programs include instructions for executing a dynamic expression recognition method as described in Embodiment 1.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic expression recognition, characterized in that: The following steps are involved: Step 1: Sample the input video into n video segments with reference to the time step of expression changes, input the sampled video segments into the online learning module and the momentum learning module respectively, and perform data enhancement strategy on the input video; The data augmentation strategy is to randomly select one of the data augmentation strategies from the picture space cropping and radiometric color enhancement and use it before each video segment is input; Step 2: In the momentum learning module and the online learning module, three-dimensional convolution is performed on each input video segment to obtain a dense feature map with spatiotemporal semantics; the parameters of the momentum learning module are updated by performing exponential moving average on the parameters of the online learning module, and the formula is: ξ←τξ+(1-τ)θ, Among them, ξ represents the parameters of the momentum learning module; θ represents the parameters of the online learning module; τ represents a momentum coefficient close to 1, with a value of 0.99; Three-dimensional convolution refers to convolution in the time dimension and space dimension. Assume that the input video tensor is X, and its dimension is D×H×W×C in , respectively represent the number of frames D, height H, width W and the number of input channels C in ; Let the convolution kernel be K, and its size be k d ×k h ×k w ×C in ×C out , represents the depth k d , height k h , width k w 、Number of input channels C in and the number of output channels C out , the formula is: Among them, Y is the output dense feature map, the dimension is D out ×H out ×W out ×C out ; (d, h, w) represents the position of the output feature map; i, j, l are the indices of the convolution kernel in depth, height and width respectively; m is the input channel index; c is the output channel index; Step 3: The prior knowledge module combines the video label with the facial action unit AU as prior knowledge to obtain a prior knowledge feature vector; Step 4: The facial feature fusion module uses the prior knowledge feature vector as a facial feature query and performs feature fusion with the dense feature map to finally obtain a facial weighted feature map; Step 5: Use the attention encoder to process the facial weighted feature map to obtain the attention mask, calculate the contrast loss between the attention masks obtained by the two encoders and the facial weighted feature map, calculate the similarity loss of the facial weighted feature map, and perform a self-supervised learning process; the attention encoder includes two multi-layer perceptrons MLP, which process the facial weighted feature map and finally obtain the attention mask The contrast loss is calculated using InfoNCE loss, making similar attention masks Close to the facial weighted feature map Z and far away from the dissimilar one, it is used to supervise the attention encoder's perception of facial features. The formula is: Among them, τ is the temperature parameter used to control the sensitivity to similarity; sim(·,·) represents the similarity measure, and its formula is: The similarity loss is calculated using MSE loss, and its formula is: in, and is the i-th facial weighted feature map output by the momentum learning module and the online learning module; N is the total number of feature maps; Step 6: After the self-supervised learning converges, the output dynamic facial features with temporal and spatial consistency are input into the pre-trained dynamic emotion recognizer for emotion recognition.

2. A method for dynamic expression recognition according to claim 1, characterized in that: In step 2, during the training of the online learning module and the momentum learning module, assuming that the total number of video segments obtained from step 1 is n, each video segment is input into the online learning module as 1 query sample, and n-1 reference samples are input into the momentum learning module.

3. A method for dynamic expression recognition according to claim 1, characterized in that: In step 3, the prior knowledge module uses AU as prior knowledge related to expression, combines the labels of the expression video in the data set as the classification of facial actions in the video segment, forms various query statements for facial components, inputs the query statements into the CLIP text encoder, and obtains the prior knowledge feature vector.

4. A method for dynamic expression recognition according to claim 1, characterized in that: The facial feature fusion module in step 4 uses the Transformer Decoder architecture to fuse the dense feature map with the prior knowledge feature vector, and its formula is: Among them, Z is the weighted feature map of the face; Q is the prior knowledge feature vector; Y fa It means to transform Y from 4D tensor D out ×H out ×W out ×C out Expand to (T k ,C out ), where T k =D out ×H out ×W out .

5. An electronic device, characterized in that: The system comprises one or more processors and a memory, wherein one or more programs are stored in the memory, and the programs comprise instructions for executing a dynamic expression recognition method as claimed in claim 1.

6. A computer-readable storage medium, characterized in that: Contains one or more programs for execution by one or more processors of an electronic device, wherein the programs include instructions for executing a dynamic expression recognition method as claimed in claim 1.

Images