Facial expression prediction method under uncontrolled conditions based on dual-branch video

Abstract

The present invention discloses a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video. The technical field is the field of facial expression prediction of human faces, comprising: obtaining initial video data, processing the initial video data to obtain video clips; constructing a dual-branch network structure, processing the video clips through the dual-branch network structure to obtain multi-scale depth feature maps; performing hierarchical feature fusion on the multi-scale depth feature maps based on an attention mechanism to obtain global and local information; and classifying the facial expression fusion features of the initial video clip samples based on the global and local information to complete facial expression classification under uncontrolled conditions. The features of different levels of the present invention contain information at different levels of abstraction. By fusing these features, the model can better understand and represent the input data, thereby improving the model's robustness to scale changes and the model's semantic expression ability.

Description

Technical Field

[0001] The present invention belongs to the field of human facial expression prediction, and in particular relates to a method for predicting human facial expression under uncontrolled conditions based on dual-branch video. Background Art

[0002] Facial expressions are a crucial way for humans to express emotions and engage in social interactions, and they are also a crucial avenue for human-computer interaction. In 2006, Hinton et al. proposed deep belief networks (DBNs), which brought renewed attention to deep learning and deep-learning-based facial expression recognition. The application of deep learning in expression recognition, primarily based on the VGGNet, GoogleNet, and ResNet network models, all employ deep convolutional neural networks (DCNNs) as their core architecture, is experiencing rapid growth. However, traditional feature extraction and classification methods typically rely on standard facial expression datasets to verify recognition rates, failing to adequately consider complex and changing environments, thus diminishing the applicability of expression recognition systems.

[0003] A fundamental assumption underlying facial expression recognition under controlled conditions is that the user imitates standard facial expressions, produces corresponding expressions, and then manually selects qualified samples. Therefore, the noise content of data samples collected under controlled conditions is relatively low. However, real-world scenarios are plagued by numerous interfering factors, including common ones such as identity, posture, lighting, gender, race, and age, as well as potential interfering factors such as hairstyle, accessories, and occlusion. Therefore, facial expression recognition under uncontrolled conditions remains extremely challenging. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video, so as to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video, comprising:

[0006] Acquiring initial video data, and processing the initial video data to obtain video clips;

[0007] Constructing a dual-branch network structure, processing the video clip through the dual-branch network structure to obtain a multi-scale depth feature map;

[0008] Based on the attention mechanism, the multi-scale deep feature map is hierarchically fused to obtain global and local information;

[0009] The facial expression fusion features of the initial video clip samples are classified based on the global and local information to complete the facial expression classification under uncontrolled conditions.

[0010] Preferably, the process of obtaining the video clip includes:

[0011] Performing length unification processing on the initial video data to obtain length-unified data;

[0012] Interpolating and cropping the length-uniform data to obtain a face region for each frame;

[0013] The length uniform data is extracted based on the face area of ​​each frame to obtain the video clip.

[0014] Preferably, the process of obtaining a multi-scale depth feature map includes:

[0015] Constructing a deep learning model of the self-attention mechanism, inputting the video clip into the deep learning model of the self-attention mechanism for processing, and obtaining output features of each stage;

[0016] Constructing a convolutional neural network model, inputting the video clip into the convolutional neural network model for processing, and obtaining output features between each layer;

[0017] The output features of each stage and the output features between each layer are merged to obtain the multi-scale depth feature map.

[0018] Preferably, each stage of the branch in the deep learning model of the self-attention mechanism is connected in the form of a block using SwinTransformeblock and processed and output;

[0019] The convolutional neural network model uses the basic BasicBlock of Resnet18 for skip-layer connection and output.

[0020] Preferably, the process of obtaining the global and local information includes: performing key information learning on the multi-scale depth feature map, and then fusing the feature maps of each level on which the key information learning is performed to obtain the global and local information.

[0021] Preferably, the classification network is constructed, and the global and local information are input into the classification network for processing to obtain a classification result;

[0022] Perform iterative training on the classification results to obtain a loss function,

[0023] The classification result is iteratively reprocessed based on the loss function to obtain the facial expression classification under the uncontrolled condition.

[0024] Preferably, the loss function is expressed as:

[0025]

[0026] Among them, y n is label (1 or 0), p n is the predicted probability of the object.

[0027] The technical effects of the present invention are:

[0028] The model uses a new initial video clip sample as input and employs a novel dual-branch feature extraction method to extract multi-scale deep features. This method further improves the extraction of information from the input image and increases network accuracy. A novel attention-based key information learning and hierarchical feature fusion method effectively fuse features from different layers, thereby improving model performance. Features at different layers contain information at different levels of abstraction. By fusing these features, the model can better understand and represent the input data, improving its robustness to scale variations and its semantic expression capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0030] Figure 1 Schematic diagram of the structure of a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video clips in an embodiment of the present invention;

[0031] Figure 2 It is a diagram illustrating the processing of original video and how to predict the result through the model in an embodiment of the present invention;

[0032] Figure 3 Schematic diagram of the activation feature map at each stage and the size of the feature maps at different scales at each stage in an embodiment of the present invention;

[0033] Figure 4 Schematic diagram of the attention mechanism used in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment provides a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video, including:

[0038] Obtaining an initial video clip sample, and processing the initial video clip sample to obtain a video clip as a model input;

[0039] Obtain multi-scale depth features through the Transformer branch and the CNN branch according to the initial video clip sample;

[0040] Based on the processing of the initial video clip samples, the extraction of the dual-branch structure features and the key information learning based on the attention mechanism, the features of each layer are fused, the facial expression fusion features of the initial video clip samples are classified, and the facial expression classification under uncontrolled conditions is completed.

[0041] Optionally, the method of processing the initial video clip sample to obtain a video clip as a model input specifically includes:

[0042] According to the initial video clip sample processing, we unify the input video length to 105 frames, detect the face area of ​​each frame through interpolation and cropping operations, the size of each frame is 224×224, randomly select one from the first thirty frames as the starting frame, and extract the subsequent 75 consecutive frames to form a video.

[0043] Optionally, the method of obtaining multi-scale depth features through the Transformer branch and the CNN branch according to the initial video clip sample specifically includes:

[0044] The processed initial video clip samples are input into a dual-branch model for multi-layer deep feature extraction. The output features of each stage of the Transformer branch and the output features between each layer of the CNN are connected in the form of blocks using SwinTransformeblock and processed and output. The CNN branch uses the basic BasicBlock of Resnet18 for skip-layer connection and output to obtain multi-scale deep features. The size of the output feature map of each stage of the Transformer branch is consistent with the size of the output feature map of the skip-layer connection of the CNN branch (the feature map size of the first stage is 56*56, the feature map size of the second stage is 28*28, the feature map size of the third stage is 14*14, and the feature map size of the fourth stage is 7*7).

[0045] Optionally, obtaining multi-scale deep features through a dual-branch structure based on the initial video clip sample to perform key information learning based on an attention mechanism, and obtaining global and local information specifically includes:

[0046] Through two experimental data sets, it was detected that the learning effect of the self-attention mechanism is better. According to the key information learning, more complete information learning is obtained by combining the self-attention mechanism.

[0047] Optionally, based on the learning of the multi-scale deep feature key information, global and local features of different scales are obtained, and a method for performing hierarchical feature fusion specifically includes:

[0048] The features of the same size at each stage of the dual-branch structure are used for key information learning. Then, the key information learning feature maps of each layer are fused to increase the information richness and expressiveness of the features, enhance the diversity of the network, and reduce the number of parameters and computational complexity.

[0049] Optionally, based on the initial video clip sample processing, the multi-scale deep feature map extraction, and the key information learning and hierarchical feature fusion based on the attention mechanism, we pay attention to the weight of each channel of the input feature layer, with the focus on obtaining the weight of each channel of the input feature layer. Using SelfAttention, we can make the network focus on the channels it needs to pay most attention to. Specifically, it includes:

[0050] Perform global average pooling on the input feature layer.

[0051] Then two full connections are performed. The first full connection has a smaller number of neurons, and the second full connection has the same number of neurons as the input feature layer.

[0052] After completing two full connections, we take Sigmoid again to fix the value between 0 and 1. At this time, we obtain the weight of each channel of the input feature layer (between 0 and 1).

[0053] After obtaining this weight, we multiply it by the original input feature layer.

[0054] Optionally, the facial expression classification based on the obtained initial video clip sample specifically includes:

[0055] Based on the model, iterative training is performed on the classification results of the input video clip to obtain a loss function, and then an update iterative process is performed according to the loss function to obtain the facial expression classification results of the initial video clip sample.

[0056] Optionally, the loss function calculation includes:

[0057]

[0058] Among them, y n is label (1 or 0), p n is the predicted probability of the object.

[0059] Example 2

[0060] This embodiment provides a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video, including:

[0061] Obtaining an initial video clip sample, and processing the initial video clip sample to obtain a video clip as a model input;

[0062] Obtain multi-scale depth features through the Transformer branch and the CNN branch according to the initial video clip sample;

[0063] Based on the processing of the initial video clip samples, the extraction of the dual-branch structure features and the key information learning based on the attention mechanism, the features of each layer are fused, the facial expression fusion features of the initial video clip samples are classified, and the facial expression classification under uncontrolled conditions is completed.

[0064] Model architecture: After the processed image is input into the network, the features output by each stage of the Transformer branch and the features output between each layer of the CNN are connected in the form of blocks using SwinTransformeblock and processed and output. The CNN branch uses the basic BasicBlock of Resnet-18 for skip-layer connection and output to obtain multi-scale deep features. The output feature map size of each stage of the Transformer branch is consistent with the output feature map size of the skip-layer connection of the CNN branch (the feature map size of the first stage is 56*56, the feature map size of the second stage is 28*28, the feature map size of the third stage is 14*14, and the feature map size of the fourth stage is 7*7). The key information of the multi-scale deep features is learned (self-attention), and global and local features of different scales are obtained, and hierarchical feature fusion is performed. The fused features are input into the classification network, and the final output is compared with the groundtruth to calculate the loss function to obtain the network loss function.

[0065] Learning of key information of multi-scale deep features, the output of the four stages of the dual-branch network is used as input, and the Squeeze operation (i.e. Figure 4 In the figure, Fsq(·) compresses the spatial dimensions, converting each two-dimensional feature map into a real number. This is equivalent to a pooling operation with a global receptive field, while the number of feature channels remains unchanged. Next comes the Excitation operation (Fex(·) in the figure), which generates weights for each feature channel using the parameter w. w is learned to explicitly model the correlation between feature channels. This is achieved using a fully connected layer with a two-layer bottleneck structure (first reducing the dimensionality and then increasing the dimensionality) and a sigmoid function. After obtaining the weights for each feature channel, these weights are applied to each original feature channel to better learn information at each stage.

[0066] Hierarchical feature fusion learns key information from features of the same size at each stage of the dual-branch structure. Then, the key information learning feature maps of each layer are fused to increase the information richness and expressiveness of the features, enhance the diversity of the network, and reduce the number of parameters and computational complexity.

[0067] The loss function used is the cross entropy loss function. It is defined as follows:

[0068]

[0069] y n is label (1 or 0), p n is the predicted probability of the next active object.

[0070] Our method was evaluated on the DFEW and AFEW datasets. DFEW is a large-scale database of unconstrained dynamic facial expressions, consisting of 16,372 video clips extracted from over 1,500 movies. It contains 12,059 single-label video clips and seven emotion labels: anger, disgust, fear, happiness, sadness, surprise, and neutral. Since 2013, AFEW has served as the evaluation platform for the annual EmotiW. AFEW includes seven emotion labels: anger, disgust, fear, happiness, sadness, surprise, and neutral. AFEW contains videos with spontaneous expressions collected from various movies and TV series, and also contains a large amount of non-expression noise such as head pose variations, speech-related mouth movements, occlusions, and lighting. AFEW is divided into three parts: Train (738 videos), Val (352 videos), and Test (653 videos).

[0071] In this embodiment, the input image is first resized to 3*224*224, the pyramid feature hierarchy and locality are introduced into the Transformer branch design, and the CNN branch adopts skip-layer connection to obtain more feature information to prevent the features from being changed or covered; in particular, for the baseline structure, we first use pre-trained ResNet18 and Transformer to extract multi-scale deep features of each frame, perform key information learning and hierarchical feature fusion, and then input the classifier using 3 encoder layers, followed by another 3 decoder layers to gradually translate the extracted features into the final expression recognition results. Each of the encoder layers used includes a series of self-attention operations and multi-layer perception operations, and each decoder layer includes a series of cross-attention operations and multi-layer perception operations. Both self-and cross-attention operations use 4 heads for segmentation processing.

[0072] The classification results of the DFEW dataset and the information of the baseline method are shown in Table 1.

[0073] Table 1

[0074]

[0075] Compared to dual-branch video clip feature extraction, there are also frame-based methods for video-based FER. Frame-based methods can be divided into two groups: frame aggregation methods that strategically fuse deep features learned from static-based FER networks, and peak extraction methods that focus on identifying frames with peak high-intensity expressions. Dynamic sequence methods for video-based FER. To explore the spatial-temporal representation of expressions, dynamic sequence-based methods take a video sequence as a single input and exploit texture information and temporal correlations in the sequence for more robust expression recognition. In recent years, long-term memory (LSTM) and C3D are two widely used spatiotemporal methods.

[0076] ⑴3D ResNet-18, EC-STF, C3D, EC-STFL, P3D, EC-STFL, R3D18, EC-STFL, VGG11+LSTM, EC-STFL:

[0077] We propose a new large-scale "in-the-wild" dynamic facial expression database, DFEW (Dynamic Facial Expressions in the Wild), consisting of more than 16,000 video clips from thousands of movies. These video clips contain various challenging perturbations in real scenes, such as extreme lighting, occlusions, and erratic pose changes. A new approach called Expression Clustering Spatiotemporal Feature Learning (EC-STFL) framework is used to handle dynamic FER in the wild. Extensive benchmark experiments are conducted on DFEW using a large number of spatiotemporal deep feature learning methods as well as our proposed EC-STFL. Experimental results show that DFEW is a well-designed and challenging database, and the proposed EC-STFL can promisingly improve the performance of existing spatiotemporal deep neural networks in dealing with the problem of dynamic FER in the wild.

[0078] ⑵Former-DFER:

[0079] Former-DFER (dynamic facial expression recognition transformer) uses the Transformer's self-attention mechanism to learn the correlation between regional facial features and the correlation between facial features over time, thus addressing these challenges. The CS-former consists of five convolutional blocks and N spatial encoders, designed to guide the network to learn facial features that are robust to occlusion and pose from a spatial perspective. The T-former consists of M temporal encoders, designed to allow the network to learn contextual facial features from a temporal perspective.

[0080] (3)EST:

[0081] EST (Expression snippet transformer for robust video-based facial expression recognition) designs an attention-augmented segment feature extractor (AA-SFE) for intra-segment modeling, enhancing the encoding of subtle facial movements in each segment by gradually focusing on more salient information. Furthermore, for inter-segment modeling, a stochastic segment order prediction (SSOP) head and corresponding loss are introduced to improve modeling by training a transformer to recognize subtle motion changes in subsequent segments, even in the order of randomly played segments. Extensive experiments are conducted on four challenging datasets (i.e., BU-3DFE, MMI, AFOL, and DFEW).

[0082] Table 2 shows the comparison of the performance of the baseline model and our model on the AFEW dataset.

[0083] Table 2

[0084]

[0085] To test the effectiveness of each component of the model, we also report the performance of the model under different configurations. The results and analysis are as follows:

[0086] ⑴Compared with baseline:

[0087] The performance of our method and the baseline methods are shown in Tables 1 and 2. Our method significantly outperforms the baseline methods. Compared to the best-performing baseline method, our method improves by 0.13% (WAR) and 0.22% (UAR) on the DFEW dataset, and by 1.77% (WAR) and 5.8% (UAR) on the AFEW dataset. This demonstrates the effectiveness of our network structure.

[0088] ⑵The effectiveness of the dual-branch network in extracting multi-scale features and processing the initial video clips:

[0089] When all other parts of the network remain unchanged, removing the Transformer branch lacks the ability to capture rich, emotional features. Trivial or irrelevant information and extremely subtle visual changes can hinder the model's performance, hindering its ability to extract useful features and make accurate predictions. To address these issues, we propose using fragments to decompose the modeling of facial changes across the entire video into a series of sub-videos as model input. This demonstrates the effectiveness of our dual-branch network in extracting multi-scale deep features, learning key information, and merging hierarchical features. This improves the directness of the learning process in the middle of the network, allowing the network to be more fully trained.

[0090] The present invention discloses a method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video clips. The method uses the processing of video clips as the input of a model to simulate the state of a human face under uncontrolled conditions, extracts multi-scale deep features through the dual-branch clip structure to obtain richer information, adopts a new key information learning method to improve the accuracy of model training, and adopts a hierarchical feature fusion method to better fuse information of different abstract levels contained in features of different levels, thereby improving the robustness of the model to scale changes. It improves the directness of the learning process of the middle part of the network, and the network is more fully trained, thereby improving the training speed and the prediction accuracy of the classified objects.

[0091] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for predicting facial expressions under uncontrolled conditions based on dual-branch video, characterized in that: The following steps are involved: Acquiring initial video data, and processing the initial video data to obtain video clips; Constructing a dual-branch network structure, processing the video clip through the dual-branch network structure to obtain a multi-scale depth feature map; Based on the attention mechanism, the multi-scale deep feature map is hierarchically fused to obtain global and local information; Classifying the facial expression fusion features of the initial video clip sample based on the global and local information to complete facial expression classification under uncontrolled conditions; The process of obtaining a multi-scale depth feature map includes: Constructing a deep learning model of the self-attention mechanism, inputting the video clip into the deep learning model of the self-attention mechanism for processing, and obtaining output features of each stage; Constructing a convolutional neural network model, inputting the video clip into the convolutional neural network model for processing, and obtaining output features between each layer; Merging the output features of each stage and the output features between each layer to obtain the multi-scale depth feature map; Each stage of the branch in the deep learning model of the self-attention mechanism is connected in the form of a Swin Transformer block and processed and output; The convolutional neural network model uses the basic BasicBlock of Resnet18 for skip-layer connection and output.

2. The method for predicting facial expressions of human faces under uncontrolled conditions based on dual-branch video according to claim 1, characterized in that The process of obtaining the video clip includes: Performing length unification processing on the initial video data to obtain length-unified data; Interpolating and cropping the length-uniform data to obtain a face region for each frame; The length uniform data is extracted based on the face area of ​​each frame to obtain the video clip.

3. The method for predicting facial expression of a human face under uncontrolled conditions based on dual-branch video according to claim 1, wherein: The process of obtaining global and local information includes: performing key information learning on the multi-scale depth feature map, and then fusing the feature maps of each level for key information learning to obtain the global and local information.

4. The method for predicting facial expression of a human face under uncontrolled conditions based on dual-branch video according to claim 1, wherein: Constructing a classification network, inputting the global and local information into the classification network for processing to obtain a classification result; Perform iterative training on the classification results to obtain a loss function, The classification result is iteratively reprocessed based on the loss function to obtain the facial expression classification under the uncontrolled condition.

5. The method for predicting facial expression of a human face under uncontrolled conditions based on dual-branch video according to claim 4, characterized in that: The expression of the loss function is: ； in, It is a label. is the predicted probability of the object.

Images