Contrastive learning combined with masked image modeling for self-supervised facial expression recognition

Abstract

The application discloses a self-supervised facial expression recognition method combining contrast learning and mask image modeling, and the method comprises the following steps: constructing a self-supervised facial expression recognition model combining contrast learning and mask image modeling; performing unsupervised pre-training on the model; performing linear probe evaluation and fine-tuning evaluation on the pre-trained model on a labeled verification set; using the fine-tuned model to perform facial expression recognition on an input facial image; the method learns facial expression representation through a convolution-free twin network, pre-trains the twin network by using contrast loss and mask image modeling loss, so as to simultaneously understand high-level visual semantics and image internal structure, and maximize the consistency between the outputs of the student network and the teacher network. The application can learn rich visual information, is robust to various interference unrelated to facial expression, and simultaneously achieves excellent results in linear probe and fine-tuning evaluation on a facial expression dataset.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification, and particularly relates to a self-supervised facial expression recognition method combining contrast learning and mask image modeling. BACKGROUND

[0002] Facial expressions provide rich emotional information and are one of the most powerful, most common and most natural ways for humans to express emotions and intentions. In interpersonal communication, facial expressions are an important information carrier that cannot be ignored. In recent years, with the continuous development of artificial intelligence, facial expression recognition (FER) is widely used in intelligent devices, medical services, personal emotion management, fatigue driving monitoring and other human-computer interaction systems, and is one of the research hotspots in the fields of affective computing and computer vision (CV). The famous American psychologist Paul Ekman proposed six basic expressions of humans, which are now considered as a universal standard: anger, disgust, fear, happiness, sadness and surprise, and later added a new basic expression of contempt.

[0003] Before the rise of deep learning, traditional FER methods were mainly based on hand-crafted features and shallow learning (e.g., histogram of oriented gradients (HOGs), local binary patterns (LBP) and sparse representation). In recent years, with the development of data-driven technology, deep learning-based methods have gradually replaced traditional methods and achieved the most advanced FER performance. Deep learning has strong representation ability, but there are still several problems when applied to facial expression recognition tasks:

[0004] 1) Dependence on large-scale labeled data. The success of current deep learning-based facial expression recognition methods relies on supervised training methods, which require a huge amount of data. The more labeled data and the better the quality of labeling, the more significant the model effect, and the lack of sufficient training data will lead to overfitting problems. However, current manual labeling of facial expression data has problems of inconsistent labeling quality and high cost. On the one hand, differences in understanding and cognition of different labelers will lead to subjective and low-accuracy labeling results, which will greatly interfere with the network training process. On the other hand, manual labeling of facial expression data requires multiple tedious steps, which is low in efficiency and high in cost, and it is difficult to obtain high-quality labeled facial expression data.

[0005] 2) Disturbance of factors irrelevant to expression recognition in natural scenes. Facial expression recognition in natural scenes is affected by many external factors, such as different lighting conditions, complex backgrounds, different head poses, partial occlusion, facial distortion, blur, etc., so it is a very difficult task.

[0006] Supervised Learning (SL) methods are highly dependent on a large amount of labeled data, and the lack of labeled data prompts researchers to find alternative methods, i.e., how to make use of a large amount of unlabeled data, so Self-Supervised Learning (SSL) has received extensive attention. Self-Supervised Learning belongs to unsupervised learning, which can directly use the data itself as supervision information (pseudo-label) without human-labeled information. Self-Supervised Learning first learns the feature representation of the image in the pre-training stage through the pre-task, and this stage does not need labeled data. Then, the learned parameters are migrated to the downstream task as a pre-trained model, and fine-tuning is performed using part of the labeled data to achieve good results. Self-Supervised Learning has the following advantages compared with Supervised Learning: 1) does not require a large amount of labeled data; 2) pays more attention to the internal information of the data, and the information provided by the data is more abundant than the sparse label; 3) can learn more general knowledge, and the representation obtained by Self-Supervised Learning can be better migrated to the downstream task.

[0007] In order to reduce the dependence on classification labels in the training process, a Self-Supervised Learning (SSL) method can be used to pre-train a deep neural network. The SSL framework can be divided into contrastive learning (CL) and generative learning. Contrastive learning pulls the representations of different augmented views from the same image closer, while avoiding feature collapse. It performs well in linear probing. However, contrastive learning mainly focuses on the center region and high-level semantics, and is poor in capturing local features, which is similar to supervised pre-training. Recently, Masked Image Modeling (MIM) has gradually received extensive attention, and MIM trains the model to restore the original content of the occluded image, which belongs to the generative learning framework. MIM can better understand the complex local structure within the image, thus performing well in fine-tuning, but performs poorly in linear probing. CL and MIM methods have advantages and disadvantages. CL simply learns the global representation of the whole image, so it cannot describe the internal structure of the image. MIM works separately in each image, paying more attention to low-level information, but does not consider the interaction between images. MIM is difficult to learn the representation of images with similar high-level semantics, resulting in poor linear probing performance. SUMMARY

[0008] The present application aims at the problems existing in the prior art, and provides a self-supervised facial expression recognition method combining contrastive learning and masked image modeling.

[0009] The technical solution for achieving the object of the present application is as follows: a self-supervised facial expression recognition method combining contrastive learning and mask image modeling, comprising the following steps:

[0010] Step 1: constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling;

[0011] Step 2: performing unsupervised pre-training on the self-supervised facial expression recognition model;

[0012] Step 3: performing linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on a labeled validation set;

[0013] Step 4: using the self-supervised facial expression recognition model after fine-tuning evaluation to perform facial expression recognition on an input facial image.

[0014] A self-supervised facial expression recognition device combining contrastive learning and mask image modeling, comprising the following modules:

[0015] A model construction module for constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling;

[0016] A model training module for performing unsupervised pre-training on the self-supervised facial expression recognition model;

[0017] A model evaluation module for performing linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on a labeled validation set;

[0018] A model recognition module for using the self-supervised facial expression recognition model after fine-tuning evaluation to perform facial expression recognition on an input facial image;

[0019] The device executes the steps in the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in the process of facial expression recognition.

[0020] A mobile terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the self-supervised facial expression recognition method combining contrastive learning and mask image modeling.

[0021] A computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps in the self-supervised facial expression recognition method combining contrastive learning and mask image modeling.

[0022] Compared with the prior art, the present application has the following advantages: (1) combining the respective advantages of contrastive learning and mask image modeling, the inter-image relationship and intra-image structure can be considered simultaneously: on the one hand, the similarity of classification tokens in different enhanced views of the same image is promoted to capture high-level visual semantics, thereby achieving semantic alignment; on the other hand, the HOG features of mask images from different enhanced views of the same image are predicted, thereby modeling the local structure within each image; (2) the hand-crafted feature HOG is introduced in the mask image modeling loss, which well fits the prediction target of the random mask image block, can make the model robust to interference such as lighting and contrast changes in natural scenes to a certain extent, and can prevent the model from overfitting to pixel detail information.

[0023] The application will be described in further detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Model diagram of the self-supervised facial expression recognition method combining contrastive learning and mask image modeling.

[0025] Figure 2 Structure diagram of the backbone network f (ViT-S / 16) of the self-supervised facial expression recognition network combining contrastive learning and mask image modeling.

[0026] Figure 3 Visual attention map of a pre-training model of the self-supervised facial expression recognition network combining contrastive learning and mask image modeling.

[0027] Figure 4 Visual attention map of another pre-training model of the self-supervised facial expression recognition network combining contrastive learning and mask image modeling. DETAILED DESCRIPTION

[0028] The self-supervised facial expression recognition method combining contrastive learning and mask image modeling comprises the following steps:

[0029] Step 1, constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling;

[0030] Step 2, performing unsupervised pre-training on the self-supervised facial expression recognition model;

[0031] Step 3, performing linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on a labeled validation set;

[0032] Step 4, using the self-supervised facial expression recognition model after fine-tuning evaluation to perform facial expression recognition on an input facial image.

[0033] As a specific example, the step 1 described construction of the contrast learning combined mask image modeling self-supervised facial expression recognition model, specifically includes:

[0034] Step 1-1, calculate the direction gradient histogram (Histogram of Oriented Gradient, HOG) feature z of the input image x hog ;

[0035] Step 1-2, process the input image, given the input image First divided into N = HW / P 2 Image blocks Where C is the number of channels, 3 channels (RGB); (H, W) is the resolution of the input image, that is, (224, 224); (P, P) is the resolution of each image block, size (16, 16), N is 196;

[0036] Step 1-3, randomly transform the input image x to generate two related enhanced view represented as u and v, as a positive sample pair, respectively into the student network g s And the teacher network g t ;

[0037] Step 1-4, by linear projection, two enhanced views u and v of the two-dimensional image x are flattened into image blocks, represented as And Both are regarded as a sequence of image block visual tokens with length N;

[0038] Step 1-5, for the image visual token sequence Mask image modeling first generates a random mask m ∈ {0, 1} N According to the mask prediction ratio r, where the number of visual tokens is N; Then replace the image block visual token u i With mask visual token [M], generate a masked image

[0039]

[0040] Where, m is 1, indicating that u i Is masked, that is, replaced by mask visual token [M]; m is 0, indicating that it is not masked; Mask image modeling aims to recover the masked visual token from the masked image ;

[0041] Step 1-6, using the encoder of ViT-S / 16 (Vision Transformer-Small) as the backbone network f of the self-supervised facial expression recognition model, / 16 represents that the image block size is 16, and the number of encoder layers is 12;

[0042] Step 1-7, the enhanced views u and v are converted into a set of embedding sequences through a linear layer, and an additional learnable token (visual word) is added in the sequence; the role of the category visual word [CLS] is to aggregate the information of the entire sequence without adding any label or supervision; after the mask of view u, the image block visual word and the set of category visual words [CLS] are sent to the Transformer encoder f of the student network s ; view v is not masked, and the image block visual word v and the category visual word [CLS] are directly sent to the encoder f of the teacher network t ;

[0043] Step 1-8, the projection head h of the self-supervised facial expression recognition model is composed of a 3-layer multi-layer perceptron (MLP), an l2 normalization layer, and a weight-normalized K-dimensional fully connected layer;

[0044] Step 1-9, the network g of the self-supervised facial expression recognition model is composed of the backbone network f and the projection head h, the student network and the teacher network share the same architecture g and have different parameter sets θ s and θ t ; the Transformer is pre-trained and fine-tuned using images with a size of 224x224, the total number of image block visual words N is 196, the output dimension of the shared projection head is set to 108, and the final output vector dimension K is 21168.

[0045] As a specific example, the unsupervised pre-training on the self-supervised facial expression recognition model described in step 2 specifically includes:

[0046] Step 2-1, initialize the network parameters of the self-supervised facial expression recognition model and the number of iterations;

[0047] Step 2-2, randomly mask the enhanced view u of the original image x to obtain the masked view

[0048] Step 2-3, the student network outputs the predicted HOG feature for the masked view The teacher network outputs the feature for the unmasked view v where P(·) represents converting the input into a K-dimensional probability distribution, θ s and θ​t denote the parameter sets of the student and teacher networks, respectively;

[0049] The output of network g is normalized by using a softmax function

[0050]

[0051] where τ t > 0 is a temperature parameter, and for temperature τ s The formula of L is also similar:

[0052]

[0053] Step 2-4, the training objective of Masked Image Modeling (MIM) is defined as:

[0054]

[0055] where L is the masked image modeling loss, N is the number of visual words, i represents the visual word index of the image block, m i is 1 indicating that the image block is masked, m i is 0 indicating that the image block is not masked, i.e., the reconstruction loss is only calculated on the masked image blocks; z hog is the target feature of the self-supervised facial expression recognition model, which is a one-dimensional vector. First, the HOG feature map is extracted on the whole image of the masked view, and then it is reshaped into image blocks. The histogram of the masked image block is flattened and connected into z hog .

[0056] Step 2-5, in order to maintain semantic alignment, the self-supervised facial expression recognition model also performs cross-entropy loss on the class visual word [CLS] of different views as a contrastive loss L By minimizing the cross-entropy loss to match information with similar semantics:

[0057]

[0058] where denotes the class semantic feature output by the teacher network for the non-masked view v, denotes the class semantic feature output by the student network for the masked view .

[0059] Step 2-6, a stop-gradient operator sg is applied on the teacher network, and only the student network is used to propagate the gradient; the parameters θ t of the student network are updateds Exponential Moving Average (EMA) is performed, and the update rule is:

[0060] θ t ←λθ t +(1-λ)θ s

[0061] wherein, lambda is a hyperparameter, following cosine learning rate adjustment from 0.996 to 1 during training; the self-supervised facial expression recognition model optimizes the target of the mask image modeling through momentum update, so it does not need an additional training phase as a pretreatment setting;

[0062] Step 2-7, the self-supervised facial expression recognition model learns the parameter theta by minimizing and through the stochastic gradient descent method. s .

[0063] As a specific example, the pre-trained self-supervised facial expression recognition model is subjected to linear probing evaluation and fine-tuning evaluation on the labeled validation set according to step 3, which specifically includes:

[0064] Step 3-1, after unsupervised pre-training, a supervised linear classifier is learned on the frozen features of the teacher network without the projection head to perform linear probing evaluation;

[0065] Step 3-2, for fine-tuning evaluation, a supervised linear classifier is learned on the teacher network without the projection head based on the weights of the unsupervised pre-trained teacher network encoder, and the labeled validation set data is used to update the parameters of the teacher network and the classification layer, i.e., the parameters of the teacher network are not frozen;

[0066] Step 3-3, linear probing and fine-tuning are performed on the labeled validation set for 100 epochs to perform linear probing and fine-tuning evaluation.

[0067] The present application also provides a self-supervised facial expression recognition device based on contrastive learning combined with mask image modeling, which comprises the following modules:

[0068] A model construction module is configured to construct a self-supervised facial expression recognition model based on contrastive learning combined with mask image modeling.

[0069] A model training module is configured to perform unsupervised pre-training on the self-supervised facial expression recognition model.

[0070] a model evaluation module, configured to perform linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on a labeled validation set;

[0071] a model identification module, configured to perform expression recognition on an input facial image by using the self-supervised facial expression recognition model after fine-tuning evaluation.

[0072] The device performs the steps in the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in the process of facial expression recognition.

[0073] The application also provides a mobile terminal, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the self-supervised facial expression recognition method combining contrastive learning and mask image modeling when executing the program.

[0074] The application also provides a computer readable storage medium, which stores a computer program, and the program implements the steps in the self-supervised facial expression recognition method combining contrastive learning and mask image modeling when executed by a processor.

[0075] In order to make the objects, technical solutions and advantages of the application clearer, the application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0076] Embodiment 1

[0077] In this embodiment, a self-supervised facial expression recognition method combining contrastive learning and mask image modeling is provided, which comprises the following steps:

[0078] Step 1, constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling;

[0079] Step 2, performing unsupervised pre-training on the self-supervised facial expression recognition model combining contrastive learning and mask image modeling;

[0080] Step 3, performing linear probe evaluation and fine-tuning evaluation on the pre-trained model on a labeled validation set.

[0081] Further, in one of the embodiments, the self-supervised facial expression recognition method combining contrastive learning and mask image modeling comprises the following steps: Figures 1-2 , Figure 1 Figure 1 is a model diagram of the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in the embodiment, Figure 2 Figure 2 is a structure diagram of a network backbone f(ViT-S / 16) of the self-supervised facial expression recognition network combining contrastive learning and mask image modeling built in the embodiment, which is composed of a multi-head self-attention network, a residual connection and layer normalization, and a feedforward (fully connected) neural network.

[0082] The step 1 constructs a self-supervised facial expression recognition model combining contrastive learning and mask image modeling, and specifically comprises:

[0083] Step 1-1, calculating the histogram of oriented gradient (HOG) feature z of the input image x ho ;

[0084] Step 1-2, randomly transforming the input image x to generate two relevant augmented view representations u and v as a positive sample pair;

[0085] Step 1-3, processing the input image, given the input image First, it is divided into N = HW / P 2 image patches: i∈N, where C is the number of channels, which is 3 channels (RGB); (H, W) is the resolution of the input image, i.e. (224, 224); (P, P) is the resolution of each image patch, which is (16, 16), so N is 196. Through linear projection, the two-dimensional image x is flattened into image patches: which can be regarded as an image visual token sequence;

[0086] Step 1-4, for the image visual token sequence Mask image modeling first generates a random mask m∈{0,1} N according to the mask prediction ratio r, where N is the number of visual tokens. Then the mask visual token [M] is replaced with the image patch visual token x i to generate a masked image

[0087]

[0088] where m is 1, indicating that x i is masked, and m is 0, indicating that it is not masked. Mask image modeling aims to recover the masked visual tokens from the masked image ;

[0089] Steps 1-5 use ViT-S / 16 as the backbone network f of the model, where / 16 represents an image patch size of 16 and an encoder layer of 12. These image patches are then transformed into a set of embedding sequences through a linear layer, with an additional learnable token (visual word) added to the sequence. This category token [CLS] aggregates information from the entire sequence without attaching any labels or supervision. The set of image patch tokens and [CLS] tokens is then fed into a standard Transformer network. The Transformer is pre-trained and fine-tuned using images of size 224×224, with a total of 196 image patch tokens. The output dimension of the shared projector is set to 108, resulting in a final output vector dimension of 21168.

[0090] Steps 1-6: The model's projection head h consists of a 3-layer multilayer perceptron (MLP), an l2 normalized layer, and a weight-normalized K-dimensional fully connected layer.

[0091] Steps 1-7: The model's network g consists of a backbone network f and a projection head h: g = h°f. The student and teacher networks share the same architecture g but have different parameter sets θ. s and θ t .

[0092] Preferably, in one embodiment, the image enhancement method described in steps 1-2 is applied sequentially as follows: random cropping and then adjusting to a size of 224×224, random horizontal flipping, random color jittering, random Gaussian blur, and overexposure.

[0093] Furthermore, in one embodiment, step 2, which involves unsupervised pre-training on a self-supervised facial expression recognition model based on contrastive learning combined with mask image modeling, specifically includes:

[0094] Step 2-1: Initialize the model network parameters and the number of iterations;

[0095] Step 2-2: Randomly mask the enhanced view u of the original image x to obtain the masked view.

[0096] Steps 2-3, student network as masked view Output its predicted HOG features The teacher network is a non-masked view of the output features. Where P(·) transforms the input into a K-dimensional probability distribution, which is obtained by normalizing the output of network g using the softmax function:

[0097]

[0098] where τ t >0 is a temperature parameter, for temperature τ s of the formula is also similar;

[0099] Step 2-4, the training objective of the mask image modeling can be defined as:

[0100]

[0101] where N is the number of visual words, m i = 1 indicates that the image block is masked, i.e. only the reconstruction loss is calculated on the masked image block; z hog as the target feature of the model, is a one-dimensional vector, first extract the HOG feature map on the whole image of the mask view, then reshape it into an image block, and the histogram of the masked image block is flattened and connected into z ho ;

[0102] Step 2-5, in order to maintain semantic alignment, the model also performs cross-entropy loss on the [CLS] tokens of different views as a contrastive loss. By minimizing the cross-entropy loss to match information with similar semantics:

[0103]

[0104] Step 2-6, an sg (stop-gradient) operator is applied on the teacher network to stop the gradient, only through the student network to propagate the gradient. The parameters θ t of the teacher network are exponentially moving average (EMA) with the parameters θ s of the student network, the update rule is:

[0105] θ t ← λθ t + (1-λ)θ s

[0106] where λ follows a cosine learning rate schedule from 0.996 to 1 during training. The model optimizes the target of the mask image modeling through momentum update, so there is no need for an additional training phase as a pre-processing setup.

[0107] Step 2-7, the model learns the parameters θ s by minimizing and through stochastic gradient descent.

[0108] Here, the parameters are preferably set as: pre-training with AdamW optimizer on FERplus training set, batch size of 64. 300 epochs of unsupervised pre-training with ViT-S / 16 as backbone network. In the first 10 epochs, the learning rate linearly rises to its benchmark value, which is determined by the linear scaling rule: lr = 0.0005 x batchsize / 256, and cosine learning rate decay is used after warm-up. The decay of weights also follows the cosine adjustment from 0.04 to 0.4. Temperature τ s is set to 0.1, and τ t Linear warm-up from 0.04 to 0.07 is used. The random mask ratio r is 40% of all image blocks. All image datasets are adjusted to 224x224 size.

[0109] Further, in one of the embodiments, the step 3 of performing linear probing evaluation and fine-tuning evaluation on the pre-trained model on the labeled validation set specifically includes:

[0110] Step 3-1, after unsupervised pre-training, a supervised linear classifier is learned on the frozen features of the teacher network without the projection head, and linear probing evaluation is performed;

[0111] Step 3-2, for fine-tuning evaluation, the network is initialized based on the unsupervised pre-trained weights, and the labeled data is used to update the parameters of the entire network including the classification layer and the feature extraction part;

[0112] Step 3-3, linear probing and fine-tuning are performed on the labeled validation set for 100 epochs for evaluation.

[0113] Embodiment 2

[0114] Exemplarily, in this embodiment, experiments are performed using the same facial expression dataset, and the model of the present application is compared with the latest mainstream self-supervised learning methods (DINO, iBOT, MAE), and the evaluation criteria are linear probing and fine-tuning; and the model of the present application is compared with the most advanced supervised learning method on FERplus, RAF-DB, and CK+ datasets.

[0115] The results of the comparative experiments are shown in Tables 1-2.

[0116] Table 1: Comparison of results of self-supervised learning methods on FERplus dataset

[0117]

[0118] Table 2 Comparison of results with supervised learning methods

[0119]

[0120] As can be seen from Table 1, the self-supervised facial expression recognition method of the present application combining contrast learning and mask image modeling takes ViT-S / 16 as the backbone network, achieves a linear probe accuracy of 84.62%, which is 2.53% and 0.47% higher than DINO and iBOT respectively; the fine-tuning accuracy is 89.28%, which is 1.47% higher than MAE, and about 0.2% higher than DINO and iBOT. In general, the present method achieves the most advanced performance in linear probe and fine-tuning evaluation, and the advantage in linear probe accuracy is more obvious.

[0121] As can be seen from Table 2, the present method uses less labeled data, and is superior to other supervised training methods on FERplus, and achieves results comparable to supervised methods on RAF-DB and CK+. Compared with RAN and SCN, the total improvement of the present method on FERplus is 1.43% and 1.27% respectively. Compared with the baseline method DLP-CNN, the present method obtains a gain of 3.53% on RAF-DB. The results of the present method on RAF-DB and CK+ are only slightly lower than the most advanced supervised method, but the present method pre-trains to learn features without labels, which undoubtedly provides an advantage by reducing the dependence on output labels.

[0122] Figures 3-4 Visualization of attention map of model pre-trained on FERplus, different heads are represented by different colors. Different heads can process different semantic regions of the image, even if they are occluded (such as Figure 4 mouth); the model can highlight the distinctive regions, i.e. the regions related to facial expression information, and ignore irrelevant regions (such as Figure 3 background and interference in the face image). Therefore, the model can learn rich facial information related to facial expression, and focus on the key regions of expression.

[0123] In summary, the proposed method learns facial expression representation by a convolution-free twin network, where the student network takes the randomly masked face image as input, while the teacher network takes the unmasked image as input. To maximize the consistency between the outputs of the student and teacher networks, the proposed method pre-trains the twin network with contrastive loss and mask image modeling loss, which can simultaneously understand high-level visual semantics and image internal structure. Experimental results show that the proposed method can learn rich visual information and is robust to various disturbances irrelevant to facial expression in face expression recognition in natural scenes. Without using additional pre-trained weights or supervision, the proposed method achieves excellent performance on linear probing and fine-tuning evaluation on face expression datasets.

[0124] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and all such changes and improvements fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A self-supervised facial expression recognition method combining contrastive learning and masked image modeling, characterized in that, The method comprises the following steps: Step 1, constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling; Step 2, unsupervised pre-training on the self-supervised facial expression recognition model; Step 3, linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on the labeled validation set; Step 4, using the self-supervised facial expression recognition model after fine-tuning evaluation, performing expression recognition on the input face image; Step 1 of constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling, specifically comprising: Step 1-1, computing the input image's direction gradient histogram HOG features ;​ Steps 1-2, processing the input image, given the input image is first divided into image patches: where C is the number of channels, which is 3 channels RGB; (H, W) is the resolution of the input image, which is (224, 224); (P, P) is the resolution of each image patch, which is (16, 16), and N is 196; Steps 1-3, input image A random transformation is performed to generate two associated enhanced view and , as a positive sample pair, are fed into the student network and the teacher network respectively; Steps 1-4: Two-dimensional image obtained through linear projection. Two enhanced views and Flattened into image patches, respectively represented as and Each of these can be considered as a sequence of visual word tokens for image blocks of length N; Step 1-5, for image visual word sequence Mask image modeling first predicts the proportion of masks according to the mask Generate a random mask Where the number of visual words is N; Then replace the image block visual word with the mask visual word [M] Generate a masked image : ； wherein, 1 means that the visual word is masked, i.e. replaced by a mask visual word [M]; 0 means that the visual word is not masked; the mask image modeling aims at recovering the masked visual words from the masked image ;​ Step 1-6, using the encoder of visual Transformer ViT-S / 16 as the backbone network of the self-supervised facial expression recognition model , / 16 indicates that the image block size is 16 and the number of layers of the encoder is 12; Step 1-7, augmented view and Transformed to a set of embedded sequences by a linear layer, an extra learnable visual token is added in the sequence; the visual token is the class visual token [CLS], which serves to aggregate the information of the entire sequence without adding any label or supervision; view After masking, the image block visual token and the set of class visual tokens [CLS] are sent to the Transformer encoder of the student network ; view Without masking, the image block visual token and the class visual token [CLS] are directly sent to the encoder of the teacher network ; Step 1-8, projection head of the self-supervised facial expression recognition model consists of a 3-layer multi-layer perceptron, MLP, a normalization layer and a weight-normalized K-dimensional fully connected layer; Step 1-9, network of self-supervised facial expression recognition model consists of a backbone network and a projection head , the student network and the teacher network share the same architecture g and have different parameter sets and ; the Transformer is pre-trained and fine-tuned using images with a size of 224x224, the total number of image block visual words N is 196, the output dimension of the shared projection head is set to 108, and the final output vector dimension K is 21168.

2. The self-supervised facial expression recognition method of claim 1, wherein, Step 2 of unsupervised pre-training on the self-supervised facial expression recognition model, specifically comprising: Step 2-1, initializing the network parameters and iteration times of the self-supervised facial expression recognition model; Step 2-2, random mask is applied to the original image to get a masked view ;​​ Step 2-3, student network for masked view Output predicted HOG features , teacher network for unmasked view Output features ; where represents converting the input to a probability distribution over dimensions, and represent the parameter sets of the student and teacher networks, respectively; By normalizing the output of the network using a softmax function, we get : ； wherein is a temperature parameter, for a temperature of of The formula for is also similar: ； Step 2-4, the training target of mask image modeling MIM is defined as: ； wherein, is the mask image modeling loss, N is the number of visual words, denotes the visual word index of the image patch, is 1 if the image patch is masked, is 0 if the image patch is not masked, i.e. the reconstruction loss is only computed on the masked image patches; is the target feature of the self-supervised facial expression recognition model, which is a one-dimensional vector. First, the HOG feature map is extracted on the whole image of the masked view, and then it is reshaped into image patches. The histogram of the masked image patch is flattened and connected into ; Step 2-5, To maintain semantic alignment, the self-supervised facial expression recognition model also performs cross-entropy loss on the class visual word [CLS] of different views as a contrastive loss , by minimizing the cross-entropy loss to match information with similar semantics: ； wherein representing the teacher network as a non-masked view outputted class semantic features, representing the student network as a masked view outputted class semantic features; Step 2-6, apply a stop gradient operator on the teacher network, only propagate the gradient through the student network; the parameters of the teacher network with the parameters of the student network Exponential moving average EMA, update rule: ； wherein, is a hyperparameter, following a cosine learning rate schedule from 0.996 to 1 during training; the self-supervised facial expression recognition model is co-optimized with the masked image modeling objective by momentum updates, so no additional training phase is needed as a pre-processing setup; Step 2-7, the self-supervised facial expression recognition model learns parameters by minimizing and using stochastic gradient descent .

3. The self-supervised facial expression recognition method of claim 2, wherein, Step 3 of linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on the labeled validation set, specifically comprising: Step 3-1, after unsupervised pre-training, a supervised linear classifier is learned on the frozen features of the teacher network without the projection head for linear probe evaluation; Step 3-2, for fine-tuning evaluation, based on the weights of the unsupervised pre-trained teacher network encoder, a supervised linear classifier is learned on the teacher network without the projection head, and the labeled validation set data is used to update the parameters of the teacher network and the classification layer, that is, the parameters of the teacher network are not frozen; Step 3-3, linear probe and fine-tuning for 100 iterations on the labeled validation set to perform linear probe and fine-tuning evaluation.

4. A self-supervised facial expression recognition device that contrastive learning combines masked image modeling, characterized in that, The device comprises the following modules: A model construction module for constructing a self-supervised facial expression recognition model combining contrastive learning and mask image modeling; A model training module for unsupervised pre-training on the self-supervised facial expression recognition model; A model evaluation module for linear probe evaluation and fine-tuning evaluation on the pre-trained self-supervised facial expression recognition model on the labeled validation set; A model recognition module for using the self-supervised facial expression recognition model after fine-tuning evaluation to perform expression recognition on the input face image; The device performs the steps of the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in any one of claims 1-3 in the process of facial expression recognition.

5. A mobile terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in any one of claims 1-3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the self-supervised facial expression recognition method combining contrastive learning and mask image modeling in any one of claims 1-3.

Images