Self-supervised small sample facial expression recognition method and system

Abstract

The present disclosure provides a small sample facial expression recognition method and system based on self-supervised vision, relating to the technical field of emotion recognition, including four pre-tasks for each original image of unlabelled facial expression image, generating four sets of loss functions through the ViT model; obtaining expression sequence and peak expression image from laboratory facial expression image and extracting space-time features, using space-time features for expression classification, and fine-tuning all parameters of the ViT model; dividing the natural environment facial expression image into support samples and query samples, then inputting into the fine-tuned ViT model to calculate the feature vector, calculating the prototype vector of each expression category based on the feature vector of the support sample, calculating the Euclidean distance between each prototype vector and the feature vector of the query sample, and determining the expression classification result of the query set sample by using the Euclidean distance; effectively reducing the training cost of the small sample facial expression recognition task and improving the classification accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of emotion recognition, in particular to a small sample facial expression recognition method and system based on self-supervision. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Facial expression is the most important, natural and direct channel for humans to express their inner feelings. Psychologist Mehrabian found through research that human emotional expression = 7% of words + 38% of voice expression + 55% of facial expression. In recent years, facial expression recognition (FER) has attracted widespread attention in the computer vision community and has broad application prospects in online education, medical health and public safety.

[0004] In order to enable the FER model to obtain better visual feature learning ability from image or video data, the usual practice is to follow the idea of supervised learning, using large-scale labeled data and multiple iterations to train a large number of parameters of the model. Although based on this mode, the FER model has made significant progress on the facial expression dataset collected in the laboratory, but when implemented in a natural environment, its recognition performance will often be greatly reduced. Today, the Vision Transformer (ViT) has achieved great success in the field of computer vision, and its performance is outstanding when pre-trained using large-scale supervised data or with some form of co-supervision. Related research shows that a fully trained ViT can be effectively applied to the facial expression recognition task and can improve the facial expression recognition accuracy in a natural environment.

[0005] However, the prior art has the following technical problems:

[0006] Facial expression data needs to be annotated by psychology professionals, which is expensive and time-consuming. Meanwhile, due to factors such as head posture or annotator subjectivity, the data annotation is inaccurate, resulting in a small amount of valid annotated data in the FER dataset, which greatly limits the performance of the ViT model. Some researchers propose that pre-training the model based on the self-supervised learning (SSL) method can enable the model to learn visual features from unlabeled images or videos. However, in existing self-supervised pre-training techniques, the pre-task lacks pertinence, resulting in poor generalization ability of the model in downstream tasks; and the number of pre-tasks is insufficient, resulting in insufficient pre-training of the model. Therefore, how to pertinently set multiple pre-tasks and jointly use a multi-scale loss function to pre-train the ViT model has become a problem to be solved.

[0007] On the other hand, due to the individual differences of human expressions, the FER dataset is usually class-uneven, making the deep learning model prone to serious overfitting in training and having low recognition accuracy for expression classes with fewer samples. Some researchers propose that the few-shot learning (FSL) technology can train the model to recognize classes that have never been seen before through a small number of labeled samples. The research of Ciubotaru and Zou et al. shows that the FSL technology can effectively solve the small sample facial expression classification problem. However, the existing FSL model usually uses a deep learning model pre-trained in other fields as a feature extractor, which will have a certain semantic gap when directly migrated to the small sample FER task, resulting in low classification accuracy of the model. Therefore, how to find an effective FSL model migration method and reduce the training cost of the small sample facial expression recognition task is a technical problem to be solved at present. SUMMARY

[0008] To solve the above problems, the present disclosure proposes a small sample facial expression recognition method and system based on self-supervision, which uses a self-supervised pre-training method to ensure the diversity of training samples, and uses a phased parameter fine-tuning method to enable the model to obtain certain prior knowledge and accelerate the model parameter tuning and migration process. The small sample facial expression recognition method recognizes facial expression classes with uneven data distribution and improves the accuracy of the FER task in a natural environment.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions:

[0010] The small sample facial expression recognition method based on self-supervision comprises:

[0011] constructing an unlabeled face data image set, a facial expression image set collected in a laboratory, and a facial expression image set collected in a natural environment;

[0012] For the unlabeled facial expression image, four kinds of transformation are performed on each original image to form four pre-task, and the ViT model is used to solve the pre-task to generate four sets of loss functions;

[0013] The expression sequence and peak expression image are obtained from the laboratory facial expression image to extract the time feature and the space feature, the time feature and the space feature are fused to obtain the space-time feature for expression classification, and all parameters of the ViT model are fine-tuned in the process;

[0014] The natural environment facial expression image is divided into support samples and query samples, the support samples and the query samples are input into the fine-tuned ViT model to calculate the output feature vector, the prototype vector of each expression category is calculated based on the feature vector of the support sample, the Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample is calculated, and the expression classification result of the query set sample is determined by using the Euclidean distance.

[0015] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0016] The small sample facial expression recognition system based on self-supervision includes:

[0017] The data preprocessing module is configured to construct an unlabeled facial data image set, a laboratory collected facial expression image set and a natural environment collected facial expression image set;

[0018] The self-supervised pre-training module is configured to, for the unlabeled facial expression image, perform four kinds of transformation on each original image to form four pre-task, and the ViT model is used to solve the pre-task to generate four sets of loss functions;

[0019] The parameter fine-tuning module is configured to obtain the expression sequence and the peak expression image from the laboratory facial expression image to extract the time feature and the space feature, the time feature and the space feature are fused to obtain the space-time feature for expression classification, and all parameters of the ViT model are fine-tuned in the process;

[0020] The small sample expression classification module is configured to divide the natural environment facial expression image into support samples and query samples, input the support samples and the query samples into the fine-tuned ViT model to calculate the output feature vector, calculate the prototype vector of each expression category based on the feature vector of the support sample, calculate the Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample, and determine the expression classification result of the query set sample by using the Euclidean distance.

[0021] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0022] A computer-readable storage medium, characterized in that, a plurality of instructions are stored therein, the instructions are suitable for being loaded by a processor of a terminal device and executing the self-supervised small sample facial expression recognition method.

[0023] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0024] A terminal device, comprising a processor and a computer-readable storage medium, the processor is used to implement instructions; the computer-readable storage medium is used to store a plurality of instructions, the instructions are suitable for being loaded by the processor and executing the self-supervised small sample facial expression recognition method.

[0025] Compared with the prior art, the present disclosure has the beneficial effects that:

[0026] First, through analysis, it is found that when training the Vision Transformer model based on the supervised learning idea, it not only depends on a large amount of manually labeled data, but also has problems such as generalization error, false association and adversarial attack, in order to avoid time-consuming and expensive data labeling, so the present disclosure uses an unlabeled face data set to pre-train the ViT model. The problem of low model classification accuracy caused by insufficient model training due to insufficient amount of effective labeled data when implementing the FER task in the past natural environment is solved.

[0027] Second, in the past self-supervised pre-training technology, the design of the pre-task lacks pertinence, which leads to poor generalization ability of the model in the downstream task; the number of pre-tasks is too small, which leads to insufficient pre-training of the model and low classification accuracy in the downstream task. The present disclosure pertinently sets four pre-tasks of image denoising reconstruction, rotation angle prediction, image block splicing and occluded image restoration, constructs a self-adaptive weighted loss function to integrate the losses generated by the four pre-tasks, and jointly pre-trains the ViT model, so that the pre-trained model has the ability to extract texture features, pose features, spatial features and context features of facial expressions.

[0028] Third, in the past parameter fine-tuning technology, the pre-trained model is prone to semantic gap when migrating to the downstream task, which leads to performance degradation of the fine-tuned model. The present disclosure designs a parameter fine-tuning method, the pre-trained ViT is used to extract the spatio-temporal features of facial expressions in the FER data set collected in the laboratory, and the expression classification result is calculated, in this process, the parameters of all layers of the ViT model are fine-tuned. The fine-tuned model has prior knowledge of the FER task, and only needs to fine-tune the parameters of the model classification layer in the small sample classification task, so as to realize accurate classification, which can effectively reduce the training cost of the small sample facial expression recognition task and improve the classification accuracy.

[0029] Fourth, the probability of human producing various expressions in the natural environment is uneven, so the collected facial expression dataset is unbalanced in samples, which leads to overfitting in the model training process and low classification accuracy for certain facial expression categories. In order to enable the model to accurately identify such small sample expressions, the present disclosure designs a small sample expression classification method, which uses the fine-tuned ViT model to extract the features of the samples and constructs a prototype network to calculate the small sample expression classification results. The identification accuracy of specific categories with fewer samples in the FER task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which form a part of this disclosure, are intended to provide further understanding of the disclosure and are incorporated herein in their entirety. The schematic embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute improper limitations on the present disclosure.

[0031] Figure 1 is a small sample facial expression recognition method flow chart based on self-supervised visual Transformer in the embodiment one of the present disclosure;

[0032] Figure 2 is an example diagram of the small sample facial expression recognition method based on self-supervised Transformer in the embodiment one of the present disclosure;

[0033] Figure 3 is a self-supervised pre-training method design diagram in the embodiment one of the present disclosure;

[0034] Figure 4 is a parameter fine-tuning method design diagram in the embodiment one of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure will be further described below in conjunction with the drawings and embodiments.

[0036] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0038] Embodiment 1

[0039] In an embodiment of the present disclosure, a small sample facial expression recognition method based on self-supervision is provided, comprising the following steps:

[0040] Step S101: constructing an unlabeled face data image set, a laboratory collected facial expression image set, and a facial expression image set collected in a natural environment;

[0041] Specifically, the original unlabeled face data set D is initialized as input * , the laboratory collected facial expression data set D , and the facial expression data set collected in a natural environment D After data preprocessing of each image of the three data sets, three new data sets are obtained as the unlabeled face data image set D, the laboratory collected facial expression image set D fer , and the facial expression image set collected in a natural environment D fsl .

[0042] Step S102: transforming each original image in four ways to form four pre-task, and solving the pre-task by a ViT model to generate four loss functions;

[0043] Specifically, on the pre-processed unlabeled face data image set D, each original image is processed by four transformation methods to form four pre-tasks of image denoising reconstruction, rotation angle prediction, image block splicing, and occluded image restoration. A ViT model is used to solve the four pre-tasks simultaneously and generate four loss functions. An adaptive weighted loss function is used to integrate the losses of the four pre-tasks to obtain the total loss function of the self-supervised pre-training stage to optimize and update the ViT model parameters;

[0044] Step S103: obtaining expression sequences and peak expression images from the laboratory facial expression images to extract time features and spatial features, fusing the time features and spatial features to obtain spatio-temporal features for expression classification, and fine-tuning all parameters of the ViT model in this process;

[0045] Specifically, on the pre-processed laboratory collected facial expression image set D fer , a peak expression image set is formed by selecting peak expression images from each expression sequence. For the jth sample, the expression sequence and the peak expression image are simultaneously sent into the pre-trained ViT to extract time features and spatial features, and the two are fused to obtain spatio-temporal features The spatio-temporal features are sent into a classification layer to calculate the expression classification result. In this process, the loss of the fine-tuning stage is calculated and optimized while fine-tuning all parameters of the pre-trained ViT model;

[0046] Step S104: Divide the facial expression images in the natural environment into support samples and query samples. Input the support samples and query samples into the fine-tuned ViT model to calculate the output feature vectors. Calculate the prototype vector of each expression category based on the feature vector of the support samples. Calculate the Euclidean distance between the prototype vector of each expression category and the feature vector of the query samples. Use the Euclidean distance to determine the expression classification result of the query set samples.

[0047] Specifically, the facial expression image set D was collected in a preprocessed natural environment. fsl Above, based on the expression category, they are divided into support sets S train and query set Q train The support samples and query samples in the support set and query set are respectively fed into the fine-tuned ViT model to calculate the feature vectors. Then, based on the feature vectors of the support samples, the prototype vector of each expression category is calculated. The Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample is calculated. The expression classification result of the query set samples is determined by using the Euclidean distance.

[0048] As one embodiment, in step S101, this disclosure applies to the unlabeled face dataset D * FER datasets collected in natural environments The data preprocessing operations were the same: each image in the original dataset was cropped using the Dlib toolkit to remove faces. The cropped image size was set to 224×224 pixels, with the aim of removing information irrelevant to facial expressions. This was also done for the FER dataset collected in the lab. The data preprocessing operations included two steps: face cropping and data augmentation. Face cropping was also implemented using the Dlib toolkit, while data augmentation employed an offline method. This involved rotating each training image by an angle of {-15°, -10°, -5°, 0°, 15°, 10°, 15°} and then flipping the rotated image along the X-axis. This expanded the dataset by 14 times, aiming to increase the amount of data in the FER dataset collected in the lab and ensure the diversity of training data during ViT model fine-tuning. After preprocessing, three new datasets D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15, D16, D17, D18, D19, D10 ...0, D11, D12, D13, D14, D15, D16, D17, D18, D19, D10, D10, D10, D11, D12, D13, D14, D15, D16, D17, D18, D19, fer and D fsl .

[0049] As one embodiment, in step S102, such as Figure 3 As shown, this disclosure simultaneously defines four self-supervised pre-tasks—image denoising and reconstruction, rotation angle prediction, image patching and stitching, and occluded image restoration—on a pre-processed unlabeled face image dataset D for pre-training the ViT model, wherein:

[0050] S1021: The design motivation of the image denoising reconstruction task is to hope that the model can extract color, texture and structure information from the image. As shown in Figure 3 (a), first, for each original image D (i) , random noise is introduced to obtain a noisy image Then, the ViT encoder is used to extract the features of the noisy image, and the linear projection layer is used as the decoder to reconstruct the image features into the image R (i) . Finally, the mean square error of the pixels D (i) of the original image and the reconstructed image R (i) after denoising is calculated as the loss function of the image denoising reconstruction task defined as follows:

[0051]

[0052] where I is the number of images, W and H are the width and height of the image, and (w, h) represents the pixel coordinates of the image.

[0053] S1022: The design motivation of the rotation angle prediction task is to hope that the ViT encoder can learn the concept of human face to prevent the disturbance of head posture in the process of facial expression recognition. As shown in Figure 3 (b), the image rotation prediction task is to train the ViT to predict the rotation angle of the input image. First, for each original image, rotate it by {0°, 90°, 180°, 270°} and label the rotation angle of each image to obtain a set of rotated images Then, the ViT encoder is used to extract the features of the rotated image, and the features are sent into the MLP layer to predict the rotation angle of the image Finally, the cross-entropy is used to calculate the loss, and the loss function of the rotation angle prediction task defined as follows:

[0054]

[0055] where each image corresponds to a rotation label a, P rot is the prediction set of all rotated images, I is the number of images, represents the parameters in the ViT encoder.

[0056] S1023: The design motivation of the image block jigsaw task is to make the model understand the components of the face, which is conducive to the model to extract spatial features related to facial expressions. As shown in Figure 3 (c), first, for each original image, it is cut into image blocks. Second, the arrangement order of the image blocks is disturbed and each image block is given a relative position label, obtaining a set of image block sequence sets of M different arrangement ways Meanwhile, the M permutations are labeled as a sequence set O. T Third, the ViT encoder is used as a feature extractor with shared parameters. The features of each image patch are extracted separately and concatenated. An MLP layer is then used to predict the arrangement of the image patches. P Finally, cross-entropy is used to calculate the loss generated during the training process. The loss function for the image patchwork task is defined as follows:

[0057]

[0058] Where I is the number of images, This represents the parameters in the ViT encoder.

[0059] S1024: The design motivation for the occluded image restoration task is to train a model to capture high-level semantic and contextual features of facial expressions. For example... Figure 3 As shown in (d), firstly, for the i-th original face image D in D... () Divide it into N image blocks to form an image block sequence. Second, from Image patches are randomly sampled at a sampling rate of 75%, their positions in the original sequence are marked, and their pixel values ​​are erased to obtain a set of occluded image patches. and the set of unoccluded image patches Third, two asymmetric ViT modules are designed as an encoder and a decoder. The ViT encoder is responsible for extracting features from the unoccluded image patches, while the decoder is responsible for reconstructing the pixels of the occluded image patches, thus reconstructing the occluded image patches from the output. Finally, the mean squared error between the predicted image patch and the original image patch is calculated to update the model parameters. Therefore, the loss function for the occluded image restoration task is... The definition is as follows:

[0060]

[0061] in, The parameters represent those in the ViT encoder, where w and h represent the coordinates of the image pixels.

[0062] In step S102 of the embodiment, the flexibility of the ViT model structure allows it to combine multiple loss functions for joint training. However, the losses generated by the four pre-tasks designed in this disclosure are multi-scale and differ in magnitude. Therefore, it is necessary to set hyperparameters to constrain the loss functions generated by different tasks to avoid the vanishing gradients of other tasks due to excessive loss generated by one task. This disclosure constructs an adaptive weighted loss function that integrates the losses of the four pre-tasks by setting four learnable hyperparameters. and For four loss functions and Weighted fusion is performed, and the homoscedasticity uncertainty method is used to simultaneously learn four sets of losses at different scales, resulting in the overall loss function for the self-supervised pre-training stage.

[0063]

[0064] As one embodiment, in step S103, such as Figure 4 As shown, this disclosure presents a parameter fine-tuning method that allows a pre-trained model to perform an expression classification task on a laboratory-controlled FER dataset to fine-tune the ViT encoder. During model fine-tuning, prior knowledge is established for recognizing facial expressions in more complex natural environments. The fine-tuned model parameters are fixed, and only the parameters of the classification layer are updated in the downstream few-shot expression classification task, effectively reducing the training cost of few-shot facial expression recognition tasks and improving classification accuracy. Detailed steps include:

[0065] S1031: Facial expression dataset collected in the lab after preprocessing. Above, from each expression sequence Peak expression images were selected to form a peak expression image set. Where J is the number of facial expression sequences in the dataset.

[0066] S1032: Transform peak expression image The image is divided into N small patches, and its embedding representation is computed through a linear projection layer. A learnable category label The pre-defined embedding representation sequence is added to the image patch, simultaneously generating a position embedding matrix. This is then added before the embedding representation of each image patch to obtain a vector matrix. Extracting spatial features from the input ViT encoder

[0067] S1033: For any expression sequence in the dataset The sequence length is normalized to N, meaning it contains N images. Each image is fed into a max-pooling layer for downsampling to a size of 16×16 pixels to address the issue of excessively large vector lengths when the entire image is fed into the transformer. Then, its embedding representation is computed through a linear projection layer. Generate a position embedding matrix according to the frame order in the image sequence. This is then added before the embedding representation of each image, along with learnable category labels. Obtain the vector matrix Extracting temporal features from the input ViT encoder

[0068]

[0069] S1034: Connect the spatial features of facial expressions and temporal features to form spatio-temporal features Then the spatio-temporal features are sent to an MLP layer to realize facial expression classification, and the calculation process is as follows:

[0070]

[0071]

[0072] wherein, is the expression classification result of the jth facial expression sequence.

[0073] S1035: Calculate the loss of the facial expression classification task The pre-trained ViT model parameters are further optimized using a loss function to realize parameter fine-tuning. The loss function is defined as follows:

[0074]

[0075] wherein, Y fer is the real label set of facial expressions, P fer is the expression prediction result set, represents the pre-training parameters of the ViT encoder.

[0076] As an embodiment, in step S104, in order to cope with the model overfitting caused by the uneven distribution of classes in the FER dataset, the present disclosure designs a small sample expression classification method, which uses the fine-tuned ViT model to extract the features of the samples, uses the sample features to construct a prototype network, and uses the prototype network to calculate the small sample expression classification result. The recognition accuracy of specific classes with fewer samples in the FER task is improved. The detailed steps include:

[0077] S1041: For the pre-processed FER dataset D fsl collected in a natural environment, divide it into a training set Train fsl and a test set Test fsl . Wherein, x and y represent images and labels respectively, C train and C test represent the number of images in the training set and the test set, Z train represents the number of classes in the training set, and Z test represents the number of classes in the test set.

[0078] S1042: Randomly set 500 groups of small sample classification scenarios for training model, each scenario contains a support set and a query set where the support set contains K classes randomly selected from Z train , each class contains C train samples selected from C trainsup , and the query set contains the remaining classes in Z train , each class contains C trainque samples.

[0079] S1043: The fine-tuned ViT encoder is used as a feature extraction module, for the query set Q train , the ViT encoder is used to extract the features of all samples in it, and for the support set S train , the ViT encoder maps all samples belonging to the same category in it to an embedding space and forms an average vector as the category prototype, then the prototype of the kth class in the support set is calculated as follows:

[0080]

[0081] where f encoder (·) is the ViT encoder, represents the cth sample in the kth class of the training set.

[0082] S1044: Calculate the probability of each class to which the samples in the query set belong using the prototypes of the support set. For a sample train from Q , calculate the Euclidean distance between it and the prototype of each query set category, then perform a normalization operation to obtain the classification probability. The probability of being predicted as the kth class can be expressed as:

[0083]

[0084] where ed(·) represents the Euclidean distance calculation function.

[0085] S1045: Calculate the loss function generated in the small sample expression classification stage Only the last layer parameters of the ViT model are updated in the process of optimizing the loss function, and the loss function calculation formula is:

[0086]

[0087] where represents the fine-tuned ViT model parameters.

[0088] Embodiment 2

[0089] In an embodiment of the present disclosure, a small sample facial expression recognition system based on self-supervision is provided, comprising:

[0090] A data preprocessing module is configured to construct an unlabeled face data image set, a laboratory collected facial expression image set, and a facial expression image set collected in a natural environment.

[0091] A self-supervised pre-training module is configured to perform four types of transformation on each original image to form four pre-task for the unlabeled facial expression image, and generate four sets of loss functions by solving the pre-task through a ViT model.

[0092] A parameter fine-tuning module is configured to extract time features and spatial features from the expression sequence and peak expression image of the laboratory facial expression image, fuse the time features and spatial features to obtain spatio-temporal features for expression classification, and fine-tune all parameters of the ViT model in the process.

[0093] A small sample expression classification module is configured to divide the facial expression image in a natural environment into support samples and query samples, input the support samples and query samples into the fine-tuned ViT model to calculate output feature vectors, calculate the prototype vector of each expression category based on the feature vector of the support sample, calculate the Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample, and determine the expression classification result of the query set sample using the Euclidean distance.

[0094] It should be noted that the above data preprocessing module, self-supervised pre-training module, parameter fine-tuning module, and small sample classification module correspond to steps S1 to S4 in Embodiment 1, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0095] Embodiment 3

[0096] In an embodiment of the present disclosure, a computer readable storage medium is provided, wherein a plurality of instructions are stored, the instructions are adapted to be loaded and executed by a processor of a terminal device, and the steps of the small sample facial expression recognition method based on self-supervision are executed.

[0097] Embodiment 4

[0098] In an embodiment of the present disclosure, a terminal device is provided, comprising a processor and a computer readable storage medium, the processor is used to implement instructions, and the computer readable storage medium is used to store a plurality of instructions, the instructions are adapted to be loaded and executed by the processor, and the steps of the small sample facial expression recognition method based on self-supervision are executed.

[0099] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0100] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 steps for functionally implementing the steps in one or more flow or blocks

[0101] Although the present disclosure has been described with reference to specific implementations, it will be apparent to those skilled in the art that various modifications and changes can be made to the implementations without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the disclosure is not to be limited by what has been presented in the description above, but is to be understood as including all variations falling within the scope of the present disclosure.

Claims

1. A self-supervised few-sample facial expression recognition method, characterized in that, include: Construct a set of unlabeled facial data images, a set of facial expression images collected in the laboratory, and a set of facial expression images collected in the natural environment; For unlabeled facial expression images, four transformations are performed on each original image to form four pre-tasks. The ViT model is used to solve the pre-tasks and generate four sets of loss functions. For the preprocessed unlabeled face dataset, four pre-tasks are constructed: image denoising and reconstruction, rotation angle prediction, image patching, and occluded image restoration. An adaptive weighted loss function is used to integrate the losses of the four pre-tasks to optimize and update the ViT model parameters, specifically: Four transformation methods are applied to each original image to construct four pre-tasks: image denoising and reconstruction, rotation angle prediction, image patching, and occluded image restoration. The ViT model is then used to solve these four pre-tasks simultaneously and generate four sets of losses. , , and The overall loss function for the self-supervised pre-training stage is obtained by integrating the losses from the four pre-tasks using an adaptive weighted loss function. ; Calculate and optimize the loss function Update the ViT model parameters; Facial expression sequences and peak facial expression images were obtained from laboratory facial expression images to extract temporal and spatial features. The temporal and spatial features were then fused to obtain spatiotemporal features for expression classification. During this process, all parameters of the ViT model were fine-tuned. For the For each sample, its facial expression sequence and peak facial expression image are simultaneously fed into a pre-trained ViT dataset to extract temporal and spatial features. These two features are then fused to obtain the spatiotemporal features. The spatiotemporal features are fed into the classification layer to calculate the expression classification result. During this process, the loss in the fine-tuning stage is calculated and optimized. Meanwhile, all parameters of the pre-trained ViT model are fine-tuned; Natural environment facial expression images are divided into support samples and query samples. The support samples and query samples are respectively input into the fine-tuned ViT model to calculate the output feature vector. Based on the feature vector of the support samples, the prototype vector of each expression category is calculated. The Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample is calculated. The expression classification result of the query set samples is determined by using the Euclidean distance.

2. The self-supervised few-sample facial expression recognition method as described in claim 1, characterized in that, The steps for constructing the unlabeled facial image dataset, the laboratory-collected facial expression image dataset, and the natural environment-collected facial expression image dataset include: A dataset of unlabeled faces, a dataset of facial expression images collected in a natural environment, and a dataset of facial expression images collected in a laboratory are acquired. Images from each dataset are extracted and preprocessed. Face cropping is performed on each image in the unlabeled face dataset and the dataset of facial expression images collected in a natural environment. Face cropping and data augmentation are performed on each image in the dataset of facial expression images collected in a laboratory. The preprocessed images are then used to construct a dataset of unlabeled face images, a dataset of facial expression images collected in a laboratory, and a dataset of facial expression images collected in a natural environment.

3. The self-supervised few-sample facial expression recognition method as described in claim 1, characterized in that, For the preprocessed FER dataset collected in the lab, the spatiotemporal features of the images were extracted using a pre-trained ViT model. The spatiotemporal features were then used to calculate the facial expression classification results. All parameters of the pre-trained ViT model were fine-tuned, specifically as follows: Peak expression images are selected from each expression sequence to form a peak expression image set; the expression sequence and peak expression images are simultaneously fed into the pre-trained ViT to extract temporal and spatial features, and the two are fused to obtain spatiotemporal features; Spatiotemporal features are fed into the classification layer to calculate the facial expression classification result and generate a loss function. ; Calculate and optimize the loss function During this process, all parameters of the ViT model are fine-tuned.

4. The self-supervised few-sample facial expression recognition method as described in claim 1, characterized in that, The preprocessed facial expression dataset collected from the natural environment is divided into a support set and a query set. A fine-tuned ViT model is used to extract features from the support and query set samples. A prototype network is constructed based on the features of the support set samples. The prototype network is then used to calculate the classification results of small sample facial expressions. Specifically: The FER dataset collected from the natural environment is divided into training and test sets. The training and test sets are used to construct small sample classification scenarios, where each scenario contains a support set and a query set. For the query set, the features of all samples in the query set are extracted using a fine-tuned ViT encoder. For the support set, the fine-tuned ViT encoder maps all samples belonging to the same category to the same embedding space and calculates the average vector to obtain the category prototype. Calculate the Euclidean distance from each query set sample to the prototype of each query set category, then perform a normalization operation to obtain the small sample expression classification probability, and generate the loss function. ; Optimize the loss function During the optimization of the loss function, only the parameters of the last layer of the ViT model are updated.

5. The self-supervised few-sample facial expression recognition method as described in claim 1, characterized in that, The occluded image restoration task involves training a model to capture high-level semantic and contextual features of facial expressions. This includes: First, for the original face image, it is divided into N image blocks to form an image block sequence; then, image blocks are randomly sampled at a 75% sampling rate, their positions in the original sequence are marked, and their pixel values ​​are erased, resulting in a set of occluded image blocks and a set of unoccluded image blocks; two asymmetric ViT modules are designed as an encoder and a decoder. The ViT encoder is responsible for extracting features from the unoccluded image blocks, while the decoder is responsible for reconstructing the pixels of the occluded image blocks, and the output is reconstructed as occluded image blocks.

6. The self-supervised few-sample facial expression recognition method as described in claim 2, characterized in that, The data augmentation method involves dividing each training image according to... The dataset is expanded by 14 times by rotating it by an angle and flipping the rotated image on the X-axis.

7. A self-supervised few-sample facial expression recognition system, characterized in that, Specifically, the self-supervised few-sample facial expression recognition method as described in any one of claims 1-6 includes: The data preprocessing module is configured to construct an unlabeled facial image dataset, a set of facial expression images collected in the laboratory, and a set of facial expression images collected in the natural environment. The self-supervised pre-training module is configured to perform four pre-tasks on each original image for unlabeled facial expression images, and generate four sets of loss functions through the ViT model. The parameter fine-tuning module is configured to extract temporal and spatial features from facial expression images in the laboratory, fuse the temporal and spatial features to obtain spatiotemporal features for expression classification, and fine-tune all parameters of the ViT model. For the For each sample, its facial expression sequence and peak facial expression image are simultaneously fed into a pre-trained ViT dataset to extract temporal and spatial features. These two features are then fused to obtain the spatiotemporal features. The spatiotemporal features are fed into the classification layer to calculate the expression classification result. During this process, the loss in the fine-tuning stage is calculated and optimized. Meanwhile, all parameters of the pre-trained ViT model are fine-tuned; The few-sample facial expression classification module is configured to divide natural environment facial expression images into support samples and query samples. The support samples and query samples are respectively input into the fine-tuned ViT model to calculate the output feature vector. Based on the feature vector of the support samples, the prototype vector of each expression category is calculated. The Euclidean distance between the prototype vector of each expression category and the feature vector of the query sample is calculated. The Euclidean distance is used to determine the expression classification result of the query set samples.

8. A computer-readable storage medium, characterized in that, It stores multiple instructions, which are adapted to be loaded and executed by the processor of the terminal device using the self-supervised few-sample facial expression recognition method according to any one of claims 1-6.

9. A terminal device, characterized in that, The method includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions adapted to be loaded by the processor and executed as described in any one of claims 1-6.

Images