Facial expression recognition method based on self-supervised contrastive learning and semi-supervised hybrid

Abstract

The present invention relates to the field of facial expression recognition technology, and discloses a facial expression recognition method based on self-supervised contrastive learning and semi-supervised hybrid, comprising the following steps: S1, by performing instance-level differentiation on an unlabeled facial expression dataset, and adopting a BYOL model for training for 200 rounds, finally obtaining an output feature layer of a backbone model to be used as a pre-training weight for a subsequent semi-supervised method; S2, in order to better use the contrastive learning model, adopting the pytorch_lighting framework for training, and adopting a special weight extraction method, the specific process is: traversing the pytorch_lighting weight format, according to the weight specification of resnrt, sequentially extracting the contents of the weight dictionary and assigning them one by one to the initial structure of resnet, and finally obtaining the desired partial weights before performing semi-supervised fine-tuning training. The method disclosed in the present invention is based on a self-supervised method, and then cooperates with a semi-supervised method for efficient learning, thereby achieving low-cost and high-quality facial expression recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, in particular to a facial expression recognition method based on a mixture of self-supervised contrastive learning and semi-supervised learning. Background Art

[0002] Expressions are an important way to convey emotion in interpersonal communication. Facial expression recognition uses computers to extract facial expression features from detected faces, enabling computers to interpret and process facial expressions according to human thinking and respond to people's needs, thereby establishing a friendly and intelligent human-computer interaction environment. In recent years, facial expression recognition has played a vital role in a range of applications, including human-computer interaction, social robotics, and mental health monitoring.

[0003] Most current facial expression recognition technologies rely on large-scale, high-quality labeled facial expression datasets. However, these high-quality labeled datasets often require significant time and effort from professionals, resulting in extremely high costs. Furthermore, varying levels of accuracy can lead to inconsistent and noisy labels, further hindering the model's learning capabilities. Therefore, it is imperative to develop a method that reduces reliance on labeled data while maintaining high facial expression recognition performance. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a facial expression recognition method based on self-supervised contrastive learning and semi-supervised hybrid, which solves the problems that the current facial expression recognition technology is time-consuming and labor-intensive, and the cost is extremely high, and different levels will lead to inconsistent labels and noisy labels, which will further hinder the learning ability of the model.

[0006] (2) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a facial expression recognition method based on a hybrid of self-supervised contrastive learning and semi-supervised learning, comprising the following steps:

[0008] S1. We perform instance-level discrimination on an unlabeled facial expression dataset and train the BYOL model for 200 rounds to obtain the output feature layer of the backbone model, which is used as the pre-training weight for the subsequent semi-supervised method.

[0009] S2. To better utilize the contrastive learning model, we used the pytorch_lighting framework for training and adopted a special weight extraction method. The specific process is as follows: traversing the pytorch_lighting weight format, extracting the contents of the weight dictionary one by one according to the ResNet weight specification, and assigning them to the initial structure of ResNet one by one. Finally, after obtaining the desired partial weights, semi-supervised fine-tuning training is performed.

[0010] S3, first use the segmentation algorithm on the facial expression dataset RAF-DB to divide it into a labeled dataset and an unlabeled dataset;

[0011] S4. For labeled datasets, the pre-trained model is directly used for fully supervised fine-tuning. For unlabeled datasets, adaptive confidence boundaries are used to divide the dataset into high-confidence unlabeled datasets and low-confidence unlabeled datasets. For high-confidence unlabeled datasets, the fully supervised fine-tuning method is adopted, and for low-confidence unlabeled datasets, the supervised contrastive learning method SupCon is used for supervised contrast training.

[0012] S5. Extract the backbone of the finally trained weights, and then implement the facial expression recognition design to achieve the effect of inputting a picture and automatically determining which facial expression category the picture belongs to.

[0013] Preferably, in S1, instance-level differentiation mainly refers to maximizing the similarity of the same image at the representation level and minimizing the similarity of different instances (negative sample pairs) under different data augmentation maps (positive sample pairs).

[0014] Preferably, in S3, the segmentation algorithm adopts specific rules. When the index is 1, all labels are shuffled, and then the first 250 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs; when the label index is 0 and 2-6, all labels are shuffled, and then the first (4000-250) / 6 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs. Here, the 4000 label data value can be modified manually.

[0015] (3) Beneficial effects

[0016] The present invention provides a facial expression recognition method based on a hybrid of self-supervised contrastive learning and semi-supervised learning, which has the following beneficial effects:

[0017] The method disclosed in this invention is based on a self-supervised approach, which is then combined with a semi-supervised approach for efficient learning, thereby achieving low-cost, high-quality facial expression recognition. It can also significantly reduce the reliance on large-scale, high-quality labeled datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention provides a technical solution: a facial expression recognition method based on a hybrid of self-supervised contrastive learning and semi-supervised learning, comprising the following steps:

[0021] S1. Perform instance-level discrimination on an unlabeled facial expression dataset. Instance-level discrimination primarily involves maximizing the similarity of identical images and minimizing the similarity of different instances (negative pairs) at the representation level under different data augmentation maps (positive pairs). Train the model for 200 rounds using either BYOL or SimSiam models, ultimately obtaining the output feature layer of the backbone model to serve as pre-training weights for subsequent semi-supervised methods.

[0022] S2. To better utilize the contrastive learning model, we used the pytorch_lighting framework for training and adopted a special weight extraction method. The specific process is as follows: traversing the pytorch_lighting weight format, extracting the contents of the weight dictionary one by one according to the ResNet weight specification, and assigning them to the initial structure of ResNet one by one. Finally, after obtaining the desired partial weights, semi-supervised fine-tuning training is performed.

[0023] S3. First, a segmentation algorithm is used on the facial expression dataset RAF-DB to divide the labeled dataset and the unlabeled dataset. The segmentation algorithm adopts specific rules. When the index is 1, all labels are shuffled, and then the first 250 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs; when the label index is 0 and 2-6, all labels are shuffled, and then the first (4000-250) / 6 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs. Here, the 4000 label data value can be manually modified;

[0024] S4. For labeled datasets, the pre-trained model is directly used for fully supervised fine-tuning. For unlabeled datasets, adaptive confidence boundaries are used to divide the dataset into high-confidence unlabeled datasets and low-confidence unlabeled datasets. For high-confidence unlabeled datasets, the fully supervised fine-tuning method is adopted, and for low-confidence unlabeled datasets, the supervised contrastive learning method SupCon is used for supervised contrast training.

[0025] S5. Extract the backbone of the finally trained weights, and then implement the facial expression recognition design to achieve the effect of inputting a picture and automatically determining which facial expression category the picture belongs to.

[0026] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0027] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A facial expression recognition method based on a hybrid of self-supervised contrastive learning and semi-supervised learning, characterized in that: The following steps are involved: S1. Perform instance-level discrimination on an unlabeled facial expression dataset and train it for 200 rounds using a BYOL or SimSiam model to obtain the output feature layer of the backbone model, which is used as the pre-training weight for subsequent semi-supervised methods. S2. To better utilize the contrastive learning model, we used the pytorch_lighting framework for training and adopted a special weight extraction method. The specific process is as follows: traversing the pytorch_lighting weight format, extracting the contents of the weight dictionary one by one according to the ResNet weight specification, and assigning them to the initial structure of ResNet one by one. Finally, after obtaining the desired partial weights, semi-supervised fine-tuning training is performed. S3. First, a segmentation algorithm is used on the facial expression dataset RAF-DB to divide the labeled dataset and the unlabeled dataset. The segmentation algorithm adopts specific rules. When the index is 1, all labels are shuffled, and then the first 250 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs. When the label index is 0 and 2-6, all labels are shuffled, and then the first (4000-250) / 6 in the list are taken as train_labeled_idxs, and the rest are train_unlabeled_idxs. Here, the 4000 label data values are modifiable. S4. For labeled datasets, directly use the pre-trained model to perform fully supervised fine-tuning; For unlabeled datasets, on the one hand, adaptive confidence boundaries are adopted to divide this part of the dataset into high-confidence unlabeled datasets and low-confidence unlabeled datasets; For high-confidence unlabeled datasets, a fully supervised fine-tuning method is adopted, and for low-confidence unlabeled datasets, a supervised contrastive learning method SupCon is used for supervised contrastive training; S5. Extract the backbone of the finally trained weights, and then implement the facial expression recognition design to achieve the effect of inputting a picture and automatically determining which facial expression category the picture belongs to.

2. The facial expression recognition method based on self-supervised contrastive learning and semi-supervised hybrid according to claim 1, characterized in that: In S1, instance-level differentiation mainly refers to maximizing the similarity of the same image and minimizing the similarity of different instances at the representation level under different data augmentation graphs.

Images