Self-calibration negative learning based semi-supervised facial expression recognition method and system

Abstract

The present application relates to the field of computer vision, and proposes a semi-supervised facial expression recognition method and system based on self-calibration negative learning, through designing a kind of class center contrast module, using momentum updating class center, class confidence and adaptive threshold, high confidence samples are stabilized and intra-class aggregation is driven, and a Top-K guided pseudo label optimization module is also designed, Top-K guided pseudo label optimization is used for low confidence fuzzy samples, negative consistency optimization and low quality contrast loss optimization are applied to strengthen the utilization of fuzzy sample information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a semi-supervised facial expression recognition method and system based on self-calibration negative learning. BACKGROUND

[0002] With the rapid development of computer intelligence industry, deep learning is gradually widely used in various fields, especially in the field of facial expression recognition. However, in the prior art, supervised deep learning often needs to train and optimize the model through a large number of high-quality labeled samples, and the labeling process of the samples is usually difficult, expensive and time-consuming, which makes the design of semi-supervised deep learning scheme gradually become a crucial link.

[0003] In the prior art, there are still many problems in the traditional semi-supervised facial expression recognition. First, the quality of the label predicted by the model is not high, because the model does not have enough discrimination between classes, which leads to a large difference between the real label and the predicted label in the process of label prediction of unlabeled samples, and finally leads to low quality of the predicted label of the whole unlabeled sample. Secondly, the utilization of low-quality data (the model predicts each class in the data, and the probability value of the maximum class in the predicted value is lower than the set threshold) is not high, and the potential information of the low-quality data is not fully developed, which leads the model to only focus on high-quality data and ignore the processing of low-quality data.

[0004] Therefore, how to design a semi-supervised facial expression recognition method to improve the recognition accuracy has become a problem to be solved. SUMMARY

[0005] Therefore, the present application proposes a semi-supervised facial expression recognition method and system based on self-calibration negative learning, which designs a class center contrast module, uses momentum updated class center, class confidence and adaptive threshold, stabilizes high confidence samples and drives intra-class aggregation, and designs a Top-K guided pseudo label optimization module to optimize the low confidence fuzzy samples using Top-K guided pseudo label optimization, applies negative consistency optimization and low quality contrast loss optimization to strengthen the utilization of fuzzy sample information.

[0006] The semi-supervised facial expression recognition method based on self-calibration negative learning proposed by the present application comprises:

[0007] Obtain a facial expression image and pre-process it, divide the pre-processed facial expression image into data to obtain a labeled data set and an unlabeled data set;

[0008] The labeled data set and the unlabeled data set are respectively subjected to enhancement processing to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, the enhancement processing including weak enhancement processing and strong enhancement processing;

[0009] The weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set are respectively subjected to feature extraction to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features;

[0010] The weakly enhanced unlabeled data set output features are subjected to reliable sample division according to a Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset, and the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features are subjected to pseudo label optimization according to a Top-K guided algorithm, the pseudo label optimization including negative learning optimization and low-quality contrast loss optimization;

[0011] The weakly enhanced labeled data set output features are subjected to reliable sample division according to a class center contrast module to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, and class center updating is performed according to the labeled sample ambiguous subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result, the reliable sample division being based on class confidence and a class adaptive threshold.

[0012] In summary, according to the above-mentioned semi-supervised facial expression recognition method based on self-calibration negative learning, a class center contrast module is designed, and the class center, class confidence and adaptive threshold are updated using momentum, which stabilizes the high-confidence samples and drives the intra-class aggregation. A Top-K guided pseudo label optimization module is also designed to optimize the Top-K guided pseudo label for low-confidence ambiguous samples, and negative consistency optimization and low-quality contrast loss optimization are applied to enhance the utilization of ambiguous sample information. Specifically, facial expression images are obtained and preprocessed, the preprocessed facial expression images are subjected to data division processing to obtain a labeled data set and an unlabeled data set, the labeled data set and the unlabeled data set are subjected to enhancement processing respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, the enhancement processing includes weak enhancement processing and strong enhancement processing, the weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set are subjected to feature extraction respectively to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features, the weakly enhanced unlabeled data set output features are subjected to reliable sample division according to the Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset, and the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features are subjected to pseudo label optimization according to the Top-K guided algorithm, the pseudo label optimization includes negative learning optimization and low-quality contrast loss optimization, the weakly enhanced labeled data set output features are subjected to reliable sample division according to the class center contrast module to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, and the labeled sample ambiguous subset, the labeled sample reliable subset and the unlabeled sample reliable subset are subjected to class center updating to obtain a class prediction result, and the reliable sample division is based on the class confidence and the class adaptive threshold.

[0013] Further, the step of enhancing the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set specifically includes:

[0014] The facial expression images are obtained and preprocessed, the facial expression images are subjected to face detection and key point alignment processing, and then subjected to scaling and cropping to match the standard input size;

[0015] The preprocessed facial expression images are subjected to data division processing to obtain a labeled data set and an unlabeled data set, and the labeled data set and the unlabeled data set are as follows:

[0016] ,

[0017] wherein, denotes a labeled dataset, denotes an unlabeled dataset, denotes a labeled image dataset, denotes a real label of the labeled image dataset after one-hot encoding, denotes a total number of the labeled image dataset, denotes an unlabeled image dataset, denotes a total number of the unlabeled image dataset, i and j denote the ordinal number of data in different datasets, respectively;

[0018] The labeled dataset and the unlabeled dataset are respectively subjected to an enhancement process, the enhancement process including once weak enhancement processing on the labeled dataset, twice weak enhancement processing and once strong enhancement processing on the unlabeled dataset, to obtain a weak enhancement labeled dataset, a weak enhancement unlabeled dataset and a strong enhancement unlabeled dataset, and the enhancement process is specifically as follows:

[0019] ,

[0020] ,

[0021] ,

[0022] , ,

[0023] ,

[0024] wherein, denotes a weak enhancement labeled dataset, denotes weak enhancement on the labeled dataset, denotes a weak enhancement labeled dataset, denotes a first weak enhancement unlabeled dataset, denotes a second weak enhancement unlabeled dataset, denotes weak enhancement on the unlabeled dataset, denotes a strong enhancement unlabeled dataset, denotes strong enhancement on the unlabeled dataset, denotes a weak enhancement unlabeled dataset obtained by twice weak enhancement processing, respectively, denotes a strong enhancement unlabeled dataset.

[0025] Further, the step of respectively subjecting the labeled dataset and the unlabeled dataset to the enhancement process further includes:

[0026] The feature extraction is performed on the weakly augmented labeled data set, the weakly augmented unlabeled data set and the strongly augmented unlabeled data set respectively to obtain weakly augmented labeled data set output features, weakly augmented unlabeled data set output features and strongly augmented unlabeled data set output features, and the feature extraction is specifically as follows:

[0027] ,

[0028] wherein, weakly augmented labeled data set output features and weakly augmented labeled data set prediction values, weakly augmented unlabeled data set output features and weakly augmented unlabeled data set prediction values obtained by the two times of weak augmentation processing respectively, strongly augmented unlabeled data set output features and strongly augmented unlabeled data set prediction values.

[0029] Further, the step of performing reliable sample division on the weakly augmented unlabeled data set output features by the Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset specifically includes:

[0030] The weakly augmented unlabeled data set output features and the weakly augmented unlabeled data set prediction values are input into the Top-K guided pseudo label optimization module, and a weighted average processing is performed on the weakly augmented unlabeled data set prediction values to obtain an average sample prediction probability distribution, and the average sample prediction probability distribution is specifically as follows:

[0031] ,

[0032] wherein, the average sample prediction probability distribution, and weakly augmented unlabeled data set prediction values corresponding to the weakly augmented unlabeled data set obtained by the two times of weak augmentation processing respectively;

[0033] According to the average sample prediction probability distribution, if the class confidence corresponding to the maximum probability of the average sample prediction probability distribution is greater than a class adaptive threshold, the corresponding weakly augmented unlabeled data set output features are determined as reliable samples and are divided into an unlabeled sample reliable subset.

[0034] The unlabeled sample reliable subset is input into a class center comparison module, and the remaining weakly augmented unlabeled data set output features in the weakly augmented unlabeled data set are subjected to pseudo label optimization.

[0035] Further, the remaining weakly augmented unlabeled dataset output features and the strongly augmented unlabeled dataset output features are output according to a Top-K guided algorithm for a pseudo-label optimization step, specifically comprising:

[0036] The pseudo-label optimization is performed on the remaining weakly augmented unlabeled dataset output features and the strongly augmented unlabeled dataset output features according to a Top-K guided algorithm, which is specifically as follows:

[0037] ,

[0038] Among them, represents the class index of the first K maximum probability, represents the first K class before screening, K represents the number of screened classes, represents the average sample prediction probability distribution.

[0039] Further, the pseudo-label optimization step specifically comprises:

[0040] The pseudo-label optimization includes negative learning optimization and low-quality contrast loss optimization;

[0041] The negative learning optimization sets the hard label of the corresponding position of the first K maximum probability class index to 0, and sets the hard label of the corresponding position of the class index not belonging to the first K maximum probability class index to 1, to obtain a negative class indication mask, which includes a weakly augmented unlabeled data negative class indication mask and a strongly augmented unlabeled data negative class indication mask. The specific algorithm for obtaining the negative class indication mask is as follows:

[0042] ,

[0043] ,

[0044] ,

[0045] Among them, represents the hard label assignment of the weakly augmented unlabeled dataset, represents the hard label assignment of the strongly augmented unlabeled dataset, represents the maximum sample prediction probability distribution of the strongly augmented unlabeled dataset, represents the class confidence threshold of the class at the moment, represents the negative class indication mask;

[0046] ​According to the negative consistency loss function, the negative consistency loss function is specifically as follows:

[0047] ,

[0048] Among them, represents the negative consistency loss, represents the fuzzy sample corresponding to the unlabeled data classification, C represents the total number of categories, and c represents the category serial number, represents the strong enhanced unlabeled data set prediction value, represents the minimum value preventing the difference from being 0;

[0049] The low-quality contrast loss optimization forms a positive and negative sample pair with the weak enhanced unlabeled data set output feature and the strong enhanced unlabeled data set output feature, and optimizes according to the low-quality contrast loss function, and the low-quality contrast loss function is specifically as follows:

[0050] ,

[0051] Among them, represents the low-quality contrast loss, represents the unlabeled available sample, i and j respectively represent the serial number of data in different data sets, represents the weak enhanced unlabeled data set output feature corresponding to the weak enhanced unlabeled data set obtained by twice weak enhancement processing, represents the strong enhanced unlabeled data set output feature, represents, is a temperature coefficient, the function is a cosine similarity function.

[0052] Further, the class center updating according to the labeled sample fuzzy subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain the category prediction result step specifically includes:

[0053] According to the class center contrast module, the weak enhanced labeled data set output feature is divided into reliable samples to obtain a labeled sample fuzzy subset and a labeled sample reliable subset, and the reliable sample division is based on the category confidence and the category adaptive threshold. The update of the category confidence and the category adaptive threshold is as follows:

[0054] ,

[0055] ,

[0056] Among them, represents the category at the moment The average of the reliable sample prediction probabilities, This represents the sum of the number of samples in the reliable subset of labeled samples and the reliable subset of unlabeled samples. yes The category at any time is The probability value corresponding to a reliable sample. express The category at any time is Category confidence threshold, express The category at any time is Category confidence threshold, This represents the momentum used to smooth the update of the class confidence threshold. Indicates a category-adaptive threshold;

[0057] The class center update is performed based on the fuzzy subset of labeled samples, the reliable subset of labeled samples, and the reliable subset of unlabeled samples. The specific algorithm for the class center update is as follows:

[0058] ,

[0059] in, express At this moment The class center corresponding to each category This represents the momentum used for smoothing class center updates. This indicates when to start updating the class center. This represents the number of samples in the reliable subset of labeled samples. This represents the number of samples in the reliable subset of unlabeled samples. i and j These represent the ordinal numbers of data in different datasets. Indicates the first time used for class center update The corresponding features of each labeled sample Indicates the first time used for class center update Corresponding features of unlabeled samples;

[0060] The class boundary discrimination optimization is performed based on the contrastive learning loss. The specific algorithm for the class boundary discrimination optimization is as follows:

[0061] ,

[0062] ,

[0063] in, This represents the optimization loss for class boundary discrimination. This represents the number of samples in the fuzzy subset of labeled samples. denotes the distance between the labeled ambiguous sample and the true class label, denotes the high-dimensional feature, denotes the class center of the th class, denotes the class center corresponding to the true class of the th sample, denotes the temperature coefficient, denotes the distance between the high-dimensional feature and the class center corresponding to the true class of the th sample.

[0064] The application provides a semi-supervised facial expression recognition system based on self-calibration negative learning, which comprises:

[0065] A data division module is configured to acquire facial expression images and perform preprocessing, and perform data division processing on the preprocessed facial expression images to obtain a labeled data set and an unlabeled data set.

[0066] A data enhancement module is configured to perform enhancement processing on the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, wherein the enhancement processing comprises weak enhancement processing and strong enhancement processing.

[0067] A feature extraction module is configured to perform feature extraction on the weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set respectively to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features.

[0068] A Top-K guided pseudo label optimization module is configured to perform reliable sample division on the weakly enhanced unlabeled data set output features to extract an unlabeled sample reliable subset, and perform pseudo label optimization on the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features according to a Top-K guided algorithm, wherein the pseudo label optimization comprises negative learning optimization and low-quality contrast loss optimization.

[0069] A class center contrast module is configured to perform reliable sample division on the weakly enhanced labeled data set output features to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, and perform class center updating according to the labeled sample ambiguous subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result, wherein the reliable sample division is based on class confidence and a class adaptive threshold.

[0070] The application provides a semi-supervised facial expression recognition method and system based on self-calibration negative learning, which has the following beneficial effects​

[0071] (1) Improve the quality of pseudo-labels. By incorporating high-confidence unlabeled samples into class center updating and calibrating pseudo-labels with reference to class centers, pseudo-labels can be made closer to the true class distribution, reducing the risk of amplifying false pseudo-labels in iterative training, thereby improving training stability and final recognition accuracy.

[0072] (2) Effectively utilize low-confidence samples to improve data utilization. The Top-K guided pseudo-label optimization module utilizes the "non-class" information and enhanced consistency information in low-confidence unlabeled samples to make discarded samples contribute to discriminative constraints, significantly increasing the number of effective training samples and enhancing the learning ability of boundary samples.

[0073] (3) Enhance class discriminability and feature discrimination ability. The class center comparison module compares and learns labeled ambiguous samples with class centers, strengthens intra-class compactness and inter-class separability, improves the discriminability of feature representation, and is beneficial to distinguishing subtle expression differences, thereby improving the recognition ability of the model among multiple expression classes.

[0074] (4) Low cost and easy to engineer and deploy. This method does not require additional manual labeling, and performance can be improved through the improvement of related modules. A common lightweight backbone network is used for feature extraction, which has small computational complexity and is easy to integrate and deploy in existing semi-supervised training processes.

[0075] The application also provides a storage medium storing one or more programs, which are executed by a processor to implement the semi-supervised facial expression recognition method based on self-calibration negative learning as described above.

[0076] The application also provides a computer device comprising a memory and a processor, wherein:

[0077] The memory is used to store a computer program;

[0078] The processor is used to execute the computer program stored in the memory to implement the semi-supervised facial expression recognition method based on self-calibration negative learning as described above. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 The flowchart of the semi-supervised facial expression recognition method based on self-calibration negative learning for the first embodiment of the application;

[0080] Figure 2 The structure diagram of the semi-supervised facial expression recognition system based on self-calibration negative learning for the second embodiment of the application;

[0081] Figure 3A logic flowchart of a first embodiment of the present application.

[0082] The following detailed description will further explain the present application with reference to the above mentioned figures. DETAILED DESCRIPTION

[0083] For the purpose of promoting an understanding of the present application, the present application will now be described in greater detail with reference to the figures. Several embodiments of the present application are illustrated in the figures. However, the present application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present application to those skilled in the art.

[0084] It should be noted that when an element is referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. When an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can also be present. The terms "vertical", "horizontal", "left", "right", and the like as used herein are for the purpose of illustration only.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0086] Referring to Figure 1 , a flowchart of a semi-supervised facial expression recognition method based on self-calibration negative learning according to a first embodiment of the present application is shown. The semi-supervised facial expression recognition method based on self-calibration negative learning includes steps S01 to S05, wherein:

[0087] Step S01: Obtain facial expression images and perform preprocessing, and perform data division processing on the preprocessed facial expression images to obtain a labeled data set and an unlabeled data set;

[0088] It should be noted that the specific logic flow of the present embodiment is described with reference to Figure 3 , wherein CCM represents a class center contrast module, TGPLO represents a Top-K guided pseudo label optimization module, represents a cross-entropy loss, represents a maximum probability value of a sample predicted probability distribution, represents a maximum probability value of an average sample predicted probability distribution, represents a class confidence threshold, WA represents a weak enhancement operation, and SA represents a strong enhancement operation.

[0089] Step S02: Perform enhancement processing on the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set;

[0090] It should be noted that in the embodiment, the enhancement processing includes weak enhancement processing and strong enhancement processing, the facial expression image is obtained and preprocessed, face detection and key point alignment processing are performed on the facial expression image, and scaling and cropping are performed to match the standard input size.

[0091] The preprocessed facial expression image is subjected to data division processing to obtain a labeled data set and an unlabeled data set, and the labeled data set and the unlabeled data set are as follows:

[0092] ,

[0093] Among them, the labeled data set, the unlabeled data set, the labeled image data, the true label corresponding to the labeled image data after one-hot encoding, the total number of labeled image data, the unlabeled image data, the total number of unlabeled image data, i and j respectively represent the serial number of data in different data sets;

[0094] The labeled data set and the unlabeled data set are subjected to enhancement processing, the enhancement processing includes once weak enhancement processing on the labeled data set, twice weak enhancement processing and once strong enhancement processing on the unlabeled data set, to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, and the enhancement processing is as follows:

[0095] ,

[0096] ,

[0097] ,

[0098] , ,

[0099] ,

[0100] Among them, the weakly enhanced labeled data, denotes weak augmentation on the labeled dataset, denotes weak augmentation on the labeled dataset, denotes the first weak augmented unlabeled data, denotes the second weak augmented unlabeled data, denotes weak augmentation on the unlabeled dataset, denotes the strongly augmented unlabeled data, denotes strong augmentation on the unlabeled dataset, denotes the weak augmented unlabeled dataset obtained by the first weak augmentation process, denotes the weak augmented unlabeled dataset obtained by the second weak augmentation process,

[0101] Step S03: feature extraction is performed on the weak augmented labeled dataset, the weak augmented unlabeled dataset and the strongly augmented unlabeled dataset respectively to obtain weak augmented labeled dataset output features, weak augmented unlabeled dataset output features and strongly augmented unlabeled dataset output features;

[0102] It should be noted that in this embodiment, feature extraction is performed on the weak augmented labeled dataset, the weak augmented unlabeled dataset and the strongly augmented unlabeled dataset respectively to obtain weak augmented labeled dataset output features, weak augmented unlabeled dataset output features and strongly augmented unlabeled dataset output features. The feature extraction is specifically as follows:

[0103] ,

[0104] wherein, denote the weak augmented labeled dataset output features and the weak augmented labeled dataset prediction value respectively, denote the weak augmented unlabeled dataset output features and the weak augmented unlabeled dataset prediction value corresponding to the weak augmented unlabeled dataset obtained by the first weak augmentation process respectively, denote the strongly augmented unlabeled dataset output features and the strongly augmented unlabeled dataset prediction value respectively.

[0105] Step S04: reliable sample division is performed on the weak augmented unlabeled dataset output features according to the Top-K guided pseudo label optimization module to extract a reliable subset of unlabeled samples, and then pseudo label optimization is performed on the remaining weak augmented unlabeled dataset output features and the strongly augmented unlabeled dataset output features according to the Top-K guided algorithm;

[0106] It should be noted that in the embodiment, the pseudo label optimization includes negative learning optimization and low quality contrast loss optimization, the weak enhanced unlabeled data set output feature and the weak enhanced unlabeled data set prediction value are input into the Top-K guided pseudo label optimization module, the weak enhanced unlabeled data set prediction value is weighted and averaged to obtain an average sample prediction probability distribution, and the average sample prediction probability distribution is specifically as follows:

[0107] ,

[0108] Among them, The average sample prediction probability distribution is represented by and respectively represent the weak enhanced unlabeled data set prediction value corresponding to the weak enhanced unlabeled data set obtained by twice weak enhancement processing respectively;

[0109] According to the average sample prediction probability distribution, if the class confidence corresponding to the maximum probability of the average sample prediction probability distribution is greater than the class adaptive threshold, the corresponding weak enhanced unlabeled data set output feature is determined as a reliable sample and is divided into an unlabeled sample reliable subset;

[0110] The unlabeled sample reliable subset is input into the class center contrast module, and the remaining weak enhanced unlabeled data set output features in the weak enhanced unlabeled data set are subjected to pseudo label optimization;

[0111] According to the Top-K guided algorithm, the pseudo label optimization is performed on the remaining weak enhanced unlabeled data set output features and the strong enhanced unlabeled data set output features, and the Top-K guided algorithm is specifically as follows:

[0112] ,

[0113] Among them, The first K maximum probability class index is represented by The first K class is represented by K The number of filtered classes is represented by The average sample prediction probability distribution is represented by

[0114] The pseudo label optimization includes negative learning optimization and low quality contrast loss optimization;

[0115] The negative learning optimization sets the hard label at the position corresponding to the first K maximum probability class index to 0, and sets the hard label of the class not belonging to the first KThe hard label of the position corresponding to the class index of the maximum probability is set to 1 to obtain a negative class indication mask, the negative class indication mask including a weak enhanced unlabeled data negative class indication mask and a strong enhanced unlabeled data negative class indication mask, and the specific algorithm for obtaining the negative class indication mask is as follows:

[0116] ,

[0117] ,

[0118] ,

[0119] wherein, represents the hard label assignment of the weak enhanced unlabeled data set, represents the hard label assignment of the strong enhanced unlabeled data set, represents the maximum sample prediction probability distribution of the strong enhanced unlabeled data set, represents the class confidence threshold of the class at the moment, represents the negative class indication mask;

[0120] According to the negative consistency loss function, the negative consistency loss function is specifically as follows:

[0121] ,

[0122] wherein, represents the negative consistency loss, represents the fuzzy sample corresponding to the unlabeled data classification, C represents the total number of classes, and c represents the class serial number, represents the prediction value of the strong enhanced unlabeled data set, represents the minimum value for preventing the difference from being 0;

[0123] The low-quality contrast loss optimization forms a positive-negative sample pair with the weak enhanced unlabeled data set output feature and the strong enhanced unlabeled data set output feature, and optimizes according to the low-quality contrast loss function, and the low-quality contrast loss function is specifically as follows:

[0124] ,

[0125] wherein, represents the low-quality contrast loss, represents the unlabeled available sample, i and j respectively represent the serial numbers of the data in different data sets, represents the weak enhanced unlabeled data set output feature corresponding to the weak enhanced unlabeled data set obtained by the two times of weak enhancement processing, This indicates the output features of a strongly augmented unlabeled dataset. express, For temperature coefficient, The function is the cosine similarity function.

[0126] Step S05: Based on the class center comparison module, perform reliable sample partitioning on the output features of the weakly enhanced labeled dataset to obtain a fuzzy subset of labeled samples and a reliable subset of labeled samples. Update the class centers based on the fuzzy subset of labeled samples, the reliable subset of labeled samples, and the reliable subset of unlabeled samples to obtain the class prediction results.

[0127] It should be noted that in this embodiment, the reliable sample partitioning is based on class confidence and class adaptive threshold. The reliable sample partitioning is performed on the output features of the weakly enhanced labeled dataset according to the class center comparison module to obtain a fuzzy subset of labeled samples and a reliable subset of labeled samples. The reliable sample partitioning is based on class confidence and class adaptive threshold, and the updates of the class confidence and class adaptive threshold are as follows:

[0128] ,

[0129] ,

[0130] in, express The category at any time is The average of the reliable sample prediction probabilities, This represents the sum of the number of samples in the reliable subset of labeled samples and the reliable subset of unlabeled samples. yes The category at any time is The probability value corresponding to a reliable sample. express The category at any time is Category confidence threshold, express The category at any time is Category confidence threshold, This represents the momentum used to smooth the update of the class confidence threshold. Indicates a category-adaptive threshold;

[0131] The class center update is performed based on the fuzzy subset of labeled samples, the reliable subset of labeled samples, and the reliable subset of unlabeled samples. The specific algorithm for the class center update is as follows:

[0132] ,

[0133] in, express At this moment The class center corresponding to each category This represents the momentum used for smoothing class center updates. This indicates when to start updating the class center. This represents the number of samples in the reliable subset of labeled samples. This represents the number of samples in the reliable subset of unlabeled samples. i and j These represent the ordinal numbers of data in different datasets. Indicates the first time used for class center update The corresponding features of each labeled sample, Indicates the first time used for class center update Corresponding features of unlabeled samples;

[0134] The class boundary discrimination optimization is performed based on the contrastive learning loss. The specific algorithm for the class boundary discrimination optimization is as follows:

[0135] ,

[0136] ,

[0137] in, This represents the optimization loss for class boundary discrimination. This represents the number of samples in the fuzzy subset of labeled samples. This represents the distance between a labeled, fuzzy sample and the true class label. Representing high-dimensional features, Indicates the first The class center of each category, Indicates the first The class center corresponding to the true class of each sample. Indicates the temperature coefficient. This represents high-dimensional features. and the The class center corresponding to the true class of each sample The distance between them.

[0138] In summary, according to the above-mentioned semi-supervised facial expression recognition method based on self-calibration negative learning, a class center contrast module is designed, the class center, class confidence and adaptive threshold are updated using momentum, the high-confidence samples are stabilized and the intra-class aggregation is driven, and a Top-K guided pseudo label optimization module is designed, the Top-K guided pseudo label optimization is adopted for the low-confidence ambiguous samples, the negative consistency optimization and low-quality contrast loss optimization are applied, and the utilization of ambiguous sample information is strengthened. Specifically, facial expression images are obtained and preprocessed, the preprocessed facial expression images are subjected to data division processing to obtain a labeled data set and an unlabeled data set, the labeled data set and the unlabeled data set are subjected to enhancement processing respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, the enhancement processing includes weak enhancement processing and strong enhancement processing, feature extraction is performed on the weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features, reliable sample division is performed on the weakly enhanced unlabeled data set output features according to the Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset, and pseudo label optimization is performed on the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features according to the Top-K guided algorithm, the pseudo label optimization includes negative learning optimization and low-quality contrast loss optimization, reliable sample division is performed on the weakly enhanced labeled data set output features according to the class center contrast module to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, class center updating is performed according to the labeled sample ambiguous subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result, and the reliable sample division is based on the class confidence and the class adaptive threshold.

[0139] Referring to Figure 2 , a structure schematic diagram of a semi-supervised facial expression recognition system based on self-calibration negative learning according to the second embodiment of the present application is shown, and the system comprises:

[0140] A data division module 10 is configured to obtain facial expression images and perform preprocessing on the facial expression images, and perform data division processing on the preprocessed facial expression images to obtain a labeled data set and an unlabeled data set.

[0141] A data enhancement module 20 is configured to perform enhancement processing on the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, and the enhancement processing includes weak enhancement processing and strong enhancement processing.

[0142] The feature extraction module 30 is used for performing feature extraction on the weakly augmented labeled data set, the weakly augmented unlabeled data set and the strongly augmented unlabeled data set respectively to obtain weakly augmented labeled data set output features, weakly augmented unlabeled data set output features and strongly augmented unlabeled data set output features.

[0143] The Top-K guided pseudo label optimization module 40 is used for performing reliable sample division on the weakly augmented unlabeled data set output features to extract an unlabeled sample reliable subset, and performing pseudo label optimization on the remaining weakly augmented unlabeled data set output features and the strongly augmented unlabeled data set output features according to a Top-K guided algorithm, wherein the pseudo label optimization includes negative learning optimization and low-quality contrast loss optimization.

[0144] The class center contrast module 50 is used for performing reliable sample division on the weakly augmented labeled data set output features to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, performing class center updating according to the labeled sample ambiguous subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result, and the reliable sample division is based on a class confidence and a class adaptive threshold.

[0145] The application further provides a computer storage medium, which stores one or more programs, and the programs are executed by a processor to implement the semi-supervised facial expression recognition method based on self-calibration negative learning.

[0146] The application further provides a computer device, which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory to implement the semi-supervised facial expression recognition method based on self-calibration negative learning.

[0147] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of ordered executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus. For the present specification, the "computer readable medium" can be any device that can contain a storage, communication, propagation or transmission program for use by or in conjunction with the instruction execution system, device or apparatus.

[0148] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: a portable computer diskette (magnetic device); a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); an optical fiber device; and a portable compact disc read-only memory (CDROM). Additionally, the computer readable medium can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0149] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques, which are well known in the art, can be used to implement the application: a hybrid of the techniques mentioned above; a combination of one or more of the techniques mentioned above; or one or more other techniques suitable for use in the computer hardware devices described above.

[0150] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0151] The above-described embodiments are merely some embodiments of the present application, and the description is specific and detailed, but should not be understood as limiting the scope of the patent of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A semi-supervised facial expression recognition method based on self-calibration negative transfer learning, characterized in that, The method comprises the steps of: acquiring facial expression images and pre-processing, and performing data division processing on the pre-processed facial expression images to obtain a labeled data set and an unlabeled data set; performing enhancement processing on the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, wherein the enhancement processing comprises weak enhancement processing and strong enhancement processing; performing feature extraction on the weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set respectively to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features; performing reliable sample division on the weakly enhanced unlabeled data set output features according to a Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset, and then performing pseudo label optimization on the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features according to a Top-K guided algorithm, wherein the pseudo label optimization comprises negative learning optimization and low-quality contrast loss optimization; the step of performing reliable sample division on the weakly enhanced unlabeled data set output features according to the Top-K guided pseudo label optimization module to extract an unlabeled sample reliable subset comprises: inputting the weakly enhanced unlabeled data set output features and the weakly enhanced unlabeled data set prediction value into the Top-K guided pseudo label optimization module, performing weighted average processing on the weakly enhanced unlabeled data set prediction value to obtain an average sample prediction probability distribution, and the average sample prediction probability distribution is specifically as follows: ， wherein, denotes the average sample prediction probability distribution, and denote the weakly augmented unlabeled dataset prediction values corresponding to the weakly augmented unlabeled dataset obtained by the two times of weak augmentation processing, respectively; performing reliable sample division according to the average sample prediction probability distribution, if the class confidence corresponding to the maximum probability of the average sample prediction probability distribution is greater than a class adaptive threshold, then the corresponding weakly enhanced unlabeled data set output features are determined as reliable samples and are divided into the unlabeled sample reliable subset; inputting the unlabeled sample reliable subset into a class center contrast module, and then performing pseudo label optimization on the remaining weakly enhanced unlabeled data set output features in the weakly enhanced unlabeled data set; performing reliable sample division on the weakly enhanced labeled data set output features according to the class center contrast module to obtain a labeled sample fuzzy subset and a labeled sample reliable subset, and performing class center updating according to the labeled sample fuzzy subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result, wherein the reliable sample division is based on class confidence and a class adaptive threshold.

2. The semi-supervised facial expression recognition method based on self- calibrated negative transfer learning according to claim 1, wherein, the step of performing enhancement processing on the labeled data set and the unlabeled data set respectively to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set comprises: acquiring facial expression images and pre-processing, performing face detection and key point alignment processing on the facial expression images, and then performing scaling and cropping to match a standard input size; performing data division processing on the pre-processed facial expression images to obtain a labeled data set and an unlabeled data set, and the labeled data set and the unlabeled data set are specifically as follows: ， wherein, denotes a labeled dataset, denotes an unlabeled dataset, denotes labeled image data, denotes the true label of the labeled image data after one-hot encoding, denotes the total number of labeled image data, denotes unlabeled image data, denotes the total number of unlabeled image data, i and j denote the ordinal number of data in different datasets, respectively; The labeled data set and the unlabeled data set are respectively subjected to enhancement processing, the enhancement processing includes once weak enhancement processing on the labeled data set, twice weak enhancement processing and once strong enhancement processing on the unlabeled data set, to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, and the enhancement processing is specifically as follows: ， ， ， , ， ， wherein, denotes weakly augmented labeled data, denotes weakly augmenting a labeled dataset, denotes a weakly augmented labeled dataset, denotes first weakly augmented unlabeled data, denotes second weakly augmented unlabeled data, denotes weakly augmenting an unlabeled dataset, denotes strongly augmented unlabeled data, denotes strongly augmenting an unlabeled dataset, denotes a weakly augmented unlabeled dataset obtained by each of the two weak augmentation processes, denotes a strongly augmented unlabeled dataset.

3. The semi-supervised facial expression recognition method based on self- calibrated negative transfer learning according to claim 2, characterized in that, The step of respectively performing enhancement processing on the labeled data set and the unlabeled data set further comprises: The weakly enhanced labeled data set, the weakly enhanced unlabeled data set and the strongly enhanced unlabeled data set are respectively subjected to feature extraction to obtain weakly enhanced labeled data set output features, weakly enhanced unlabeled data set output features and strongly enhanced unlabeled data set output features, and the feature extraction is specifically as follows: ， wherein, respectively represent weakly augmented labeled dataset output features and weakly augmented labeled dataset predicted values, respectively represent weakly augmented unlabeled dataset output features and weakly augmented unlabeled dataset predicted values corresponding to the weakly augmented unlabeled dataset obtained by the two times of weak augmentation processing respectively, respectively represent strongly augmented unlabeled dataset output features and strongly augmented unlabeled dataset predicted values.

4. The semi-supervised facial expression recognition method based on self- calibrated negative transfer learning according to claim 1, wherein, The step of performing pseudo-label optimization on the remaining weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features according to the Top-K guided algorithm specifically comprises: The Top-K guided algorithm is specifically as follows: ， wherein, represents the class index of the first K most probable class, represents the number of classes before K filtering, K represents the number of filtered classes, represents the average sample prediction probability distribution.

5. The semi-supervised facial expression recognition method based on self- calibrated negative transfer learning according to claim 4, characterized in that, The step of pseudo-label optimization specifically comprises: Pseudo-label optimization includes negative learning optimization and low-quality contrast loss optimization; The negative learning optimization will... K The hard label at the position corresponding to the category index with the highest probability is set to 0, and categories that do not belong to the previous category are excluded. K The hard label at the position corresponding to the category index with the highest probability is set to 1 to obtain the negative class indicator mask. The negative class indicator mask includes a weakly enhanced unlabeled data negative class indicator mask and a strongly enhanced unlabeled data negative class indicator mask. The specific algorithm for obtaining the negative class indicator mask is as follows: ， ， ， wherein, denotes hard label assignments for weakly augmented unlabelled dataset, denotes hard label assignments for strongly augmented unlabelled dataset, denotes maximum sample prediction probability distribution for strongly augmented unlabelled dataset, denotes class confidence threshold at time instant for class denotes negative class indicator mask; The negative consistency loss function is specifically as follows: ， wherein, represents a negative consistency loss, represents a fuzzy sample corresponding to the no-label data classification, C represents the total number of classes, and c represents the class serial number, represents a strong enhanced no-label data set prediction value, represents a minimum value preventing the difference from being 0; The low-quality contrast loss optimization forms a positive-negative sample pair of the weakly enhanced unlabeled data set output features and the strongly enhanced unlabeled data set output features, and optimizes according to a low-quality contrast loss function, and the low-quality contrast loss function is specifically as follows: ， wherein, represents a low-quality contrast loss, represents a no-label available sample, i and j respectively represent the ordinal of data in different data sets, represents the weakly enhanced no-label data set output feature corresponding to the weakly enhanced no-label data set obtained by the two times of weak enhancement processing respectively, represents the strongly enhanced no-label data set output feature, represents, is a temperature coefficient, the function is a cosine similarity function.

6. The semi-supervised facial expression recognition method based on self- calibrated negative transfer learning according to claim 1, wherein, The step of performing class center updating according to the labeled sample fuzzy subset, the labeled sample reliable subset and the unlabeled sample reliable subset to obtain a class prediction result specifically comprises: The reliable sample division of the weakly enhanced labeled data set output features according to the class center contrast module obtains a labeled sample fuzzy subset and a labeled sample reliable subset, and the reliable sample division is based on a class confidence and a class adaptive threshold, and the update of the class confidence and the class adaptive threshold is as follows: ， ， wherein, denotes the average of the reliable sample prediction probabilities of the class at the time instant t, denotes the sum of the number of samples in the reliable subset of labeled samples and the reliable subset of unlabeled samples, is the probability value corresponding to the reliable sample of the class at the time instant t, denotes the class confidence threshold of the class at the time instant t, denotes the class confidence threshold of the class at the time instant t, denotes the momentum used to smooth the class confidence threshold update, denotes the class adaptive threshold; The class center updating according to the labeled sample fuzzy subset, the labeled sample reliable subset and the unlabeled sample reliable subset is specifically as follows: ， wherein, denotes the class center corresponding to the th class at the moment, denotes the momentum for smoothing the update of the class center, denotes the number of samples in the reliable subset of labeled samples, denotes the number of samples in the reliable subset of unlabeled samples, i and j denote the ordinal of the data in different data sets, respectively, denotes the corresponding feature of the th labeled sample for the update of the class center, denotes the corresponding feature of the th unlabeled sample for the update of the class center. The class boundary discrimination optimization according to the contrast learning loss is specifically as follows: ， ， in, This represents the optimization loss for class boundary discrimination. This represents the number of samples in the fuzzy subset of labeled samples. This represents the distance between a labeled, fuzzy sample and the true class label. Representing high-dimensional features, Indicates the first The class center of each category, Indicates the first The class center corresponding to the true class of each sample. Indicates the temperature coefficient. This represents high-dimensional features. and the The class center corresponding to the true class of each sample The distance between them.

7. A semi-supervised facial expression recognition system based on self- calibrated negative transfer, characterized in that, The class boundary discrimination optimization according to the contrast learning loss is specifically as follows: It comprises: The data division module is used for obtaining and preprocessing facial expression images, and performing data division processing on the preprocessed facial expression images to obtain a labeled data set and an unlabeled data set; The data enhancement module is used for respectively performing enhancement processing on the labeled data set and the unlabeled data set to obtain a weakly enhanced labeled data set, a weakly enhanced unlabeled data set and a strongly enhanced unlabeled data set, and the enhancement processing includes weak enhancement processing and strong enhancement processing; The feature extraction module is configured to perform feature extraction on the weakly augmented labeled dataset, the weakly augmented unlabeled dataset, and the strongly augmented unlabeled dataset respectively to obtain weakly augmented labeled dataset output features, weakly augmented unlabeled dataset output features, and strongly augmented unlabeled dataset output features. The Top-K guided pseudo-label optimization module is configured to perform reliable sample division on the weakly augmented unlabeled dataset output features to extract an unlabeled sample reliable subset, and perform pseudo-label optimization on the remaining weakly augmented unlabeled dataset output features and the strongly augmented unlabeled dataset output features according to a Top-K guided algorithm, wherein the pseudo-label optimization includes negative learning optimization and low-quality contrast loss optimization. The Top-K guided pseudo-label optimization module performs reliable sample division on the weakly augmented unlabeled dataset output features to extract an unlabeled sample reliable subset, and the step specifically includes: The weakly augmented unlabeled dataset output features and weakly augmented unlabeled dataset predicted values are input into the Top-K guided pseudo-label optimization module, the weakly augmented unlabeled dataset predicted values are weighted and averaged to obtain an average sample prediction probability distribution, and the average sample prediction probability distribution is specifically as follows: ， wherein, denotes the average sample prediction probability distribution, and denote the weakly augmented unlabeled dataset prediction values corresponding to the weakly augmented unlabeled dataset obtained by the two times of weak augmentation processing, respectively; According to the average sample prediction probability distribution, if the class confidence corresponding to the maximum probability of the average sample prediction probability distribution is greater than a class adaptive threshold, the corresponding weakly augmented unlabeled dataset output features are determined as reliable samples and are divided into the unlabeled sample reliable subset; The unlabeled sample reliable subset is input into the class center contrast module, and the remaining weakly augmented unlabeled dataset output features in the weakly augmented unlabeled dataset are subjected to pseudo-label optimization. The class center contrast module is configured to perform reliable sample division on the weakly augmented labeled dataset output features to obtain a labeled sample ambiguous subset and a labeled sample reliable subset, and perform class center updating according to the labeled sample ambiguous subset, the labeled sample reliable subset, and the unlabeled sample reliable subset to obtain a class prediction result, wherein the reliable sample division is based on a class confidence and a class adaptive threshold.

8. A storage medium, characterized by The storage medium stores one or more programs, which are executed by the processor to implement the semi-supervised facial expression recognition method based on self-calibration negative learning according to any one of claims 1-6.

9. A computer device, comprising: The computer device includes a memory and a processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program stored in the memory to implement the semi-supervised facial expression recognition method based on self-calibration negative learning according to any one of claims 1-6.

Images