Self-paced co-learning based small sample facial expression recognition method and medium

Abstract

The application discloses a small sample face expression recognition method based on a self-step cooperative learning framework and a medium, comprising the following steps: acquiring basic class expression data sets and composite class expression data sets; establishing a self-step cooperative learning network, including a mixed expert convolution layer and a classification model; scaling the channel number of different mixed expert convolution layers; performing expression feature embedding learning on the mixed expert convolution layer in the Euclidean space; constructing a teacher model and a student model based on the expert model to obtain corresponding expression features; mapping the expression features obtained by the teacher model and the student model from the Euclidean space to the hyperbolic space to perform expression feature embedding learning on the self-step cooperative learning network; classifying the expression features by using the classification model to complete the final training; inputting the composite class expression data sets into the trained self-step cooperative learning network to perform expression classification calculation and output a classification result to recognize different expressions. The application effectively improves the expression recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a small sample facial expression recognition method based on a self-paced co-learning framework and a medium. BACKGROUND

[0002] Facial expression, as one of the most natural and effective ways for humans to express emotions, plays an important role in interpersonal communication. In the past few decades, facial expression recognition has attracted extensive attention in the fields of multimedia and computer vision due to its wide range of applications, such as human-computer interaction, psychological assessment, interactive entertainment, online education, and many other practical scenarios.

[0003] According to the research of famous psychologists Ekman et al. (Ekman P, Friesen W V. Constants across cultures in the face and emotion[J]. Journal of Personality and Social Psychology, 1971, 17(2): 124-129.), facial expressions can be divided into seven basic expression categories, namely, anger, surprise, disgust, fear, happiness, sadness, and neutral. Based on this, researchers have collected a series of basic expression datasets and carried out research on automated facial expression recognition. Early expression recognition methods were mainly based on texture, geometry, and other handcrafted features, which performed well on controllable indoor expression datasets but not ideal on uncontrolled real-world datasets. With the development of deep learning, neural network-based feature extraction methods have become mainstream, and the focus of expression recognition algorithms has gradually shifted from simple indoor datasets to more challenging outdoor datasets.

[0004] However, these basic expressions cannot fully capture the diversity of human emotions in real-world scenarios. To cover more human emotions, Du et al. (Du, S.; Tao, Y.; and Martinez, A. M. 2014. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15): E1454-E1462.) proposed compound expressions by combining basic expression classes and extended the expression classes to 22 classes in 2014. Other scholars collected fine-grained expression datasets based on the wheel of emotion theory and hierarchical model, further extending the classes of expression datasets. Although the basic expression recognition problem has received a lot of attention, the research on the above non-basic expression classes is relatively less. Khadija Slimani et al. (Khadija Slimani, Khadija Lekdioui, Rochdi Messoussi, and Raja Touahni. 2019. Compound facial expression recognition based on highway CNN. In Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society. 1-7.) proposed a highway convolutional network to replace the short connection line of the residual network with learnable parameters to complete the compound expression recognition task. Jianzhu Guo et al. (Jianzhu Guo, Shuai Zhou, Jinlin Wu, Jun Wan, Xiangyu Zhu, Zhen Lei, and Stan Z Li. 2017. Multi-modality network with visual and geometrical information for micro emotion recognition. In IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 814-819.) proposed a multi-modal network that simultaneously uses visual and geometric features to recognize compound expressions. However, these methods need to rely on a large amount of labeled compound expression dataset.Compared to basic class expression data, the annotation cost of composite expressions is greater, and usually requires the professional guidance of psychologists. How to use a small amount of labeled samples to recognize new composite expressions is an important problem. Therefore, small sample learning emerges as a promising learning scheme. Recently, Zou et al. (Xinyi Zou, Yan Yan, Jing-Hao Xue, Si Chen, and Hanzi Wang. 2022. When facial expression recognition meets few-shot learning: A joint and alternate learning framework. In AAAI. 5367-5375.) first proposed the cross-domain few-shot facial expression recognition task, which uses a model trained on multiple basic expression datasets (source domain) to recognize new composite expressions (target domain) involving only a limited number of reference images, which greatly reduces the cost of expensive annotation. Subsequently, Zou et al. (Zou, X.; Yan, Y.; Xue, J.-H.; Chen, S.; and Wang, H. 2022. Learn-to-decompose: Cascaded decomposition network for cross-domain few-shot facial expression recognition. In ECCV, 683-700.) proposed a cascaded decomposition network to learn a transferable feature space to improve the model's generalization ability to composite facial expressions.

[0005] For us humans, based on the cognition of the emotions we have seen, we only need a small amount of observation to judge a new unknown emotion. This learning paradigm based on a small amount of labeled samples of new classes, quickly generalizing to new classes, is called small sample learning in the field of computers. Although small sample learning has been widely studied in image recognition, target detection and other fields, there is little exploration in the field of expression recognition. The present invention aims to solve the problem of classifying new composite expressions under the small sample learning paradigm, in order to alleviate the pressure on existing composite expression classification models to pre-collect a large amount of labeled composite expression data. It is worth noting that, compared to dividing a composite expression dataset into base classes (for training) and new classes (for testing), the present invention uses a more challenging but practical scenario, namely the cross-domain few-shot setting, in which we train the model on multiple basic expression datasets that are easily available and test it on new composite expression datasets.

[0006] Existing methods utilize multiple datasets to increase the diversity of source domain and learn a transferable space. Although these methods have achieved certain effects, they do not explicitly consider the significant imbalance of expression classes in these datasets (the number of samples of different classes is inconsistent). Therefore, training a single model on multiple datasets is prone to focus on learning expression classes involving more samples, resulting in biased learning. Secondly, facial expressions have rich hierarchical relationships. This hierarchical structure information can significantly enhance the model's understanding of facial expressions, thereby improving the transferability (the ability to apply previously learned knowledge to new tasks) from seen basic expressions to unseen compound expressions according to the inherent relationship between basic and compound expressions. Existing methods mainly focus on learning feature embedding in Euclidean space. This way fails to effectively capture the inherent hierarchical relationship of facial expressions, reducing the transferability of the model. At the same time, these methods do not consider the distinction between simple and difficult samples. Therefore, the generalization performance of existing methods on unseen target domains is still not ideal.

[0007] In order to solve the above problems, it is urgent to propose a small sample facial expression recognition method based on a self-step collaborative learning framework and a medium. SUMMARY

[0008] Therefore, the purpose of the present application is to provide a small sample facial expression recognition method based on a self-step collaborative learning framework, which effectively improves the expression recognition accuracy.

[0009] In order to achieve the above technical purpose, the technical scheme adopted by the present application is:

[0010] The present application provides a small sample facial expression recognition method based on a self-step collaborative learning framework, comprising the following steps:

[0011] Step 1, obtaining basic class expression dataset and compound class expression dataset, and performing data preprocessing;

[0012] Step 2, establishing a self-step collaborative learning network, wherein the self-step collaborative learning network comprises a plurality of mixed expert convolutional layers and a classification model;

[0013] Step 3, scaling the number of channels of different mixed expert convolutional layers to change the size of the self-step collaborative learning network;

[0014] Step 4, the scaled mixed expert convolutional layer learns expression feature embedding in Euclidean space through the basic class expression dataset, to obtain an expert model;

[0015] Step 5, obtain the expression features corresponding to the teacher model and the student model through the teacher model and the student model constructed based on the expert model; and design a teacher-student correction loss to measure the difference between the expression features of the teacher model and the student model, so as to strengthen the constraint and guiding role of the teacher model on the student model;

[0016] Step 6, map the expression features obtained by the teacher model and the student model from the Euclidean space to the hyperbolic space, and perform expression feature embedding learning on the self-teacher collaborative learning network;

[0017] Step 7, classify the expression features using a classification model to complete the final training;

[0018] Step 8, input the composite class expression dataset into the trained self-teacher collaborative learning network to perform expression classification calculation, output the classification result, and identify different expressions.

[0019] Further, the step 1 specifically comprises:

[0020] Step 11, obtain a basic class expression dataset for training, including three indoor datasets and two outdoor datasets, wherein one of the outdoor datasets is obtained by downloading from the Internet, and the other outdoor dataset contains video clips extracted from movies; the basic class expression dataset carries basic expression labels, including anger, surprise, disgust, fear, happiness, sadness and neutrality, and one of the indoor datasets has an additional contempt expression label;

[0021] Step 12, obtain a composite class expression dataset for testing, including one indoor dataset and one outdoor dataset, the composite class expression dataset carries detailed expression labels;

[0022] Step 13, perform face alignment and cropping on all basic class expression datasets and composite class expression datasets as inputs of the self-teacher collaborative learning network;

[0023] Step 14, during training, perform data augmentation on the basic class expression dataset by random flipping and color jittering.

[0024] Further, the step 2 specifically comprises:

[0025] Step 21, establish a self-teacher collaborative learning network for expression recognition, the self-teacher collaborative learning network includes a plurality of hybrid expert convolutional layers for extracting expression features and a classification model for expression category classification;

[0026] Step 22: Each of the hybrid expert convolutional layers includes a parameter-sharing router and a convolutional module derived from a backbone convolutional neural network. The router includes a preference evaluation network for evaluating preference scores for experts and an expert selection network for selecting a subset of channel features from an auto-cooperative learning network based on the preference scores.

[0027] Step 23: The classification model uses a metric to calculate distance and classifies the facial expression image into the facial expression category with the smallest distance.

[0028] Furthermore, step 3 specifically includes:

[0029] Step 31: For each hybrid expert convolutional layer, the hybrid expert convolutional layer represents a convolutional layer containing multiple experts. Each hybrid expert convolutional layer is defined to contain E experts, each expert being responsible for learning facial expression features from a set of facial expression categories.

[0030] Step 32: Assume that the first hybrid expert convolutional layer in the self-synchronizing collaborative learning network has... The number of channels in the first hybrid expert convolutional layer in the self-synchronizing collaborative learning network is increased by scaling the number of channels using a scaling factor r, i.e., changing the number of convolutional kernels. Updated to Each expert is selected in the first hybrid expert convolutional layer. One channel;

[0031] Step 33: Similarly, reduce the number of channels in the i-th hybrid expert convolutional layer of the self-synchronizing collaborative learning network from... Updated to .

[0032] Furthermore, step 4 specifically includes:

[0033] Step 41: In each iteration, sample batches of data from the basic facial expression dataset. , Represents facial expressions. This indicates the expression category label corresponding to the expression image; for One of the samples First, the sample Through a collection A basic feature is extracted after preprocessing the base convolutional layers, ReLU activation function, and average pooling operation preprocessing block. Then the basic feature The preference evaluation network of the router, which is input into the first hybrid expert convolutional layer, calculates the preference scores corresponding to E experts. , the expert selection network of the router in the first mixed-expert convolutional layer selects an expert z according to the preference score The expert z with the highest score is selected.

[0034] Step 42, the expert selection network generates a mask by the selected expert z , which is used to select the channel subset corresponding to the expert z; when selecting channels, the selected channels are set to 1 in the mask, otherwise 0.

[0035] Step 43, apply the mask to extract the channel features of the expert z and take it as the input feature of the second mixed-expert convolutional layer; similarly, for each subsequent mixed-expert convolutional layer of the self-step co-learning network, the channel features from the last mixed-expert convolutional layer are sent to the router to select the optimal expert z; these selected channels form an end-to-end path, which builds an expert model suitable for the sample , and the final expression feature extracted by the expert model is represented as , where e is used to distinguish from other features and is used for expression classification.

[0036] Step 44, use the cross-entropy loss based on the preference score to constrain the experts selected by the router; first, define a function to establish the correspondence between the true label of the training sample and the expert index, that is:

[0037]

[0038] wherein, represents the true label of the training sample , and represents the expert index related to the input expression image; represents the number of expression categories of the basic category expression dataset;

[0039] The first preference loss is defined as:

[0040]

[0041] wherein, M' represents the total number of mixed-expert convolutional layers, m represents the mth convolutional layer of the current mixed-expert convolutional layer, represents the cross-entropy loss function.

[0042] Step 45, when selecting experts, the router allows the router to select experts that do not correspond to the expression category of the current sample based on the preference score using the flooding strategy; the second preference loss is defined as:

[0043]

[0044] wherein, represents a flooding level for controlling the loss fluctuation range, .

[0045] Further, the step 5 specifically comprises:

[0046] Step 51, based on the expert model, a teacher model and a student model are constructed, the knowledge of the teacher model is migrated to the student model by using a self-distillation network architecture, and the corresponding expression features are extracted using the teacher model and the student model;

[0047] Step 52, for the i-th sample , the student model is used for feature extraction to obtain the expression feature ;

[0048] Step 53, for the i-th sample, the teacher model is used for feature extraction to obtain the expression feature ;

[0049] Step 54, the teacher model updates the parameters in the form of exponential moving average of the student model;

[0050] Step 55, a teacher-student correction loss is introduced, the difference between the expression features of the teacher model and the student model is measured by using the KL divergence, so as to strengthen the constraint and guiding effect of the teacher model on the student model; the mathematical expression of the teacher-student correction loss is as follows:

[0051]

[0052] wherein, represents a teacher-student correction loss function, represents a KL divergence, represents a feature distribution of the student model on the i-th sample , represents a feature distribution of the teacher model on the i-th sample , and U represents the number of samples, represents a temperature coefficient, which is used to adjust the model output distribution, and under the guidance of the teacher model, the student model is continuously corrected and optimized.

[0053] Further, the step 6 specifically comprises:

[0054] Step 61, in the Euclidean space, for the input sample , the expression feature representations and are obtained using the teacher model and the student model respectively.Then, the learned expression features in Euclidean space are mapped to hyperbolic space, which is mathematically represented as follows:

[0055]

[0056]

[0057] where o represents the center of the Poincare ball, and c represents the curvature of the hyperbolic space. represents the standard Euclidean norm; is the hyperbolic tangent function; and and respectively represent the expression features in hyperbolic space, and respectively represent the expression features obtained after the sample is processed by the teacher model and the student model, represents the exponential mapping function;

[0058] Step 62, minimize the distance between the embeddings in hyperbolic space, which is represented as follows:

[0059]

[0060] where represents the distance between and in hyperbolic space, and respectively represent the radii of and in the Poincare ball, represents the inverse hyperbolic cosine function;

[0061] Step 63, based on the hyperbolic uncertainty in hyperbolic space:

[0062]

[0063] The random Riemannian gradient descent is used to minimize the Poincare distance between and , which is mathematically represented as follows:

[0064]

[0065]

[0066] This learning process is self-paced, and the gradient changes according to the hyperbolic uncertainty from the teacher model, that is, the larger the radius of , the easier it is to learn, and the gradient Also stronger.

[0067] Further, the step 7 is specifically:

[0068] Each iteration samples a batch of data from the basic class expression dataset , for one of the samples , input it into the expert model and the classification model to get the final expression feature and the prediction result ; each query image is assigned to the nearest support class center in the learned feature space, and the prediction result The cross-entropy loss of the true label is used to update the network parameters:

[0069]

[0070]

[0071] Where N is the number of basic class expressions, the indicator function 1 [n=yq] is 1 when and only when , and are the expression feature embedding and the corresponding expression category label of the query expression image, respectively; represents the center of expression category n, K represents the total number of expression images, and k represents the kth expression image, represents the expression feature of the kth expression image in the support set of expression category n.

[0072] Further, the step 8 specifically includes:

[0073] Step 81, using the finally trained self-step collaborative learning network, sampling a small sample classification task similar to the training process on the composite expression dataset for testing, that is, randomly sampling N classes, sampling K samples as the support set and Q samples as the query set for each class, calculating the classification accuracy of all N*Q samples in the query set as the precision evaluation of the current task;

[0074] Step 82, in the testing process, using the self-step collaborative learning network trained on the basic class expression, inputting the sampled support set and query set samples into the student model containing multiple expert models to extract features, then calculating the distance between each sample in the query set and each expression class in the support set by the measurement standard, and assigning the samples in the query set to the expression category of the support set in the feature space. Adjacent;

[0075] ​Step 83, define the number of randomly sampled expression categories N, and evaluate the performance of the self-step collaborative learning network under different K values, and each small sample classification task takes N*Q query samples for classification evaluation;

[0076] Step 84, take the average of the accuracy of multiple small sample classification tasks to evaluate the final performance;

[0077] Step 85, identify different expressions according to the classification results.

[0078] The application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the small sample facial expression recognition method based on the self-step collaborative learning framework as described above.

[0079] By adopting the technical scheme, the application has the beneficial effects compared with the prior art:

[0080] Based on the designed self-step collaborative learning network, the application learns multiple expert models (including multiple mixed expert convolutional layers), captures the diversity of expressions from multiple data sets, which effectively reduces the influence of domain differences in cross-domain small sample facial expression recognition, and effectively alleviates the bias learning problem caused by unbalanced expression categories. The hyperbolic self-step learning strategy is introduced to project the features from the Euclidean space to the hyperbolic space. In the hyperbolic space, the model is trained from simple to difficult samples according to the difficulty of the samples, which better models the hierarchical relationship between expressions, learns a transferable feature space, and captures the hierarchical relationship between facial expressions to assist model updating. The hyperbolic self-step learning strategy is used to assist the learning of the multiple expert models, and the flexible and high-precision new class expression recognition task is completed. Only a small amount of labeled new class expression data is needed to quickly generalize and perform unknown new class expression recognition tasks. The application not only achieves good recognition performance on multiple public data sets, but also is a more flexible and practical solution compared with traditional supervised learning, which is used to solve the small sample facial expression recognition problem. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0082] Figure 1 It is a kind of small sample facial expression recognition method based on self-step collaborative learning framework provided by the embodiment of the application.

[0083] Figure 2 is a whole network framework diagram provided by an embodiment of the present application.

[0084] Figure 3 is a schematic diagram of a computer readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0085] The present application will be further described below in conjunction with the drawings and embodiments. It is particularly pointed out that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only part of the embodiments of the present application, not all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the present application.

[0086] Please refer to Figure 1 and Figure 2 A small sample face expression recognition method based on a self-paced co-learning framework is provided by the present application. The self-paced co-learning framework is a training strategy used to train and optimize the model, including the following steps:

[0087] Step 1, obtaining basic class expression dataset and composite class expression dataset, and performing data preprocessing;

[0088] In this embodiment, the step 1 specifically includes:

[0089] Step 11, obtaining a basic class expression dataset for training, including three indoor datasets (such as CK+, MMI, OULU) and two outdoor datasets (such as RAF, SFEW). The indoor dataset is obtained by shooting in an indoor environment, and has controllable external factors such as light and posture. The outdoor dataset refers to a dataset collected under natural conditions, and has uncontrollable external factors. One of the outdoor datasets (RAF) is obtained by downloading from the Internet, and the other outdoor dataset (SFEW) contains video clips extracted from movies; the basic class expression dataset carries basic expression annotations, including anger, surprise, disgust, fear, happiness, sadness and neutrality, and one of the indoor datasets (CK+) has additional contempt expression annotations;

[0090] Step 12, obtaining a composite class expression dataset for testing, including an indoor dataset (CFEE) and an outdoor dataset (EmotioNet), the composite class expression dataset carries detailed expression annotations; has more detailed expression information annotations than the basic class expression,

[0091] Step 13, all basic class expression data sets and composite class expression data sets are subjected to face alignment and cutting (cutting to 224*224) by MTCNN algorithm (a deep learning algorithm for face detection and alignment) as input of the self-step collaborative learning network;

[0092] Step 14, during training, the basic class expression data set is subjected to data enhancement by random flipping and color jittering (changes in brightness, contrast, saturation and hue) to enrich the diversity of training data.

[0093] Step 2, a self-step collaborative learning network is established, which includes a plurality of mixed expert convolution layers and a classification model; the mixed expert convolution layer represents a special convolution layer containing a plurality of experts, where the expert refers to a self-defined small network;

[0094] In this embodiment, the step 2 specifically includes:

[0095] Step 21, a self-step collaborative learning network for expression recognition is established, which includes a plurality of mixed expert convolution layers for extracting expression features and a classification model for expression category classification; the mixed expert convolution layer is used to extract the facial expression features in the expression image, and then the classification model is used to classify the obtained facial expression features. For example, for a face picture containing an unknown expression (assuming that the label of happy is 0 and the label of sad is 1), the feature f is obtained through the self-step collaborative learning network, and then it is input into the classification model for classification. If the classification result is 0, it means that the expression in the picture is happy; if the classification result is 1, it means sad;

[0096] Step 22, each mixed expert convolution layer includes a parameter-shared router and a convolution module derived based on a backbone convolution neural network (ResNet-12 is used, which is a widely used deep convolution neural network for image classification), the router includes a preference evaluation network for evaluating preference scores for experts and an expert selection network for selecting a part of channel features from the self-step collaborative learning network according to the preference scores; through a plurality of mixed expert convolution layers, diversified expression feature patterns in different data sets are captured, so that the self-step collaborative learning network can obtain a better decision boundary to alleviate the biased learning problem in multiple data sets (due to the different number of samples in different categories in the data set, the model performs better on the category with more data and worse on the category with less data), and complete small sample face expression recognition;

[0097] Step 23: The classification model uses a metric to calculate distances and classifies facial expression features into the expression category with the smallest distance. Specific metrics are used to evaluate the similarity or distance between data points, thereby performing classification. Common metrics include Euclidean distance, Manhattan distance, and cosine similarity.

[0098] Step 3: Scale the number of channels in different hybrid expert convolutional layers to change the size of the autosynchronous collaborative learning network;

[0099] In this embodiment, step 3 specifically includes:

[0100] Step 31: For each hybrid expert convolutional layer, the hybrid expert convolutional layer represents a convolutional layer containing multiple experts. Each hybrid expert convolutional layer is defined to contain E experts, each expert being responsible for learning facial expression features from a set of facial expression categories.

[0101] Step 32: Assume that the first hybrid expert convolutional layer in the self-synchronizing collaborative learning network has... The number of channels in the first hybrid expert convolutional layer in the self-synchronizing collaborative learning network is increased by scaling the number of channels using a scaling factor r, i.e., changing the number of convolutional kernels. Updated to Each expert is selected in the first hybrid expert convolutional layer. One channel;

[0102] Step 31: Similarly, reduce the number of channels in the i-th hybrid expert convolutional layer of the self-synchronizing collaborative learning network from... Updated to .

[0103] Step 4: The scaled hybrid expert convolutional layer learns facial expression feature embedding in Euclidean space using a basic class facial expression dataset to obtain an expert model; feature embedding refers to mapping high-dimensional, sparse, discrete or symbolic features to a low-dimensional, dense vector space for representation.

[0104] In this embodiment, step 4 specifically includes:

[0105] Step 41: In each iteration, sample batches of data from the basic facial expression dataset. , Represents facial expressions. This indicates the expression category label corresponding to the expression image; for One of the samples First, the sample Through a collection A basic feature is extracted after preprocessing the base convolutional layers (each kernel is 2 pixels high and 2 pixels wide), ReLU activation function, and average pooling operation. Then the basic feature The preference evaluation network of the router, which is input into the first hybrid expert convolutional layer, calculates the preference scores corresponding to E experts. The expert selection network of routers in the first hybrid expert convolutional layer is based on preference scores. Select the expert z with the highest score;

[0106] Step 42: The expert selection network generates a mask using the selected expert z. This is used to select the subset of channels corresponding to expert z; when selecting channels, the elements of the selected channels in the mask are set to 1, otherwise to 0;

[0107] Step 43: Apply a mask To extract the channel features of expert z This is then used as the input feature for the second hybrid expert convolutional layer; similarly, for each subsequent hybrid expert convolutional layer of the self-synchronizing co-learning network, the channel features from the previous hybrid expert convolutional layer are fed into the router to select the optimal expert z; these selected channels form an end-to-end path, constructing a suitable sample... The expert model, whose final expression features are represented as... The 'e' symbol is used to distinguish it from other features and for facial expression classification;

[0108] Step 44: Utilize preference-based scores The cross-entropy loss is used to constrain the expert in router selection; first, a function is defined to build the training samples. The correspondence between real labels and expert indexes, namely:

[0109]

[0110] in, Indicates training samples The true label, Represents the expert index associated with the input facial expression image; This represents the number of expression categories in the basic expression dataset;

[0111] First preference loss Defined as:

[0112]

[0113] wherein M' represents the total number of mixed expert convolution layers, m represents the mth layer of the current mixed expert convolution layer, i.e., the mth layer of the mixed expert convolution layer, represents the cross-entropy loss function; in order to optimize the router, a function is defined to establish the correspondence between the facial expression class label and the expert index, and it is ensured that each expert can process a subset of classes with similar sample quantities; in the training data, the facial expression class labels of different classes are represented by numbers, such as the labels of happy and sad are 0 and 1 respectively. The index of the E experts is also represented by numbers: 0, 1, …, E. Establishing the correspondence between the label and the index means that the expression of a certain class is processed by the expert with the specified index, such as happy 0 and expert 0 establish a relationship, which means that the expression of the happy class is processed by expert 0.

[0114] Step 45, when selecting an expert, the router uses a flooding strategy based on the preference score to allow the router to select an expert that does not correspond to the expression class of the current sample; a second preference loss is defined as:

[0115]

[0116] wherein, represents the flooding level for controlling the fluctuation range of the loss, In order to improve the generalization performance of the expert model on complex expressions, a flooding strategy is introduced (flooding strategy: a method for improving the generalization performance of the model by limiting the minimum value of the training loss), so that the router has a certain error boundary when selecting an expert (the router can select an expert that does not correspond to the current sample class based on the preference score). Therefore, each expert can not only learn the expressions of fixed classes, but also learn information from other expression classes. This helps better generalize from basic class expressions to unseen complex expressions.

[0117] Step 5, obtain the expression features corresponding to the teacher model and the student model through the teacher model and the student model constructed based on the expert model; and design a teacher-student correction loss to measure the difference between the expression features of the teacher model and the student model, thereby strengthening the constraint and guiding role of the teacher model on the student model;

[0118] In this embodiment, the step 5 specifically comprises:

[0119] Step 51, based on the expert model, a teacher model and a student model are constructed, the knowledge of the teacher model is migrated to the student model using a self-distillation network architecture, the teacher model provides guidance and constraint for the update of the student model, and the corresponding expression features are extracted using the teacher model and the student model;

[0120] Step 52: For the i-th sample Feature extraction was performed using a student model to obtain facial expression features. ;

[0121] Step 53: For the i-th sample, use the teacher model to extract features and obtain facial expression features. ;

[0122] Step 54: The teacher model updates its parameters using the student model's Exponential Moving Average (EMA). EMA is a weighted moving average method that assigns higher weights to recent data when calculating the average. It is widely used in time series data analysis, financial technical analysis, and parameter smoothing in machine learning. EMA's characteristics include: decreasing weights (higher weights for recent data points and lower weights for earlier data points, making it more sensitive to recent changes); significant smoothing effect (due to the greater weight allocated to recent data points, EMA reacts to data changes more quickly than Simple Moving Average (SMA) while still smoothing short-term fluctuations); and recursive calculation (the current EMA value depends on all previous data points, but the weights decrease exponentially over time).

[0123] Step 55: Introduce teacher-student correction loss. KL divergence is used to measure the difference in facial expression features between the teacher and student models, thereby strengthening the teacher model's constraint and guidance effect on the student model. The mathematical representation of the teacher-student correction loss is as follows:

[0124]

[0125] in, This represents the teacher-student correction loss function. Denotes KL divergence, This indicates that the student model is in the i-th sample. Feature distribution on, This indicates that the teacher model is in the i-th sample. The characteristic distribution on the , where U represents the sample size, This represents the temperature coefficient, used to adjust the model output distribution. Under the guidance of the teacher's model, the student model is continuously corrected and optimized.

[0126] The teacher model can be updated using the exponential moving average of the student model, which can be expressed by the following formula: ,in, It is a controllable weight; This represents the current facial expression features of the teacher model. The expression feature learned by the student model in all previous iterations is represented; in order to make the model learning process more stable, the knowledge of the teacher model is migrated to the student model by using a self-distillation network architecture to provide guidance and constraints for the update of the student model.

[0127] The expression features obtained by the teacher model and the student model are mapped from the Euclidean space to the hyperbolic space, and expression feature embedding learning is performed on the self-step collaborative learning network; in order to learn the rich hierarchical relationships between facial expressions (there are the same movement units between different facial expressions, for example, happy and surprised will have the movement unit of mouth corner up) and make the model adaptively train from easy to difficult (first train simple samples, and then train difficult samples), a hyperbolic self-step learning strategy is designed to optimize the proposed self-step collaborative learning network, and the hyperbolic space suitable for modeling hierarchical relationships is used to learn expression feature embedding (in the hyperbolic space, the features of the expression are learned, and these features contain the hierarchical relationships between expressions: multi-level and multi-dimensional differences between different expressions), so as to improve the generalization ability of the model to new class expressions;

[0128] In the embodiment, the step 6 specifically includes:

[0129] Step 61, in the Euclidean space, for the input sample , the expression feature representations and are obtained by using the teacher model and the student model respectively; then, the expression features learned in the Euclidean space are mapped to the hyperbolic space, and the hierarchical relationships between the models are promoted, which are mathematically represented as follows:

[0130]

[0131]

[0132] Wherein, o represents the center of the Poincaré ball, and c represents the curvature of the hyperbolic space; represents the standard Euclidean norm; is the hyperbolic tangent function; are and respectively, and and respectively represent the expression features obtained by the sample after the teacher model and the student model, represents the exponential mapping function;

[0133] Step 62, the distance between the embeddings in the hyperbolic space is minimized, which is represented as follows:

[0134]

[0135] in, In hyperbolic space and The distance between them and They represent the contents of the Poincaré ball. and radius, Represents the inverse hyperbolic cosine function;

[0136] Step 63: In order to optimize the model parameters, based on the hyperbolic uncertainty inherent in hyperbolic space... :

[0137]

[0138] Stochastic Riemann gradient descent is used to minimize and The Poincaré distance between them can be mathematically represented as follows:

[0139]

[0140]

[0141] The learning process is self-stepping, with gradients based on the hyperbolic uncertainty from the teacher model. Change, that is radius The larger, The easier it is to learn, the more gradients there are. The stronger the hyperbolic space, the better. This approach allows for training strategies that progress from simple to difficult samples. Furthermore, unlike Euclidean space, hyperbolic space naturally embeds hierarchical structures. Therefore, during optimization in hyperbolic space, the hierarchical relationships of facial expression features are implicitly captured by the multi-expert collaborative learning network, enabling the router to better understand facial expression features, calculate more accurate preference scores, and enhance the optimization of the expert model. This results in a better transferable feature space (feature spaces learned on training data can be well applied to other unseen datasets). Moreover, the hyperbolic auto-learning strategy adaptively assigns larger gradient changes (to accelerate model optimization) to simple samples and smaller gradient changes to difficult samples, without introducing additional computational costs like previous auto-learning methods.

[0142] Step 7: Use a classification model to classify facial expression features to complete the final training; use the traditional batch training method for the auto-cooperative learning network to learn the classification model of all basic facial expressions.

[0143] In this embodiment, step 7 specifically includes:

[0144] Batch data is sampled from the basic class expression dataset for each iteration , for one of the samples , input it into the expert model and the classification model to get the final expression feature and the predicted result ; each query image is assigned to the nearest support class center in the learned feature space, and the predicted result is calculated The cross-entropy loss of the real label is used to update the network parameters:

[0145]

[0146]

[0147] where N is the number of basic class expressions, and the indicator function 1 [n=yq] is 1 if and only if , and and are the expression feature embedding and the corresponding expression class label of the query expression image, respectively; is the center of expression class n, K represents the total number of expression images, and k represents the kth expression image, is the expression feature of the kth expression image of expression class n in the support set.

[0148] Step 8, input the composite class expression dataset into the trained self-step co-learning network to perform expression classification calculation, output the classification result, and identify different expressions;

[0149] In this embodiment, step 8 specifically comprises:

[0150] Step 81, using the finally trained self-step co-learning network, sample a small sample classification task (meta-task) similar to the training process on the composite expression dataset for testing, that is, randomly sample N classes, sample K samples as the support set (that is, construct an N-way K-shot task) for each class, and sample Q samples as the query set (for example, a 5-way 5-shot task represents 5 classes, each class has 5 labeled pictures, a total of 25 pictures. Labeled means that the number 0, 1, …, n, that is, it is known that the expression contained in the picture is happy or sad. The support set refers to the dataset used to train the model, and the query set is used for testing), based on the trained self-step co-learning network, calculate the classification accuracy of all N*Q samples of the query set as the precision of the current task;

[0151] ​Step 82, during the test process, the self-cooperative learning network trained on the basic class expression is used, the samples of the support set and the query set are respectively sent into the student model containing multiple expert models to extract features, then the distance between each sample in the query set and each expression class in the support set is calculated through the measurement standard, and the samples in the query set are assigned to the expression class of the support set in the feature space which is most adjacent to them;

[0152] Step 83, the number of randomly sampled expression classes N is defined, and the performance of the self-cooperative learning network is evaluated under different K values, and each small sample classification task takes N*Q query samples for classification evaluation;

[0153] Step 84, the mean of the accuracy rates of multiple small sample classification tasks is taken to evaluate the final performance;

[0154] Step 85, different expressions are identified according to the classification results.

[0155] For example, define the randomly sampled class N=5, respectively evaluate the performance of the self-cooperative learning network when K=1 and 5, each small sample classification task takes Q=16, that is, a total of 5*16=80 query samples for classification, and the mean of the accuracy rates of 1000 small sample classification tasks is taken to evaluate the performance.

[0156] Table 1

[0157]

[0158] Table 1 is the accuracy rate comparison of the present application and other small sample methods on the CFEE, EmotioNet test data set.

[0159] ProtoNet corresponds to the method proposed by Jake Snell et al. (Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. In NeurIPS. 4077-4087.);

[0160] MatchingNet corresponds to the method proposed by Oriol Vinyals et al. (Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In NeurIPS. 3630-3638.);

[0161] RelationNet corresponds to the method proposed by Flood Sung et al. (Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In CVPR. 1199-1208.);

[0162] GNN corresponds to the method proposed by Victor Garcia et al. (Victor Garcia and Joan Bruna. 2018. Few-shot learning with graph neural networks. In ICLR. 1-13.);

[0163] InfoPatch corresponds to the method proposed by Liu, C. et al. (Liu C, Fu Y, Xu C, et al. 2021. Learning a few-shot embedding model with contrastive Learning. In AAAI. 8635-8643.);

[0164] DSN corresponds to the method proposed by Christian Simon et al. (Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. 2020. Adaptive subspaces for few-shot learning. In CVPR. 4136-4145.);

[0165] Softmax, Cosmax corresponds to the method proposed by Chen et al. (Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. 2019. A closer look at few-shot classification. In ICLR. 1-17.);

[0166] Arcmax corresponds to the method proposed by Afrasiyabi, A. et al. (Afrasiyabi, A.; Lalonde, J.-F.; and Gagné, C. 2020. Associative alignment for few-shot image classification. In ECCV. 18–35.);

[0167] RFS corresponds to the method proposed by Tian et al. (Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. 2020. Rethinking few-shot image classification: A good embedding is all you need? In ECCV. 266–282.);

[0168] LR+DC corresponds to the method proposed by Yang, S. et al. (Yang, S.; Liu, L.; and Xu, M. 2021. Free lunch for fewshot learning: Distribution calibration. In ICLR. 1-13.);

[0169] STARTUP corresponds to the method proposed by Phoo et al. (Cheng Perng Phoo and Bharath Hariharan. 2021. Self-training for few-shot transfer across extreme task differences. In ICLR. 1–19.);

[0170] Meta-Baseline corresponds to the method proposed by Chen et al. (Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. 2021. Meta-Baseline: Exploring simple meta-learning for few-shot learning. In ICCV. 9062–9071.);

[0171] OAT corresponds to the method proposed by Chen et al. (Mengting Chen, Yuxin Fang, Xinggang Wang, Heng Luo, Yifeng Geng, Xinyu Zhang, Chang Huang, Wenyu Liu, and Bo Wang. 2020. Diversity transfer network for few-shot learning. In AAAI. 10559-10566.);

[0172] BML corresponds to the method proposed by Zhou et al. (Ziqi Zhou, Xi Qiu, Jiangtao Xie, Jianan Wu, and Chi Zhang. 2021. Binocular mutual learning for improving few-shot classification. In ICCV. 8402-8411.);

[0173] EGS-Net corresponds to the method proposed by Zou et al. (Xinyi Zou, Yan Yan, Jing-Hao Xue, Si Chen, and Hanzi Wang. 2022. When facial expression recognition meets few-shot learning: A joint and alternate learning framework. In AAAI. 5367-5375.);

[0174] CDNet corresponds to the method proposed by Zou et al. (Zou, X.; Yan, Y.; Xue, J.-H.; Chen, S.; and Wang, H. 2022. Learn-to-decompose: Cascaded decomposition network for cross-domain few-shot facial expression recognition. In ECCV, 683-700.);

[0175] Method (B) and Method (V) represent the evaluation of the present invention in the form of batch training and RFS as the baseline method, respectively.

[0176] As Figure 3As shown, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0177] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0178] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0179] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent device or equivalent process transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A small sample facial expression recognition method based on a self-paced co-learning framework, characterized in that, The method comprises the following steps: Step 1, obtaining basic class expression dataset and composite class expression dataset, and performing data preprocessing; Step 2, establishing a self-step collaborative learning network, the self-step collaborative learning network comprising a plurality of mixed expert convolution layers and a classification model; each mixed expert convolution layer comprising a parameter-shared router and a convolution module derived based on a backbone convolutional neural network; Step 3, scaling the channel numbers of different mixed expert convolution layers to change the size of the self-step collaborative learning network; Step 4, the scaled mixed expert convolution layers performing expression feature embedding learning in the Euclidean space through the basic class expression dataset to obtain an expert model; specifically comprising: Step 41: In each iteration, sample batches of data from the basic facial expression dataset. , Represents facial expressions. This indicates the expression category label corresponding to the expression image; for One of the samples First, the sample Through a collection A basic feature is extracted after preprocessing the base convolutional layers, ReLU activation function, and average pooling operation preprocessing block. Then the basic feature The preference evaluation network of the router, which is input into the first hybrid expert convolutional layer, calculates the preference scores corresponding to E experts. The expert selection network of routers in the first hybrid expert convolutional layer is based on preference scores. Select the expert z with the highest score; Step 42, the expert selection network generates a mask by the selected expert z , for selecting the channel subset corresponding to the expert z; when selecting the channel, the element in the mask of the selected channel is set to 1, otherwise 0; Step 43, apply mask to extract channel features of expert z and take it as the input feature of the 2nd mixed expert convolution layer; similarly, for each subsequent mixed expert convolution layer of the self-step co-learning network, the channel features from the last mixed expert convolution layer are sent to the router to select the optimal expert z; these selected channels form an end-to-end path, building an expert model suitable for the sample The final expression feature extracted by the expert model is represented as , and e is a symbol to distinguish it from other features and is used for expression classification; Step 5, obtaining expression features corresponding to the teacher model and the student model through the teacher model and the student model constructed based on the expert model, and designing a teacher-student correction loss to measure the difference between the expression features of the teacher model and the student model by using KL divergence, so as to strengthen the constraint and guiding effect of the teacher model on the student model; Step 6: Map the facial expression features acquired by the teacher and student models from Euclidean space to hyperbolic space, and perform facial expression feature embedding learning on the self-synchronizing collaborative learning network; in Euclidean space, for the input sample Facial feature representations were obtained using both teacher and student models. and Then, the facial features learned in Euclidean space are mapped to hyperbolic space, and their mathematical representation is as follows: where o represents the center of the Poincare ball, and c represents the curvature of the hyperbolic space; represents the standard Euclidean norm; is the hyperbolic tangent function; respectively, and characteristic representation in the hyperbolic space, and respectively represent the expression features obtained after the sample passes through the teacher model and the student model, represents the exponential mapping function; Step 7, using the classification model to classify the expression features to complete the final training; Step 8, inputting the composite class expression dataset into the trained self-step collaborative learning network to perform expression classification calculation, sending the samples of the support set and the query set into the student model comprising a plurality of expert models respectively to extract features, calculating the distance between each sample in the query set and each expression class in the support set through a measurement standard, and assigning the samples of the query set to the expression classes of the support set in the feature space which are most adjacent to the samples, outputting the classification results, and identifying different expressions.

2. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 1, wherein, The step 1 specifically comprises: Step 11, obtaining a basic class expression dataset for training, comprising three indoor datasets and two outdoor datasets, wherein one of the outdoor datasets is obtained by downloading from the Internet, and the other outdoor dataset comprises video clips extracted from movies; the basic class expression dataset carries basic expression labels, including anger, surprise, disgust, fear, happiness, sadness and neutrality, and one of the indoor datasets has an additional contempt expression label; Step 12, obtaining a composite class expression dataset for testing, comprising an indoor dataset and an outdoor dataset, the composite class expression dataset carrying detailed expression labels; Step 13, performing face alignment and cropping on all the basic class expression dataset and the composite class expression dataset as inputs of the self-step collaborative learning network; Step 14, during training, performing data augmentation on the basic class expression dataset by random flipping and color jittering.

3. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 1, wherein, The step 2 specifically comprises: Step 21, establishing a self-step collaborative learning network for expression recognition, the self-step collaborative learning network comprising a plurality of mixed expert convolution layers for extracting expression features and a classification model for expression class classification; Step 22, the router comprising a preference evaluation network for evaluating preference scores of experts and an expert selection network for selecting a part of channel features from the self-step collaborative learning network according to the preference scores; Step 23, the classification model is a distance calculation using a metric standard, and the expression image is classified into the expression category with the smallest distance.

4. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 3, wherein, The step 3 specifically includes: Step 31, for each mixed expert convolution layer, the mixed expert convolution layer represents a convolution layer containing multiple experts, and each mixed expert convolution layer contains E experts, each of which is responsible for learning expression features from a group of expression categories; Step 32, assuming that the first mixed-expert convolutional layer in the self-step co-learning network has channels, the number of channels of the first mixed-expert convolutional layer in the self-step co-learning network is updated from to , by using a scaling ratio r to scale the number of channels of the mixed-expert convolutional layer, that is, changing the number of convolution kernels, where each expert selects channels in the first mixed-expert convolutional layer; Step 33, Similarly, the number of channels of the i-th hybrid expert convolution layer in the self-step co-learning network is updated from to .

5. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 4, wherein, The step 43 further includes: Step 44, constrain the router selection with an expert based on the cross-entropy loss of preference score First, define a function to establish the correspondence between the true label of the training sample and the expert index, that is: wherein, denotes a true label of a training sample denotes a true label of a training sample denotes an expert index related to an input expression image; denotes the number of expression classes of a basic class expression dataset The first preference loss L1 is defined as: L1 = -log(p1) wherein M' represents the total number of mixed expert convolution layers, m represents the mth layer of the current mixed expert convolution layer, represents a cross-entropy loss function; Step 45, the router allows the filter router to select the expert which does not correspond to the expression category of the current sample based on the preference score when selecting the expert, and the second preference loss is defined as: wherein, represents a flooding level for controlling the loss fluctuation range, .

6. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 1, wherein, The step 5 specifically includes: Step 51, based on the expert model, a teacher model and a student model are constructed, the knowledge of the teacher model is migrated to the student model using a self-distillation network architecture, and the corresponding expression features are extracted using the teacher model and the student model; Step 52, for the i-th sample , feature extraction is performed using the student model to obtain expression features ; Step 53, for the i-th sample, feature extraction is performed using the teacher model to obtain expression features ; Step 54, the teacher model updates the parameters through an exponential moving average of the student model; Step 55, a teacher-student correction loss is introduced to strengthen the constraint and guidance of the teacher model on the student model; the mathematical representation of the teacher-student correction loss is as follows: wherein, represents the teacher-student correction loss function, represents the KL divergence, represents the feature distribution of the student model on the i-th sample , represents the feature distribution of the teacher model on the i-th sample , U represents the number of samples, represents the temperature coefficient, used to adjust the model output distribution, under the guidance of the teacher model, the student model continues to correct and optimize.

7. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 6, wherein, The step 6 specifically includes: Step 61, minimize the distance between the embeddings in hyperbolic space, represented as follows: wherein denotes the distance between two points in hyperbolic space, and denotes the distance between two points in hyperbolic space, and denotes the radius of the Poincare ball, and denotes the radius of the Poincare ball, denotes the inverse hyperbolic cosine function; Step 62, based on hyperbolic space self-hyperbolic uncertainty : The random Riemannian gradient descent is used to minimize and the Poincare distance between and is mathematically represented as follows: The learning process is self-paced, and the gradient is based on hyperbolic uncertainty from the teacher model Changes, i.e. The radius of The larger, The easier it is to learn, and the stronger the gradient Also.

8. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 5, wherein, The step 7 specifically is: Each iteration samples a batch of data from the basic class of facial expressions dataset. For one of the samples The features are then input into an expert model and a classification model to obtain the final facial expression features. and prediction results Each query image is assigned to its nearest support class center in the learned feature space, and the prediction result is calculated. With real labels Cross-entropy loss Used to update network parameters: where N is the number of basic facial expression classes, the indicator function 1 [n=yq] is 1 if and only if is 1, and are the facial feature embedding and the corresponding facial class label of the facial image for query, respectively; denotes the center of the facial class n, K denotes the total number of facial images, and k denotes the kth facial image, denotes the facial feature of the kth facial image that supports the facial class n in the support set.

9. The small sample facial expression recognition method based on the self-paced co-learning framework of claim 1, wherein, The step 8 specifically includes: Step 81, using the finally trained self-step collaborative learning network, a small sample classification task similar to the training process is sampled on the composite expression data set for testing, that is, N classes are randomly sampled, K samples are sampled for each class as a support set, and Q samples are sampled for each class as a query set, and the classification accuracy of all N*Q samples in the query set is calculated as the precision of the current task; Step 82, define the number of randomly sampled expression categories N, and evaluate the performance of the self-step collaborative learning network under different K values, and each small sample classification task takes N*Q query samples for classification evaluation; Step 83, take the average of the accuracies of multiple small sample classification tasks to evaluate the final performance; Step 84, different expressions are identified according to the classification results.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the small sample facial expression recognition method based on the self-step collaborative learning framework as claimed in any one of claims 1 to 9.

Images