Facial expression recognition method and device across datasets, equipment and readable storage medium

Abstract

This invention provides a method, apparatus, device, and readable storage medium for cross-dataset facial expression recognition. The method includes: performing data augmentation processing on N facial images respectively to obtain a front view x corresponding to each facial image. i_q and x i_k1 Find the heatmap h of facial feature points from N facial images respectively. i The nearest neighbor corresponding facial image x l ; Create a front view x_k1 = {x ...} corresponding to N facial images i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i The nearest neighbor face images are input to a momentum encoder; an improved InfoNCE loss function is obtained based on the output of the momentum encoder; the improved InfoNCE loss function and the front view x are then used to obtain the loss function. i_q Furthermore, the encoder and classifier parameters are tuned based on the target facial expression dataset. This invention addresses the problem in existing technologies where supervised pre-trained deep models cannot achieve facial expression recognition across datasets.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and readable storage medium for cross-dataset facial expression recognition. Background Technology

[0002] Currently, deep learning technology is increasingly being applied to the field of facial expression recognition, with supervised pre-trained deep models dominating the field. However, most current supervised pre-trained deep models rely on transformations introduced by predefined data augmentation on the same samples. This fails to connect multiple samples that may belong to the same semantic category, resulting in a lack of learning signals between visually identical instances. This reduces the invariance of supervised pre-trained deep models to large intra-class variations. Therefore, when each facial expression dataset provides different labels, most supervised pre-trained deep models need to be trained and tested separately for each dataset, making cross-dataset facial expression recognition impossible. Thus, we urgently need a technical solution that requires only one pre-trained model for different expression datasets and can bridge multiple samples as long as they are semantically similar. Summary of the Invention

[0003] The main objective of this invention is to provide a method, apparatus, device, and readable storage medium for cross-dataset facial expression recognition, aiming to solve the problem that supervised pre-trained deep models cannot achieve cross-dataset facial expression recognition in the prior art.

[0004] In a first aspect, the present invention provides a cross-dataset facial expression recognition method, the cross-dataset facial expression recognition method comprising:

[0005] Data augmentation is performed on N facial images to obtain the front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer;

[0006] Find the heatmap h of facial feature points from N facial images respectively. i The nearest neighbor corresponding facial image x l ;

[0007] The frontal views x_k1 = {x} corresponding to N facial images i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder;

[0008] An improved InfoNCE loss function is obtained based on the output of the momentum encoder;

[0009] Through the improved InfoNCE loss function and the front view x i_q The encoder is trained to obtain a trained encoder;

[0010] After performing face detection on the images in the target facial expression dataset, facial images are obtained and then input into the trained encoder to obtain the feature vector of the facial image.

[0011] The classifier is trained based on the feature vector to obtain the trained classifier;

[0012] Adjust the parameters of the trained encoder and the trained classifier.

[0013] Optionally, the step of training the classifier based on the feature vector to obtain the trained classifier includes:

[0014] Based on the feature vector, the classifier is trained to perform facial expression recognition tasks using linear probing. The cross-entropy loss function is used to train the classifier. The linear probing process is as follows: keep the parameters of the trained encoder unchanged, change the parameters of the classifier, until the cross-entropy loss function of the classifier reaches the optimal value.

[0015] Optionally, before the step of performing data augmentation processing on each of the N facial images, the following steps are included:

[0016] The training sets of facial expression datasets from different environments are merged to obtain a new training set;

[0017] A new set of facial images corresponding to the training set is obtained through a face detection algorithm;

[0018] Select N facial images from the set of facial images.

[0019] Optional, improved InfoNCE loss function is as follows:

[0020]

[0021] Where E represents the mean, z i *z p and z i *z n These refer to the dot product between the two vectors, τ refers to the temperature hyperparameter, and p i Let {k} be a positive set. -} represents an element in the data sample queue.

[0022] Optionally, the positive set p i From facial image x iThe enhanced image and the RGB images corresponding to the top L nearest neighbors in the heatmap space constitute the image, where,

[0023]

[0024] h i *h j The topL(·) operator represents the similarity between the heatmaps of face image i and face image j. It searches for the top L items among all N available face images and returns their indices, where L is a hyperparameter, A is the data augmentation set, and ψ(x) represents the similarity between the heatmaps of face image i and face image j. i ; a) is an augmentation function, where a is randomly sampled from a set of predefined data augmentation transformations A.

[0025] Secondly, the present invention also provides a cross-dataset facial expression recognition device, the cross-dataset facial expression recognition device comprising:

[0026] The processing module is used to perform data augmentation processing on N facial images respectively, to obtain the front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer;

[0027] The search module is used to find heatmaps h of facial feature points from N facial images. i The nearest neighbor corresponding facial image x l ;

[0028] The first input module is used to input the frontal views x_k1 = {x_k1} corresponding to N facial images. i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder;

[0029] An improved module is used to obtain an improved InfoNCE loss function based on the output of the momentum encoder;

[0030] The first training module is used to train the image using the improved InfoNCE loss function and the front view x. i_q The encoder is trained to obtain a trained encoder;

[0031] The second input module is used to perform face detection on the images in the target facial expression dataset to obtain facial images, and input them into the trained encoder to obtain the feature vector of the facial images.

[0032] The second training module is used to train the classifier based on the feature vector to obtain the trained classifier.

[0033] The parameter tuning module is used to adjust the parameters of the trained encoder and the trained classifier.

[0034] Optional, a second training module, used for:

[0035] Based on the feature vector, the classifier is trained to perform facial expression recognition tasks using linear probing. The cross-entropy loss function is used to train the classifier. The linear probing process is as follows: keep the parameters of the trained encoder unchanged, change the parameters of the classifier, until the cross-entropy loss function of the classifier reaches the optimal value.

[0036] Optionally, the cross-dataset facial expression recognition device further includes a selection module for:

[0037] The training sets of facial expression datasets from different environments are merged to obtain a new training set;

[0038] A new set of facial images corresponding to the training set is obtained through a face detection algorithm;

[0039] Select N facial images from the set of facial images.

[0040] Thirdly, the present invention also provides a cross-dataset facial expression recognition device, the cross-dataset facial expression recognition device including a processor, a memory, and a cross-dataset facial expression recognition program stored in the memory and executable by the processor, wherein when the cross-dataset facial expression recognition program is executed by the processor, it implements the steps of the cross-dataset facial expression recognition method as described above.

[0041] Fourthly, the present invention also provides a readable storage medium storing a cross-dataset facial expression recognition program, wherein when the cross-dataset facial expression recognition program is executed by a processor, it implements the steps of the cross-dataset facial expression recognition method as described above.

[0042] In this invention, data augmentation processing is performed on N facial images to obtain a front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer; find the heatmap h of facial feature points for each of the N facial images. i The nearest neighbor corresponding facial image x l ; Create a front view x_k1 = {x ...} corresponding to N facial images i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder; derive improved InfoNCE loss function based on momentum encoder output; combine improved InfoNCE loss function with front view x i_q The encoder is trained to obtain a trained encoder; face detection is performed on images from the target facial expression dataset to obtain facial images, which are then input into the trained encoder to obtain feature vectors for the facial images; a classifier is trained based on these feature vectors to obtain a trained classifier; and the parameters of the trained encoder and the trained classifier are adjusted. This invention employs an improved InfoNCE loss function and a frontal view x corresponding to each facial image. i_q The encoder is trained and then applied to different target facial expression datasets to achieve cross-dataset facial expression recognition. Simultaneously, landmark heatmap information is used to accurately capture the most distinctive facial expression features, i.e., through a heatmap h of facial feature points from N facial images. i The algorithm uses the nearest neighbor facial images to improve the encoder's invariance to intra-class changes in facial expressions. Then, it adjusts the parameters of the trained encoder and classifier to improve the performance and accuracy of facial expression recognition across datasets. This solves the problem that supervised pre-trained deep models cannot perform facial expression recognition across datasets in existing technologies. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the hardware structure of the cross-dataset facial expression recognition device involved in the embodiments of the present invention;

[0044] Figure 2 This is a flowchart illustrating the first embodiment of the cross-dataset facial expression recognition method of the present invention;

[0045] Figure 3 This is a flowchart illustrating the second embodiment of the cross-dataset facial expression recognition method of the present invention;

[0046] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the cross-dataset facial expression recognition device of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] In a first aspect, embodiments of the present invention provide a cross-dataset facial expression recognition device, which can be a device with data processing capabilities such as a personal computer (PC), a laptop computer, or a server.

[0050] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of a cross-dataset facial expression recognition device involved in an embodiment of the present invention. In this embodiment, the cross-dataset facial expression recognition device may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components; the user interface 1003 may include a display screen or an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., Wireless Fidelity, Wi-Fi); the memory 1005 may be high-speed random access memory (RAM) or stable memory (non-volatile memory), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that… Figure 1 The hardware structure shown does not constitute a limitation of the invention and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] Continue to refer to Figure 1 , Figure 1 The memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a cross-dataset facial expression recognition program. The processor 1001 can call the cross-dataset facial expression recognition program stored in the memory 1005 and execute the cross-dataset facial expression recognition method provided in this embodiment of the invention.

[0052] Secondly, embodiments of the present invention provide a method for facial expression recognition across datasets.

[0053] In one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the cross-dataset facial expression recognition method of the present invention. Figure 2 As shown, the cross-dataset facial expression recognition method includes:

[0054] Step S10: Perform data augmentation processing on each of the N facial images to obtain the front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer;

[0055] In this embodiment, data augmentation processing generates two front views of the same facial image, with each facial image corresponding to two front views, x and x'. i_q and x i_k1 Where i∈[1,N], and N is a positive integer. By performing data augmentation on N facial images respectively, 2N views can be obtained, i.e., x_k1={x i_k1 |i∈[1,N]} and x_q={x i_q |i∈[1,N]}. Data augmentation primarily aims to reduce overfitting in networks. By transforming images, networks with stronger generalization capabilities can be obtained, better adapting to application scenarios. Common data augmentation methods include flipping, rotating, scaling, cropping, shifting, Gaussian noise, and color enhancement.

[0056] Step S20: Find the heatmap h of facial feature points for each of the N facial images. i The nearest neighbor corresponding facial image x l ;

[0057] In this embodiment, the facial image x is searched. i heatmap of facial feature points h i The nearest neighbor corresponding facial image x l Find the heatmap h of facial feature points from N facial images respectively. i The nearest neighbor corresponding facial image x l ,get

[0058] Step S30, generate the front view x_k1 = {x} corresponding to the N facial images. i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder;

[0059] In this embodiment, the frontal view x_k1 = {x} corresponding to N facial images is used. i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input the momentum encoder to obtain the improved InfoNCE loss function.

[0060] Step S40: Obtain the improved InfoNCE loss function based on the output of the momentum encoder;

[0061] In this embodiment, an improved InfoNCE loss function is obtained based on the output of the momentum encoder. When M data points enter the data sample queue from the momentum encoder output, the first M elements to enter the data sample queue are removed from the queue to maintain its integrity. The elements in the data sample queue represent negative views.

[0062] Furthermore, in one embodiment, the improved InfoNCE loss function is as follows:

[0063]

[0064] Where E represents the mean, z i *z p and z i *z n These refer to the dot product between the two vectors, τ refers to the temperature hyperparameter, and p i Let {k} be a positive set. -} represents an element in the data sample queue.

[0065] In this embodiment, the InfoNCE loss function is the loss function of the contrastive learning algorithm. An improved InfoNCE loss function is obtained based on the output of the momentum encoder. The improved InfoNCE loss function is as follows:

[0066]

[0067] Where E represents the mean, z i *z p and z i *z n These refer to the dot product between two vectors, (z) i , z p ) is correct, (z i , z n ) is the embedding of any negative pair and many negative instances z. n ∈N i τ refers to the temperature hyperparameter p in Wu, Zhirong et al., “Unsupervised Feature Learning via Non-parametric Instance Discrimination.” 2018 IEEE / CVFConference on Computer Vision and Pattern Recognition (2018):3733-3742. i Let {k} be a positive set. -} represents an element in the data sample queue, the number of elements in the data sample queue (queue size) is K, and the numerator is defined as the face image x. i Data augmented images and positive sets p i The sum of similarities between them. Before calculating the dot product in the loss, each embedding representation, i.e., the feature vector, is L2 normalized.

[0068] Furthermore, in one embodiment, the positive set p i From facial image x i The enhanced image and the RGB images corresponding to the top L nearest neighbors in the heatmap space constitute the image, where,

[0069]

[0070] h i *h j The topL(·) operator represents the similarity between the heatmaps of face image i and face image j. It searches for the top L items among all N available face images and returns their indices, where L is a hyperparameter, A is the data augmentation set, and ψ(x) represents the similarity between the heatmaps of face image i and face image j. i ; a) is an augmentation function, where a is randomly sampled from a set of predefined data augmentation transformations A.

[0071] In this embodiment, the positive set p i From facial image x i The enhanced image and the RGB images corresponding to the top L nearest neighbors in the heatmap space constitute the image, where,

[0072]

[0073] h i *h j The topL(·) operator represents the similarity between the heatmaps of face image i and face image j. It searches for the top L items among all N available face images and returns their indices, where L is a hyperparameter, A is the data augmentation set, and ψ(x) represents the similarity between the heatmaps of face image i and face image j. i ; a) is an augmentation function, where a is randomly sampled from a predefined set of data augmentation transformations A. The settings of the data augmentation set A are the same as those in Chen, Ting et al., “ASimple Framework for Contrastive Learning of Visual Representations.” In ICML, 2020.

[0074] It should be noted that, in order to bring different facial images belonging to the same expression closer together, this scheme proposes a heatmap nearest neighbor contrastive learning method. It utilizes the nearest neighbors of facial landmark heatmaps to find more diverse positive pairs, going beyond single-instance positive examples and increasing the richness of the latent representation (feature vector). In addition to using two enhanced images to form positive pairs, it also uses images corresponding to the nearest neighbors of the facial landmark heatmaps to form positive pairs. Negative samples are obtained from the queue, allowing reuse of features already encoded in the current mini-batch. Simultaneously, the data sample queue decouples the size of the negative sample set from the mini-batch size, allowing it to be much larger than the mini-batch size. The larger queue better samples the latent continuous high-dimensional expression space. Unlike other schemes, this scheme's queue stores not only the encodings of enhanced images from different batches but also the encodings of images corresponding to the nearest neighbors of the facial landmark heatmaps. Furthermore, to maintain the consistency of features in the data sample queue, this scheme employs a momentum encoder.

[0075] Step S50, using the improved InfoNCE loss function and the front view x i_q The encoder is trained to obtain a trained encoder;

[0076] In this embodiment, the front view x_q = {x} corresponding to N facial images is used. i_q |i∈[1,N]} are input into the encoder network, and the improved InfoNCE loss function is used to process the frontal views x_q={x_i∈[1,N]} corresponding to the input N facial images. i_q The encoder is trained using the formula |i∈[1,N]} to obtain the trained encoder. The heatmap proximity contrast learning method uses the facial feature point heatmap space to find actual semantic samples.

[0077] Step S60: After performing face detection on the images in the target facial expression dataset, obtain facial images, and input them into the trained encoder to obtain the feature vector of the facial images.

[0078] In this embodiment, face detection is performed on the images in the target facial expression dataset to obtain facial images. These facial images are then input into a trained encoder to obtain a representation of the facial images, i.e., a feature vector. The facial expression dataset to be identified is the target facial expression dataset.

[0079] Step S70: Train the classifier based on the feature vector to obtain the trained classifier;

[0080] In this embodiment, a classifier is trained based on the feature vector of the facial image to obtain a trained classifier, which is then used to recognize facial expressions.

[0081] Further, in one embodiment, step S70 includes:

[0082] Based on the feature vector, the classifier is trained to perform facial expression recognition tasks using linear probing. The cross-entropy loss function is used to train the classifier. The linear probing process is as follows: keep the parameters of the trained encoder unchanged, change the parameters of the classifier, until the cross-entropy loss function of the classifier reaches the optimal value.

[0083] In this embodiment, based on the feature vectors of facial images, a classifier is trained to perform facial expression recognition tasks using linear probing. The classifier is trained using the cross-entropy loss function. The linear probing process involves keeping the parameters of the trained encoder constant while changing the parameters of the classifier until the cross-entropy loss function reaches its optimal value. Cross-entropy measures the degree of difference between two different probability distributions of the same random variable; in machine learning, it represents the difference between the true probability distribution and the predicted probability distribution. The smaller the cross-entropy value, the better the model's prediction performance.

[0084] Step S80: Adjust the parameters of the trained encoder and the trained classifier.

[0085] In this embodiment, the trained encoder and the trained classifier are retrained together, that is, the parameters of the trained encoder and the trained classifier are adjusted so that the loss function of the trained encoder and the trained classifier reaches the optimal value.

[0086] In this embodiment, data augmentation processing is performed on N facial images to obtain a front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer; find the heatmap h of facial feature points for each of the N facial images. i The nearest neighbor corresponding facial image x l ; Create a front view x_k1 = {x ...} corresponding to N facial images i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder; derive improved InfoNCE loss function based on momentum encoder output; combine improved InfoNCE loss function with front view x i_qThe encoder is trained to obtain a trained encoder; face detection is performed on the images in the target facial expression dataset to obtain facial images, which are then input into the trained encoder to obtain feature vectors for the facial images; a classifier is trained based on these feature vectors to obtain a trained classifier; and the parameters of the trained encoder and the trained classifier are adjusted. This embodiment employs an improved InfoNCE loss function and a frontal view x corresponding to each facial image. i_q The encoder is trained and then applied to different target facial expression datasets to achieve cross-dataset facial expression recognition. Simultaneously, landmark heatmap information is used to accurately capture the most distinctive facial expression features, i.e., through a heatmap h of facial feature points from N facial images. i The algorithm uses the nearest neighbor facial images to improve the encoder's invariance to intra-class changes in facial expressions. Then, it adjusts the parameters of the trained encoder and classifier to improve the performance and accuracy of facial expression recognition across datasets. This solves the problem that supervised pre-trained deep models cannot perform facial expression recognition across datasets in existing technologies.

[0087] Furthermore, in one embodiment, reference is made to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the cross-dataset facial expression recognition method of the present invention. Figure 3 As shown, before the step of performing data augmentation processing on each of the N facial images, the following steps are included:

[0088] Step S001: Merge all training sets of facial expression datasets from different environments to obtain a new training set;

[0089] Step S002: Obtain the set of facial images corresponding to the new training set through the face detection algorithm;

[0090] Step S003: Select N facial images from the facial image set.

[0091] In this embodiment, different environments include laboratory environments and field environments. All training sets of facial expression datasets from both laboratory and field environments are merged to obtain a new training set. A face detection algorithm is used to obtain a set of facial images corresponding to the new training set. N facial images {x1, x2, ..., x...} are randomly selected from this set. N}, for execution of step S10.

[0092] Thirdly, embodiments of the present invention also provide a facial expression recognition device across datasets.

[0093] In one embodiment, reference is made to Figure 4 , Figure 4This is a functional module diagram of an embodiment of the cross-dataset facial expression recognition device of the present invention. Figure 4 As shown, a cross-dataset facial expression recognition device includes:

[0094] Processing module 10 is used to perform data augmentation processing on N facial images respectively, to obtain the front view x corresponding to each facial image. i_q and x i_k1 Where i∈[1,N], and N is a positive integer;

[0095] Module 20 is used to find heatmaps h of facial feature points from N facial images respectively. i The nearest neighbor corresponding facial image x l ;

[0096] The first input module 30 is used to input the frontal views x_k1 = {x_k1} corresponding to N facial images. i_k1 Heatmap h of facial feature points from |i∈[1,N]} and N facial images i Nearest neighbor corresponding facial images Input momentum encoder;

[0097] Improvement module 40 is used to obtain an improved InfoNCE loss function based on the output of the momentum encoder;

[0098] The first training module 50 is used to train the image using an improved InfoNCE loss function and a front view x. i_q The encoder is trained to obtain a trained encoder;

[0099] The second input module 60 is used to perform face detection on the images in the target facial expression dataset to obtain facial images, and input them into the trained encoder to obtain the feature vector of the facial images.

[0100] The second training module 70 is used to train the classifier based on the feature vector to obtain the trained classifier.

[0101] The parameter adjustment module 80 is used to adjust the parameters of the trained encoder and the trained classifier.

[0102] Furthermore, in one embodiment, the second training module 70 is used for:

[0103] Based on the feature vector, the classifier is trained to perform facial expression recognition tasks using linear probing. The cross-entropy loss function is used to train the classifier. The linear probing process is as follows: keep the parameters of the trained encoder unchanged, change the parameters of the classifier, until the cross-entropy loss function of the classifier reaches the optimal value.

[0104] Furthermore, in one embodiment, the cross-dataset facial expression recognition device further includes a selection module for:

[0105] The training sets of facial expression datasets from different environments are merged to obtain a new training set;

[0106] A new set of facial images corresponding to the training set is obtained through a face detection algorithm;

[0107] Select N facial images from the set of facial images.

[0108] Furthermore, in one embodiment, the improved InfoNCE loss function is as follows:

[0109]

[0110] Where E represents the mean, z i *z p and z i *z n These refer to the dot product between the two vectors, τ refers to the temperature hyperparameter, and p i Let {k} be a positive set. -} represents an element in the data sample queue.

[0111] Furthermore, in one embodiment, the positive set p i From facial image x i The enhanced image and the RGB images corresponding to the top L nearest neighbors in the heatmap space constitute the image, where,

[0112]

[0113] h i *h j The topL(·) operator represents the similarity between the heatmaps of face image i and face image j. It searches for the top L items among all N available face images and returns their indices, where L is a hyperparameter, A is the data augmentation set, and ψ(x) represents the similarity between the heatmaps of face image i and face image j. i ; a) is an augmentation function, where a is randomly sampled from a set of predefined data augmentation transformations A.

[0114] The functions of each module in the aforementioned cross-dataset facial expression recognition device correspond to the steps in the aforementioned cross-dataset facial expression recognition method embodiment, and their functions and implementation processes will not be described in detail here.

[0115] Fourthly, embodiments of the present invention also provide a readable storage medium.

[0116] The present invention provides a cross-dataset facial expression recognition program stored on a readable storage medium, wherein when the cross-dataset facial expression recognition program is executed by a processor, it implements the steps of the cross-dataset facial expression recognition method as described above.

[0117] The method implemented when the cross-dataset facial expression recognition program is executed can be referred to in various embodiments of the cross-dataset facial expression recognition method of the present invention, and will not be repeated here.

[0118] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0119] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of the present invention.

[0121] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cross dataset facial expression recognition method, characterized in that, The cross-dataset facial expression recognition method comprises the following steps: N face images are respectively subjected to data enhancement processing to obtain a front view corresponding to each face image and wherein, N is a positive integer; Respectively find the face feature point heat map of N face images Face image corresponding to the neighbor ; N face images corresponding to the front view and the heat map of the face feature points of the N face images the face image corresponding to the nearest neighbor input momentum encoder; An improved InfoNCE loss function is obtained based on the output of the momentum encoder; By improved InfoNCE loss function and positive view Train the encoder to obtain a trained encoder; After face detection is performed on the pictures of the target facial expression dataset, facial images are obtained, and the facial images are input into the trained encoder to obtain feature vectors of the facial images; The classifier is trained based on the feature vectors to obtain a trained classifier; Parameters of the trained encoder and the trained classifier are adjusted; The improved InfoNCE loss function is as follows: where E denotes the averaging, and denotes the dot product between two vectors, denotes the temperature hyperparameter, is the positive set, denotes an element in the queue of data samples.

2. The cross dataset facial expression recognition method of claim 1, wherein, The step of training the classifier based on the feature vectors comprises the following steps: The classifier is trained based on the feature vectors to perform a facial expression recognition task through linear probing, wherein the cross-entropy loss function is used for training the classifier, and the linear probing process is as follows: the parameters of the trained encoder are kept unchanged, and the parameters of the classifier are changed until the cross-entropy loss function of the classifier reaches an optimal value.

3. The cross dataset facial expression recognition method of claim 1, wherein, Before the step of performing data enhancement processing on the N facial images respectively, the following steps are further included: All training sets of the facial expression datasets in different environments are merged to obtain a new training set; The facial image set corresponding to the new training set is obtained through a face detection algorithm; N facial images are selected from the facial image set.

4. The cross dataset facial expression recognition method of claim 1, wherein, The positive set The enhanced image and the RGB image corresponding to the L nearest neighbors in the face image space, wherein, a similarity between a heat map of a face image i and a heat map of a face image j, The operator finds the topL items among all available N face images and returns their indices, L is a hyper-parameter, A is the data augmentation set, is an augmentation function, a is randomly sampled from a set of predefined data augmentation transformations A.

5. An apparatus for cross dataset facial expression recognition, the apparatus comprising: The cross-dataset facial expression recognition device comprises the following modules: The processing module is configured to perform data enhancement processing on the N face images respectively to obtain a front view corresponding to each face image and wherein, N is a positive integer. The finding module is configured to find the heat map of the facial feature point of each of the N facial images corresponding to the neighbor of the facial image ; The first input module is configured to input N face images corresponding to the front view and the heat map of the face feature points of the N face images The face image corresponding to the nearest neighbor The input momentum encoder An improvement module is configured to obtain an improved InfoNCE loss function based on the output of the momentum encoder; The first training module is configured to train the encoder by using an improved InfoNCE loss function and positive views The encoder is trained to obtain a trained encoder. A second input module is configured to perform face detection on pictures of a target facial expression dataset to obtain facial images, and input the facial images into the trained encoder to obtain feature vectors of the facial images; A second training module is configured to train a classifier based on the feature vectors to obtain a trained classifier; A parameter adjustment module is configured to adjust parameters of the trained encoder and the trained classifier; The improved InfoNCE loss function is as follows: where E denotes the averaging, and denotes the dot product between two vectors, denotes the temperature hyperparameter, is the positive set, denotes an element in the queue of data samples.

6. The cross dataset facial expression recognition apparatus of claim 5, wherein, The second training module is configured to: Train the classifier based on the feature vectors to perform a facial expression recognition task through linear probing, wherein the cross-entropy loss function is used for training the classifier, and the linear probing process is as follows: the parameters of the trained encoder are kept unchanged, and the parameters of the classifier are changed until the cross-entropy loss function of the classifier reaches an optimal value.

7. The cross dataset facial expression recognition apparatus of claim 5, wherein, The cross-dataset facial expression recognition device further comprises a selection module configured to: Merge all training sets of the facial expression datasets in different environments to obtain a new training set; Obtain a facial image set corresponding to the new training set through a face detection algorithm; Select N facial images from the facial image set.

8. A cross dataset facial expression recognition device, comprising: The cross-dataset facial expression recognition device comprises a processor, a memory, and a cross-dataset facial expression recognition program stored in the memory and executable by the processor, wherein when the cross-dataset facial expression recognition program is executed by the processor, the steps of the cross-dataset facial expression recognition method according to any one of claims 1 to 4 are implemented.

9. A readable storage medium, characterized by, The readable storage medium has stored thereon a cross-dataset facial expression recognition program, and the cross-dataset facial expression recognition program, when executed by the processor, implements the steps of the cross-dataset facial expression recognition method according to any one of claims 1 to 4.

Images