Data augmentation method, facial emotion recognition method and device

Abstract

The present application discloses a data augmentation method, a facial emotion recognition method and a device, and relates to the technical field of facial emotion recognition. The method includes training a basic model using a number of facial sample images labeled with facial unit labels and facial expression labels to obtain a first neural network model and a confidence threshold set; inputting a number of facial unlabeled images into the first neural network model to obtain a number of facial pseudo-label images labeled with facial unit pseudo-labels and facial expression pseudo-labels and screening them using the threshold set to obtain a plurality of facial augmented images; obtaining a facial data set based on the facial augmented images, facial sample images and text sample sets; obtaining the facial images to be tested and text data as inputs of a second neural network model trained by the facial data set, and obtaining facial unit recognition results and facial expression recognition results through the second neural network model. The present application realizes the synchronous recognition of facial expressions and facial units and joint task training.

Description

Technical Field

[0001] The present application relates to the field of facial emotion recognition technology, and in particular to a data augmentation method, a facial emotion recognition method and a device. Background Art

[0002] Facial expressions and facial action units represent human facial emotions at different granularity levels, respectively. Facial action units can be combined in various ways to form facial expressions. In previous studies, facial expression recognition and facial action unit detection are usually regarded as two independent tasks. With the development of deep learning and computer vision, some methods have tried to explore the correlation between facial expressions and facial action units, but these methods often only use the knowledge information of one to assist in the recognition of the other, such as using facial action units to assist in the recognition of facial expressions. At present, there are few methods for joint training and synchronous recognition of facial expression recognition tasks and facial action unit detection tasks. In addition, the Aff-Wild2 dataset has both FE and AU annotations, which is suitable for training neural network models for facial expression recognition tasks and facial action unit detection tasks. However, since the process of annotating facial expressions and facial action units is time-consuming and labor-intensive, the Aff-Wild2 dataset that relies on continuous video frame annotation covers a limited number of subjects and scenes, and the video resolution in the Aff-Wild2 dataset is low, these factors inevitably limit the zero-shot capability and generalization ability of the neural network model. Summary of the invention

[0003] The embodiments of the present application propose a data augmentation method, a facial emotion recognition method and a device for improving the zero-sample capability and generalization capability of a neural network model, and for realizing joint training of facial expression recognition tasks and facial unit detection tasks and simultaneous recognition of facial units and facial expressions.

[0004] On the one hand, an embodiment of the present application provides a data augmentation method, the method comprising the following steps:

[0005] Acquire a plurality of facial sample images and a plurality of facial unlabeled images, each of the facial sample images being labeled with a corresponding facial unit label and a facial expression label;

[0006] Using a plurality of the facial sample images to train a pre-built basic model, a first neural network model and a confidence threshold set are obtained, wherein the confidence threshold set includes a confidence threshold of each facial unit label and a confidence threshold of each facial expression label;

[0007] Inputting a plurality of the facial unlabeled images into the first neural network model to obtain a plurality of facial pseudo-labeled images, and selecting a plurality of facial augmented images from the plurality of the facial pseudo-labeled images according to the confidence threshold set, each of the facial pseudo-labeled images being annotated with a corresponding facial unit pseudo-label and a facial expression pseudo-label;

[0008] A facial data set is obtained according to the plurality of facial augmented images, a number of facial sample images and a text sample set, wherein the text sample set includes a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions.

[0009] On the other hand, an embodiment of the present application provides a facial emotion recognition method, the method comprising the following steps:

[0010] Acquire a facial image to be tested and text data as inputs of a second neural network model, wherein the text data includes text data associated with facial units after mask processing and text data associated with facial expressions;

[0011] Performing facial recognition on the facial image to be tested by using the second neural network model to obtain a facial unit recognition result and a facial expression recognition result;

[0012] The second neural network model is obtained by training a facial data set, and the facial data set is obtained by the above-mentioned data augmentation method. The second neural network model includes:

[0013] A basic model is used to perform encoding processing and linear processing on the facial image to be tested to obtain facial unit visual features and facial expression visual features;

[0014] A completion module is used to encode the text data associated with the facial unit to obtain facial unit encoding features, obtain facial unit text features based on the facial unit encoding features and the facial unit visual features, and encode the text data associated with the facial expression to obtain facial expression text features;

[0015] A perception module is used to obtain the facial unit recognition result based on the facial unit visual features and the facial unit text features, and to obtain the facial expression recognition result based on the facial unit visual features, the facial expression visual features, the facial unit text features and the facial expression text features.

[0016] In another aspect, an embodiment of the present application provides a data augmentation device, the device comprising:

[0017] A first acquisition module is used to acquire a plurality of facial sample images and a plurality of facial unlabeled images, each of the facial sample images being labeled with a corresponding facial unit label and a facial expression label;

[0018] A first processing module, used to train a pre-built basic model using a plurality of facial sample images to obtain a first neural network model and a confidence threshold set, wherein the confidence threshold set includes a confidence threshold of each facial unit label and a confidence threshold of each facial expression label;

[0019] A second processing module is used to input the plurality of face unlabeled images into the first neural network model to obtain a plurality of face pseudo-labeled images, and select a plurality of face augmented images from the plurality of face pseudo-labeled images according to the confidence threshold set, each of the face pseudo-labeled images being annotated with a corresponding facial unit pseudo-label and a facial expression pseudo-label;

[0020] The third processing module is used to obtain a facial data set according to the plurality of facial augmented images, the plurality of facial sample images and a text sample set, wherein the text sample set includes a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions.

[0021] On the other hand, an embodiment of the present application provides a facial emotion recognition device, the device comprising:

[0022] A second acquisition module is used to acquire a facial image to be tested and text data as inputs of a second neural network model, wherein the text data includes text data associated with facial units after mask processing and text data associated with facial expressions;

[0023] The fourth processing module is used to perform facial recognition on the facial image to be tested through the second neural network model to obtain a facial unit recognition result and a facial expression recognition result; wherein the second neural network model is obtained by training a facial data set, and the facial data set is obtained by the above-mentioned data augmentation method, and the second neural network model includes:

[0024] A basic model is used to perform encoding processing and linear processing on the facial image to be tested to obtain facial unit visual features and facial expression visual features;

[0025] A completion module is used to encode the text data associated with the facial unit to obtain facial unit encoding features, obtain facial unit text features based on the facial unit encoding features and the facial unit visual features, and encode the text data associated with the facial expression to obtain facial expression text features;

[0026] A perception module is used to obtain the facial unit recognition result based on the facial unit visual features and the facial unit text features, and to obtain the facial expression recognition result based on the facial unit visual features, the facial expression visual features, the facial unit text features and the facial expression text features.

[0027] The beneficial effects of the present application are: providing a data augmentation method, a facial emotion recognition method and a device, firstly using a first neural network model trained by labeled facial sample images to perform pseudo-label generation processing on a large number of facial unlabeled images, obtaining a large number of facial pseudo-label images, all of which are annotated with corresponding pseudo-labels, and screening these facial pseudo-label images based on a category-specific confidence threshold to obtain a plurality of highly credible facial augmented images to ensure the credibility of the pseudo-labels and reduce the noise of the pseudo-labels, and then constructing a facial data set through a plurality of facial sample images, a plurality of facial augmented images and a text sample set for describing facial expressions and facial units, and obtaining a second neural network model through training the facial data set, and then realizing synchronous recognition of facial expressions and facial units through the second neural network model. The present application improves the zero-sample capability and generalization capability of the neural network model, and realizes the joint training of facial expression recognition tasks and facial unit detection tasks and the synchronous recognition of facial units and facial expressions.

[0028] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the association between facial units and facial expressions provided by the present application;

[0030] Figure 2 is a flow chart of the data augmentation method provided by this application;

[0031] Figure 3 It is a schematic diagram of the data augmentation method and the facial emotion recognition method provided by this application;

[0032] Figure 4 It is a flowchart of the facial emotion recognition method provided by this application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] Facial Expression (FE) and Facial Action Unit (AU) represent human facial emotions at different granularity levels. FE is usually divided into seven basic categories: neutral, angry, disgusted, fearful, happy, sad, and surprised. These basic categories broadly describe the overall facial movements. According to the definition of AU by the Facial Action Coding System (FACS), AU provides a detailed and objective description of the activation state and movement details of facial muscles or muscle groups. For example, AU12 (Lip Corner Puller) describes the action of pulling the corners of the mouth, which is highly correlated with the activation of the zygomatic major muscle.

[0035] One or more AUs can be combined in various ways to form a FE. Figure 1 , AU6 (CheekRaiser), AU7 (Lid Tightener), AU12 (Lip Corner Puller) and AU25 (Lips Part) can be combined to form a happy FE, while AU4 (Brow Lowerer), AU7 and AU24 (Lips Pressor) can be combined to form an angry FE. It can be seen that understanding the correspondence between FE and AU can more accurately analyze and study facial emotions.

[0036] In past studies, the Facial Expression Recognition (FER) task and the Facial Action Unit Detection (AUD) task are usually regarded as two independent tasks. With the development of deep learning and computer vision, some methods have tried to explore the correlation between FE and AU, but these methods often only use the knowledge information of one to assist in the recognition of the other, such as using facial action units to assist in the recognition of facial expressions. Currently, there are few methods for joint training and synchronous recognition of FER and AUD tasks.

[0037] Since the Aff-Wild2 dataset has both FE and AU annotations, it is suitable for training neural network models for FER and AUD tasks. However, due to the time-consuming and labor-intensive process of annotating FE and AU, the limited subjects and scenes covered by the Aff-Wild2 dataset that relies on continuous video frame annotation, and the low video resolution in the Aff-Wild2 dataset, these factors inevitably limit the zero-shot capability and generalization ability of the neural network model. Unlabeled facial images usually cover a wide range of subjects, scenes, postures, and environments. The introduction of unlabeled facial images can significantly expand the diversity of the data and overcome the above dataset problems to a certain extent, which is crucial to the generalization performance of the neural network model. Although a large number of easily available unlabeled facial images can significantly enrich the diversity of the dataset, how to effectively and appropriately utilize these unlabeled facial data to enhance the performance of neural network models for FER and AUD tasks remains a key challenge.

[0038] In addition, compared with the traditional one-hot label based on digital encoding as a supervisory signal, text description provides richer semantic information. In recent years, Contrastive Language-Image Pre-training (CLIP) has gradually emerged. It constructs a latent space by pre-training on large-scale image-text pairs, in which image embeddings are highly aligned with text embeddings, thereby giving the neural network model strong zero-shot capability and generalization ability. Inspired by CLIP, related technologies have tried to use text descriptions as supervisory signals and directly introduce them into FER tasks and AUD tasks, but the direct introduction is not effective and has certain defects.

[0039] Specifically, for FE, the "A photo of..." template proposed in CLIP is usually used to construct text descriptions, but this design is too rigid, and the vocabulary describing the expression category cannot accurately describe the facial expressions and their differences contained in the facial image; and for AU, multiple AU text descriptions are usually defined based on the original definition of each AU in FACS, and these AU text descriptions usually contain relatively absolute degree adverbs or adjectives about muscle movements, which is not flexible enough to deal with the different activation intensities of AUs in facial images. These problems limit the potential of text descriptions as effective supervisory signals.

[0040] In view of this, the embodiments of the present application provide a data augmentation method, a facial emotion recognition method and a device, aiming to improve the zero-sample capability and generalization capability of the neural network model for FER tasks and AUD tasks through the data augmentation method, and to realize the joint training of FER tasks and AUD tasks and the simultaneous recognition of FE and AU through the facial emotion recognition method.

[0041] The data augmentation method and facial emotion recognition method provided in the embodiments of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0042] First, the implementation steps of the data augmentation method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0043] Reference Figure 2 The data augmentation method provided in the embodiment of the present application mainly includes the following steps S101-S104:

[0044] S101, obtaining a plurality of facial sample images and a plurality of facial unlabeled images, each facial sample image being labeled with a corresponding facial unit label and a facial expression label;

[0045] S102, using a plurality of facial sample images to train a pre-built basic model to obtain a first neural network model and a confidence threshold set, wherein the confidence threshold set includes a confidence threshold of each facial unit label and a confidence threshold of each facial expression label;

[0046] S103, inputting a plurality of face unlabeled images into a first neural network model to obtain a plurality of face pseudo-labeled images, and selecting a plurality of face augmented images from the plurality of face pseudo-labeled images according to a confidence threshold set, wherein each face pseudo-labeled image is annotated with a corresponding face unit pseudo-label and a facial expression pseudo-label;

[0047] S104, obtaining a facial data set according to the plurality of facial augmented images, the plurality of facial sample images and the text sample set, wherein the text sample set includes a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions.

[0048] In the embodiment of the present application, first, a plurality of facial sample images and a plurality of facial unlabeled images are obtained, each of which is annotated with a corresponding FE label and AU label; then, a pre-constructed basic model is trained using the plurality of facial sample images to obtain a first neural network model and a confidence threshold set, the first neural network model having a basic ability to identify FE and AU, the confidence threshold set including a confidence threshold of each AU label and a confidence threshold of each FE label; then, the plurality of facial unlabeled images are input into the first neural network model to obtain a plurality of facial pseudo-labeled images, each of which has a confidence threshold of each AU label. The images are annotated with corresponding FE pseudo-labels and AU pseudo-labels, thereby realizing the generation of pseudo-labels, and then multiple face augmented images with high credibility are screened out from several face pseudo-label images according to the confidence threshold set; finally, the face sample images annotated with FE labels and AU labels and the face augmented images annotated with FE pseudo-labels and AU pseudo-labels are mixed, and combined with the text sample set to construct a face dataset. The face dataset is used to train neural network models for FER tasks and AUD tasks. The text sample set mainly includes multiple masked text samples associated with AU and multiple text samples associated with FE.

[0049] The embodiment of the present application utilizes large-scale facial unlabeled images combined with pseudo-label technology to generate a large number of facial pseudo-label images, which are all annotated with corresponding FE pseudo-labels and AU pseudo-labels, and screens these facial pseudo-label images based on category-specific confidence thresholds to obtain multiple high-confidence facial augmented images to ensure the high credibility of pseudo-labels and reduce the noise of pseudo-labels, and then constructs a facial dataset for training through facial sample images annotated with FE labels and AU labels, facial augmented images annotated with FE pseudo-labels and AU pseudo-labels, and text samples used to describe AU and FE, fully considering the intrinsic correlation between different activation intensities of AU and the correlation between FE and AU. The embodiment of the present application effectively improves the generalization ability and recognition performance of the neural network model for FER tasks and AUD tasks.

[0050] In the above step S101, a number of facial sample images are obtained, and the facial sample images are used to train the basic model. Each facial sample image is annotated with a corresponding AU label and FE label, and a number of facial unlabeled images are obtained. The facial unlabeled images are used for data augmentation, and each facial unlabeled image is not annotated with a corresponding AU label and FE label.

[0051] The obtaining of the plurality of facial sample images and the plurality of facial unlabeled images may include obtaining the plurality of facial sample images through an Aff-Wild2 dataset, but is not limited thereto.

[0052] The obtaining of the plurality of facial sample images and the plurality of facial unlabeled images may include obtaining the plurality of facial unlabeled images through a VGGFace2 dataset, an IMDB-WIKI dataset, and a CelebA dataset, but is not limited thereto.

[0053] In the above step S102, a pre-constructed basic model is trained using a plurality of facial sample images to obtain a first neural network model, and a confidence threshold set is calculated based on the first neural network model and the plurality of facial sample images.

[0054] The first neural network model mentioned above has the basic ability to identify AU and FE.

[0055] The above confidence threshold set may include a confidence threshold for each AU label and a confidence threshold for each FE label, and the confidence threshold is used to screen out a plurality of high-confidence face augmented images from the face pseudo-label images generated in subsequent steps.

[0056] In the above step S103, a number of facial unlabeled images are input into the first neural network model to obtain a number of facial pseudo-label images, thereby realizing the generation of pseudo-labels. These facial pseudo-label images are annotated with corresponding FE pseudo-labels and AU pseudo-labels. The FE pseudo-labels and AU pseudo-labels can be understood as the recognition results of the facial pseudo-label images by the first neural network model. Then, a plurality of facial augmented images are screened out from the plurality of facial pseudo-label images according to the confidence threshold set to ensure the credibility of the FE pseudo-labels and the AU pseudo-labels and reduce the noise of the FE pseudo-labels and the AU pseudo-labels.

[0057] In the above step S104, the facial sample images annotated with FE labels and AU labels and the facial augmented images annotated with FE pseudo labels and AU pseudo labels are mixed and combined with the text sample set to construct a facial dataset.

[0058] The above-mentioned facial data set is used to train a neural network model for FER tasks and AUD tasks, and the neural network model refers to the second neural network model in the facial emotion recognition method provided in the embodiment of the present application.

[0059] The text sample set may include a plurality of masked text samples associated with AUs and a plurality of text samples associated with FEs.

[0060] The following will be combined with the attached Figure 3 The above steps are further explained.

[0061] Reference Figure 3 , stage one indicates the above steps S101-S102, FE classifier refers to facial expression classifier, AU classifier refers to facial unit classifier, labeled image refers to facial sample image, image encoder, FE classifier and AU classifier of stage one can constitute a basic model (but the basic model is not limited to these components), FE prediction value refers to the activation probability of FE label or the prediction probability of FE prediction label, AU prediction value refers to the activation probability of AU label or the prediction probability of AU prediction label, true value refers to the true value of FE label or the true value of AU label, FE-specific confidence threshold refers to the confidence threshold of FE label, AU-specific confidence threshold refers to the confidence threshold of AU label; stage two indicates the above step S103, unlabeled image refers to facial unlabeled image, image encoder, FE classifier and AU classifier of stage two can constitute a first neural network model (but the first neural network model may not be limited to these components).

[0062] In some embodiments, after obtaining the plurality of facial sample images and the plurality of facial unlabeled images, the data augmentation method may further include at least one of the following:

[0063] Perform similarity transformation on several facial sample images and several facial unlabeled images;

[0064] Resize several facial sample images and several facial unlabeled images;

[0065] Several facial sample images and several facial unlabeled images are enhanced.

[0066] In this embodiment, by performing at least one of similarity transformation, size adjustment and enhancement processing on a plurality of facial sample images and a plurality of facial unlabeled images, image preprocessing is achieved, which can improve the generalization ability of the neural network model.

[0067] The above-mentioned similarity transformation of several facial sample images and several facial unlabeled images may include performing at least one of in-plane rotation, uniform scaling and translation on each facial sample image and each facial unlabeled image through facial key points, so as to remove the interference of head posture while retaining facial shape and facial expression information, but is not limited to this.

[0068] The above-mentioned resizing of the plurality of facial sample images and the plurality of facial unlabeled images may include resizing each facial sample image to , resize each unlabeled face image to .

[0069] The aforementioned enhancement processing of the plurality of facial sample images and the plurality of facial unlabeled images may include performing at least one of random cropping, random horizontal flipping, and random color jittering on each facial sample image and each facial unlabeled image, wherein random cropping refers to randomly cropping the facial sample image into The random color jittering refers to performing random color jittering on the facial sample image according to contrast and brightness, but is not limited to this.

[0070] In some embodiments, the above-mentioned training of a pre-built basic model using a plurality of facial sample images to obtain a first neural network model and a confidence threshold set may include:

[0071] Use several facial sample images to perform forward propagation on the pre-built basic model to obtain the activation probability of each facial unit label and the activation probability of each facial expression label;

[0072] According to the activation probability of each facial unit label and the activation probability of each facial expression label, the loss function value of the basic model is obtained;

[0073] The network parameters of the basic model are optimized using the loss function value of the basic model;

[0074] When the basic model does not meet the preset end condition, return to the step of using several facial sample images to perform forward propagation on the pre-built basic model until the basic model meets the end condition, and output the trained basic model as the first neural network model.

[0075] In this embodiment, in order to realize the generation of pseudo labels, it is crucial to generate a first neural network model with the ability to identify FE and AU. To this end, first, a number of facial sample images are input into a pre-built basic model, and the basic model is forward propagated using these facial sample images to train the basic model, thereby obtaining the activation probability of each FE label and the activation probability of each AU label; then, the activation probability of each FE label and the activation probability of each AU label are used to calculate the loss function value of the basic model, and the network parameters of the basic model are optimized using the loss function value; then, it is determined whether the basic model meets the preset end condition, and if so, the trained basic model is output as the first neural network model, otherwise it returns to the step of forward propagating the pre-built basic model using a number of facial sample images to achieve cyclic training.

[0076] The above-mentioned optimization processing of the network parameters of the basic model using the loss function value of the basic model may include updating the network parameters of the basic model by using the loss function value of the basic model in combination with gradient backpropagation, but is not limited to this.

[0077] The above-mentioned preset end condition can be set according to the actual situation, and the embodiment of the present application does not specifically limit this. For example, the above-mentioned end condition can be that the number of training times of the basic model is greater than the preset iteration threshold; for another example, the above-mentioned end condition can be that the test accuracy of the basic model reaches the preset accuracy threshold, but it is not limited to this.

[0078] In some embodiments, the basic model may include an image encoder and a multi-cascade classifier connected in sequence, the multi-cascade classifier may include a facial expression classifier and a facial unit classifier connected in parallel, the facial expression classifier may include a plurality of first linear layers, the plurality of first linear layers correspond one-to-one to a plurality of FE labels, the facial unit classifier may include a plurality of second linear layers, the plurality of second linear layers correspond one-to-one to a plurality of AU labels; the method of forward propagating the pre-constructed basic model using a plurality of facial sample images to obtain the activation probability of each facial unit label and the activation probability of each facial expression label may include:

[0079] Encode several facial sample images through an image encoder to obtain several visual embeddings;

[0080] Processing a plurality of visual embeddings by combining each first linear layer with a preset first activation function to obtain an activation probability of a facial expression label corresponding to each first linear layer;

[0081] By processing a plurality of visual embeddings through each second linear layer in combination with a preset second activation function, an activation probability of a facial unit label corresponding to each second linear layer is obtained.

[0082] In this embodiment, in order to realize the synchronous recognition of FE and AU, a basic model is constructed and a plurality of facial sample images are used to perform forward propagation training on the pre-constructed basic model. The basic model mainly includes an image encoder, a FE classifier and an AU classifier. The FE classifier includes The AU classifier consists of a first linear layer in parallel. For the forward propagation training, first, several sizes of The facial sample image is input into the image encoder, and the image encoder performs encoding processing to obtain a number of visual embeddings with a length of 512; then, the several visual embeddings are respectively input into the FE classifier and the AU classifier, and the several visual embeddings are processed by each first linear layer in the FE classifier in combination with a preset first activation function to obtain the activation probability of the FE label corresponding to each first linear layer, and the several visual embeddings are processed by each second linear layer in the AU classifier in combination with a preset second activation function to obtain the activation probability of the AU label corresponding to each second linear layer.

[0083] The first activation function may be a Softmax activation function, but is not limited thereto.

[0084] The second activation function may be a Sigmoid activation function, but is not limited thereto.

[0085] In some embodiments, obtaining the loss function value of the basic model according to the activation probability of each facial unit label and the activation probability of each facial expression label may include:

[0086] According to the total number of facial expression labels and the activation probability and true value of each facial expression label, the cross entropy loss function value is obtained;

[0087] According to the total number of facial unit labels and the activation probability, true value and occurrence frequency of each facial unit label, the asymmetric loss function value and the dice loss function value are obtained;

[0088] According to the cross entropy loss function value, the asymmetric loss function value and the dice loss function value, the loss function value of the basic model is obtained.

[0089] In this embodiment, for the iterative training of the basic model, a cross entropy loss function is introduced to supervise the basic model's prediction of FE, and a weighted asymmetric loss function and a weighted multi-label dice loss function are introduced to supervise the basic model's prediction of AU.

[0090] Specifically, FER is often regarded as a multi-classification problem, so this embodiment introduces a cross entropy loss function to supervise the prediction of FE. More specifically, the cross entropy loss function value is calculated based on the total number of FE labels and the activation probability and true value of each FE label. The cross entropy loss function value satisfies the following formula (1):

[0091] (1);

[0092] In formula (1), represents the cross entropy loss function value, Indicates The true value of the FE label, Indicates The activation probability of FE labels, Indicates the total number of FE labels.

[0093] AUD is usually regarded as a binary multi-label classification problem, and related technologies often use a weighted multi-label cross entropy loss function to supervise the prediction of AU. However, the prediction difficulty is different for different AUs, and the common weighted multi-label cross entropy loss function does not take into account the differences between different prediction difficulties. Therefore, this embodiment introduces a weighted asymmetric loss function, focusing on activated AUs and AUs that are difficult to correctly identify and are not activated. More specifically, the asymmetric loss function value is calculated based on the total number of AU labels and the activation probability, true value and frequency of occurrence of each AU label. The asymmetric loss function value satisfies the following formula (2):

[0094] (2);

[0095] In formula (2), represents the asymmetric loss function value, Indicates The true value of the AU label, Indicates The activation probability of an AU label, Indicates the total number of AU labels; For the The mitigation weight of the AU label is mainly used to alleviate the class imbalance problem in the dataset. The mitigation weight of an AU label can be expressed as , For the The frequency of occurrence of AU tags, , Represents the sum of the occurrence frequencies of all AU labels.

[0096] The occurrence frequency of the above AU label can be calculated according to actual conditions, and this embodiment does not specifically limit this.

[0097] Considering that the AUD task is strongly biased towards non-occurrence, this embodiment introduces a weighted multi-label dice loss. More specifically, the dice loss function value is calculated based on the total number of AU labels and the activation probability, true value and occurrence frequency of each AU label. The dice loss function value satisfies the following formula (3):

[0098] (3);

[0099] In formula (3), represents the dice loss function value, is the default smoothing term.

[0100] The above-mentioned smoothing item can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above-mentioned smoothing item can be 1, but is not limited thereto.

[0101] The above-mentioned loss function value of the basic model is obtained according to the cross entropy loss function value, the asymmetric loss function value and the dice loss function value, which may include calculating the sum of the cross entropy loss function value, the asymmetric loss function value and the dice loss function value as the loss function value of the basic model. The loss function value of the basic model satisfies the following formula (4):

[0102] (4);

[0103] In formula (4), Represents the loss function value of the basic model.

[0104] In some implementations, the method of training a pre-built basic model using a plurality of facial sample images to obtain a first neural network model and a confidence threshold set may further include:

[0105] Inputting a plurality of facial sample images into the first neural network model, obtaining a facial expression prediction label and its prediction probability for each facial sample image and a facial unit prediction label and its positive sample prediction probability and negative sample prediction probability for each facial sample image, wherein the positive sample prediction probability is used to characterize the prediction probability that the facial unit prediction label is a positive sample, and the negative sample prediction probability is used to characterize the prediction probability that the facial unit prediction label is a negative sample;

[0106] For each facial expression label, the confidence threshold of the facial expression label is obtained according to the facial expression prediction label and its prediction probability of each facial sample image;

[0107] For each facial unit label, the positive sample confidence threshold of the facial unit label is obtained according to the facial unit prediction label of each facial sample image and its positive sample prediction probability, and the negative sample confidence threshold of the facial unit label is obtained according to the facial unit prediction label of each facial sample image and its negative sample prediction probability. Then, the positive sample confidence threshold and the negative sample confidence threshold of the facial unit label are used as the confidence threshold of the facial unit label.

[0108] In this embodiment, in order to facilitate the screening of FE pseudo-labels and AU pseudo-labels to ensure the credibility of FE pseudo-labels and AU pseudo-labels, a confidence threshold specific to each FE category and a confidence threshold specific to each AU category are provided. Specifically, first, a number of facial sample images are input into the first neural network model to obtain the FE prediction label and its prediction probability and the AU prediction label and its positive sample prediction probability and negative sample prediction probability of the first neural network model on each facial sample image; then, based on the above data, the confidence threshold of each FE label and the positive sample confidence threshold and negative sample confidence threshold of each AU label are calculated respectively.

[0109] More specifically, for each FE label, the confidence threshold of the FE label is obtained according to the FE prediction label and its prediction probability of the first neural network model on each facial sample image. By traversing each FE label, the confidence threshold of all FE labels can be obtained. The confidence threshold of a FE label satisfies the following formula (5):

[0110] (5);

[0111] In formula (5), Indicates The confidence threshold of each FE label; Indicates The predicted probability of the FE predicted label of the face sample image; Indicates the preset zoom factor; Indicates that the first neural network model correctly predicts that the FE prediction label is The number of face sample images with FE labels; Represents the total number of facial sample images; Indicates Predictors for face sample images.

[0112] The above The prediction factor of a facial sample image satisfies the following formula (6):

[0113] (6);

[0114] In formula (6), Indicates The FE predicted label of the facial sample image, that is, the first neural network model predicts the The prediction results obtained when performing FER on facial sample images; Indicates FE label; when the The FE predicted label of the face sample image is FE label season The prediction factor of the face sample image is 1, otherwise the prediction factor of the The prediction factor of the face sample images is 0.

[0115] The above scaling factor can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the scaling factor can be 0.95, but is not limited thereto.

[0116] For each AU label, the positive sample confidence threshold of the AU label is obtained according to the AU prediction label of each facial sample image and its positive sample prediction probability, and the negative sample confidence threshold of the AU label is obtained according to the AU prediction label of each facial sample image and its negative sample prediction probability, and the negative sample confidence threshold and positive sample confidence threshold of the AU label are used as the confidence threshold of the AU label.

[0117] No. The confidence threshold of the positive sample of an AU label satisfies the following formula (7):

[0118] (7);

[0119] In formula (7), Indicates The confidence threshold of positive samples of AU labels, Indicates the preset scaling factor, Indicates that the first neural network model correctly predicts the The number of facial sample images with AU labels as positive samples; It means that when the first neural network model When AUD is performed on facial sample images, the first neural network model correctly predicts the The predicted probability of the AU label being a positive sample, that is, The positive sample prediction probability of the AU prediction label of the face sample image.

[0120] No. The negative sample confidence threshold of an AU label satisfies the following formula (8):

[0121] (8);

[0122] In formula (8), Indicates The negative sample confidence threshold of AU labels, Indicates the preset scaling factor, Indicates that the first neural network model correctly predicts the The number of facial sample images with AU labels as negative samples; It means that when the first neural network model When AUD is performed on facial sample images, the first neural network model correctly predicts the The predicted probability that the AU label is a negative sample, that is, The negative sample prediction probability of the AU prediction label of the face sample images.

[0123] In some embodiments, the step of inputting a plurality of unlabeled facial images into a first neural network model to obtain a plurality of pseudo-labeled facial images, and selecting a plurality of augmented facial images from the pseudo-labeled facial images according to a confidence threshold set may include:

[0124] Inputting a plurality of unlabeled facial images into a first neural network model, obtaining a plurality of pseudo-labeled facial images and pseudo-label data of each pseudo-labeled facial image, wherein the pseudo-label data includes a facial unit pseudo-label, a predicted probability of the facial unit pseudo-label, a facial expression pseudo-label, and a predicted probability of the facial expression pseudo-label;

[0125] A face pseudo-label image whose pseudo-label data meets the augmentation condition is selected from a plurality of face pseudo-label images as a face augmented image.

[0126] In this implementation, after the confidence threshold set is calculated, first, a number of facial unlabeled images are input into the first neural network model, and a number of facial pseudo-label images and pseudo-label data of each facial pseudo-label image are obtained through the first neural network model. The pseudo-label data may include FE pseudo-labels, predicted probabilities of FE pseudo-labels, AU pseudo-labels, and predicted probabilities of AU pseudo-labels; then, based on the confidence threshold of each FE label and the positive sample confidence threshold and negative sample confidence threshold of each AU label, a number of facial pseudo-label images are screened, and facial pseudo-label images that meet the amplification conditions are retained as facial augmented images, thereby realizing the screening of pseudo-labels, ensuring the credibility of pseudo-labels and reducing the noise of pseudo-labels.

[0127] The above amplification conditions may include the following sub-conditions (1)-(2):

[0128] (1) The predicted probability of the FE pseudo-label of the face pseudo-label image is greater than the confidence threshold of the FE label of the same type as the FE pseudo-label. That is, for a certain face pseudo-label image, determine the type of the FE pseudo-label of the face pseudo-label image, obtain the confidence threshold of the FE label of the same type as the FE pseudo-label, and determine whether the predicted probability of the FE pseudo-label is greater than the confidence threshold of the FE label. If so, it means that the FE pseudo-label is credible, otherwise, it means that the FE pseudo-label is not credible.

[0129] (2) The AU pseudo-label of the face pseudo-label image is a positive sample and the predicted probability of the AU pseudo-label is greater than the positive sample confidence threshold of the AU label of the same type as the AU pseudo-label; or, the AU pseudo-label of the face pseudo-label image is a negative sample and the predicted probability of the AU pseudo-label is less than the negative sample confidence threshold of the AU label of the same type as the AU pseudo-label. That is, for a certain face pseudo-label image, determine the type of the AU pseudo-label of the face pseudo-label image and judge whether the AU pseudo-label is a positive sample or a negative sample. If the AU pseudo-label is a positive sample, obtain the same type as the AU pseudo-label. The positive sample confidence threshold of the AU label is obtained, and it is determined whether the predicted probability of the AU pseudo-label is greater than the positive sample confidence threshold of the AU label. If so, it means that the AU pseudo-label is a credible positive sample, otherwise, it means that the AU pseudo-label is an unreliable positive sample; if the AU pseudo-label is a negative sample, the negative sample confidence threshold of the AU label of the same type as the AU pseudo-label is obtained, and it is determined whether the predicted probability of the AU pseudo-label is greater than the negative sample confidence threshold of the AU label. If so, it means that the AU pseudo-label is a credible negative sample, otherwise, it means that the AU pseudo-label is an unreliable negative sample.

[0130] The above-mentioned face augmented image refers to a face pseudo-label image whose FE pseudo-label is credible and whose AU pseudo-label is a credible positive sample or a credible negative sample.

[0131] In some implementations, the data augmentation method may further include:

[0132] Use CLIP to construct multiple text samples associated with facial expressions;

[0133] Constructing multiple text samples associated with facial units using FACS and performing mask processing to obtain multiple masked text samples associated with facial units;

[0134] A text sample set is constructed by a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions.

[0135] In this embodiment, in order to address the problem in the prior art that text samples associated with facial units and facial expressions limit the potential of text descriptions as effective supervisory signals, this embodiment uses CLIP to construct multiple text samples associated with FEs to accurately describe the FEs and their differences contained in facial images; and uses FACS to construct multiple text samples associated with AUs and mask the more absolute degree adverbs or adjectives about muscle movements therein, thereby obtaining multiple masked text samples associated with AUs, so that the neural network model can more flexibly process AUs with different activation intensities in diverse facial images; thereafter, a text sample set is constructed using multiple masked text samples associated with AUs and multiple text samples associated with FEs.

[0136] The text sample set may include a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions.

[0137] The above-mentioned construction of multiple text samples associated with facial expressions using CLIP may include using an "A photo of ..." template in CLIP to construct multiple text samples associated with facial expressions.

[0138] Referring to Table 1, Table 1 is an example table of text samples associated with multiple FEs. In Table 1, FE represents FE label categories. The present embodiment is provided with eight FE label categories, namely, Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise and Other. FEdescription represents text samples of each FE label category, wherein, when the FE label category is neutral, its text sample is “A photo of a neutral face.”; when the FE label category is anger, its text sample is “A photo of anangry face.”; when the FE label category is disgust, its text sample is “A photo of a disgusted face.”; when the FE label category is fear, its text sample is “A photo of a fearful face.”; when the FE label category is happiness, its text sample is “A photo of a happy face.”; when the FE label category is sadness, its text sample is “Aphoto of a sad face.”; when the FE label category is surprise, its text sample is “A photo of a surprisedface."; when the FE label category is other, its text sample is "A photo of a face with an uncommonexpression.".

[0139] Table 1: Example table of text samples associated with FE

[0140]

[0141] The above-mentioned method of constructing multiple text samples associated with facial units using FACS and performing mask processing to obtain multiple text samples that have been masked and associated with facial units may include constructing multiple text samples associated with facial units based on FACS, and using at least one preset learnable vector to replace adverbs and / or adjectives used to describe the degree of facial movement in each text sample associated with the facial unit to obtain multiple text samples that have been masked and associated with the facial unit.

[0142] Referring to Table 2, Table 2 is an example table of multiple text samples that have been masked and associated with AUs. In Table 2, AUindex represents the number of the AU label, and AU name represents the name of the AU label. In this embodiment, 15 AUs are selected, namely AU1 (Inner brow rasier), AU2 (Outer brow raiser), AU4 (Browlowerer), AU6 (Cheek raiser), AU7 (Lid tightener), AU9 (Nose wrinkler), AU10 (Upper lip raiser), AU12 (Lip corner puller), AU14 (Dimpler), AU15 (Lip concer depressor), AU17 (Chinraiser), AU23 (Lip tightener), AU24 (Lip pressor), AU25 (Lips part, lips parted) and AU26 (Jaw Drop, jaw relaxed); AU description represents a text sample of each AU label; learnable vectors may include, but are not limited to, “slightly”, “pronounced”, “markedly”, “slight”, “severely”, “extremely”, “severe”, “extreme”, “maximally”, “clearly”, and “about”.

[0143] For example, the text description of AU7 is “The lower eyelid is raised [markedly] and straightened [slightly], causing [slight] bulging, and the nar-rowing of the eye aperture is marked to [pronounced]. ”, “markedly”, “slightly”, “slight”, and “pronounced” are used to mask the adverbs and adjectives used to describe the degree of the facial movement of the eyelid tightening.

[0144] Table 2: Example table of text samples associated with AUs

[0145]

[0146] Secondly, the implementation steps of the facial emotion recognition method provided in the embodiment of the present application will be explained in detail below in conjunction with the accompanying drawings.

[0147] Reference Figure 4 The facial emotion recognition method provided in the embodiment of the present application mainly includes the following steps S201-S202:

[0148] S201, obtaining a facial image to be tested and text data as inputs of a second neural network model, wherein the text data includes text data associated with facial units after mask processing and text data associated with facial expressions, and the second neural network model is obtained by training a facial data set, and the facial data set is obtained by the above-mentioned data augmentation method;

[0149] S202, performing facial recognition on the facial image to be tested by using a second neural network model to obtain facial unit recognition results and facial expression recognition results.

[0150] In the embodiment of the present application, a facial image to be tested and text data are obtained, the text data includes text data associated with AU after mask processing and text data associated with FE, and the facial image to be tested and the text data are input into the second neural network model, and the facial image to be tested is subjected to AUD and FER through the second neural network model, thereby obtaining FE recognition results and AU recognition results, wherein the second neural network model is a neural network model obtained by training a facial data set, and the facial data set is obtained by the above-mentioned data augmentation method. In this way, joint training of FER tasks and AUD tasks and synchronous recognition of FE and AU are achieved, the efficiency of AU detection and the efficiency of FE recognition are improved, and thus the accuracy and efficiency of facial emotion recognition are improved.

[0151] The above text data may include text data associated with AU after mask processing and text data associated with FE, which corresponds to the text sample set in the above data augmentation method embodiment.

[0152] The above-mentioned second neural network model may include a basic model, a completion module and a perception module. The basic model is used to encode and linearly process the facial image to be tested to obtain facial unit visual features and facial expression visual features; the completion module is used to encode text data associated with the facial unit to obtain facial unit encoding features, and obtain facial unit text features based on the facial unit encoding features and facial unit visual features, and encode text data associated with facial expressions to obtain facial expression text features; the perception module is used to obtain facial unit recognition results based on facial unit visual features and facial unit text features, and obtain facial expression recognition results based on facial unit visual features, facial expression visual features, facial unit text features and facial expression text features.

[0153] The above basic model is consistent with the basic model in the above data augmentation method embodiment.

[0154] In the above basic model, the facial image to be tested is encoded by an image encoder to obtain a visual embedding, the visual embedding is processed by a FE classifier to obtain a FE visual feature, and the visual embedding is processed by an AU classifier to obtain an AU visual feature.

[0155] In the above completion module, text data associated with AU is text encoded to obtain AU encoding features, and AU text features are obtained based on the AU encoding features and the AU visual features output by the basic model. The purpose is to supplement the masked information in the text data associated with AU through multimodal interaction. At the same time, text data associated with FE is text encoded to obtain FE text features.

[0156] In the above perception module, AUD is performed based on the AU visual features output by the AU classifier and the AU text features output by the completion module to obtain the AU recognition result. At the same time, FER is performed based on the AU visual features and FE visual features output by the basic model and the AU text features and FE text features output by the completion module to obtain the FE recognition result, thereby realizing the joint training of the FER task and the AUD task and the synchronous recognition of the FE and AU.

[0157] The above-mentioned embodiments of the data augmentation method are applicable to the embodiments of the present facial emotion recognition method.

[0158] The following will be combined Figure 3 The functions of each structure of the second neural network model are further explained.

[0159] Reference Figure 3 , Figure 3 The stage three indicates the above-mentioned facial emotion recognition method, the image encoder, FE classifier and AU classifier in the stage three constitute the basic model, FE visual embedding refers to FE visual features, AU visual embedding refers to AU visual features, dynamic AU text embedding refers to AU text features, FE text embedding refers to FE text features, AU prediction value refers to AU recognition result, FE prediction value refers to FE recognition result, FE text description and AU text description refer to text data.

[0160] In some embodiments, the completion module may include a text encoder, which is used to encode text data associated with FE to obtain FE text features, and encode text data associated with AU to obtain AU encoding features.

[0161] In some embodiments, the completion module may further include a visually perceived text completion module, where the visually perceived text completion module is used to obtain AU text features according to AU coding features and AU visual features;

[0162] The facial unit text features obtained according to the facial unit coding features and the facial unit visual features may include:

[0163] The facial unit encoding features are used as the query matrix, and the facial unit visual features are used as the key matrix and value matrix. The facial unit encoding features and facial unit visual features are processed through the attention mechanism to obtain the facial unit text features.

[0164] In this embodiment, the AU coding features output by the text encoder and the AU visual features output by the AU classifier are input into the visually perceived text completion module, and the visually perceived text completion module obtains the AU text features through cross-modal interaction, so as to supplement the masked information in the text data associated with the AU with the help of the visual features, and capture more comprehensive and rich AU detail information, thereby improving the feature extraction capability of the second neural network model. Specifically, the visually perceived text completion module uses the AU coding features as the query matrix, and the AU visual features as the key matrix and the value matrix, and performs cross-modal interaction processing on the AU coding features and the AU visual features through the attention mechanism to obtain the AU text features.

[0165] The above attention mechanism satisfies the following formula (9):

[0166] (9);

[0167] In formula (9), represents the output of the attention mechanism, represents the feature dimension; Q represents the query matrix, K represents the key matrix, and V represents the value matrix. ; Represents query, key and value respectively; They represent the learnable weights of the linear layer respectively.

[0168] The above facial unit text features satisfy the following formula (10):

[0169] (10);

[0170] In formula (10), represents AU text features; Attn() represents the attention mechanism as shown in the above formula (9). For the visually perceived text completion module, the feature dimension of its attention mechanism refers to the feature dimension of AU text features or AU visual features; Indicates AU encoding features; represents AU visual features; the AU encoding features are taken as Q, and the AU visual features are taken as K and V, aiming to encourage the masked AU encoding features to perceive the information of AU visual features, and then adaptively complete the degree information related to the intensity of facial action motion.

[0171] In some embodiments, the perception module may include a first AU association modeling module, a second AU association modeling module, and a first prediction module, wherein the first AU association modeling module is used to use AU visual features as a query matrix, a key matrix, and a value matrix, and process the AU visual features through an attention mechanism to obtain a first refined visual feature; the second AU association modeling module is used to use AU text features as a query matrix, a key matrix, and a value matrix, and process the AU text features through an attention mechanism to obtain a first refined text feature; the first prediction module is used to obtain an AU recognition result according to the first refined visual feature and the first refined text feature;

[0172] The facial unit recognition result obtained according to the facial unit visual features and the facial unit text features may include:

[0173] Taking the facial unit visual features as the query matrix, key matrix and value matrix, the facial unit visual features are processed through the attention mechanism to obtain the first refined visual features;

[0174] Taking the facial unit text features as the query matrix, key matrix and value matrix, the facial unit text features are processed through the attention mechanism to obtain the first refined text features;

[0175] A facial unit recognition result is obtained according to the first refined visual feature and the first refined text feature.

[0176] In this embodiment, the AU visual features output by the AU classifier are input into the first AU association modeling module, which uses the AU visual features as the query matrix, key matrix and value matrix, and processes the facial unit visual features through the attention mechanism to obtain the first refined visual features, which aims to mine and utilize the intrinsic correlation between different AUs to optimize the visual representation of AUs; the AU text features output by the visually perceived text completion module are input into the second AU association modeling module, which uses the AU text features as the query matrix, key matrix and value matrix, and processes the AU text features through the attention mechanism to obtain the first refined text features, which aims to mine and utilize the intrinsic correlation between different AUs to optimize the text representation of AUs; then, the first refined visual features and the first refined text features are input into the first prediction module, and the AU recognition results are obtained through the first prediction module. In this way, more detailed visual representations and text representations of AUs can be captured, thereby improving the feature extraction capability of the second neural network model and improving the accuracy of the second neural network model on the AUD task.

[0177] The first refined visual feature satisfies the following formula (11):

[0178] (11);

[0179] In formula (11), represents the first refined visual feature; Attn() represents the attention mechanism as shown in the above formula (9). For the first AU association modeling module, the feature dimension of its attention mechanism refers to the feature dimension of the AU visual feature; Represents AU visual features.

[0180] The first refined text feature satisfies the following formula (12):

[0181] (12);

[0182] In formula (12), represents the first refined text feature; Attn() represents the attention mechanism as shown in the above formula (9). For the second AU association modeling module, the feature dimension of its attention mechanism refers to the feature dimension of the AU text feature; Represents AU text features.

[0183] The above-mentioned obtaining the facial unit recognition result based on the first refined visual feature and the first refined text feature may include processing the product of the transposition of the first refined visual feature and the first refined text feature through a preset third activation function to obtain the facial unit recognition result.

[0184] The third activation function may be a Sigmoid activation function, but is not limited thereto.

[0185] The above facial unit recognition results satisfy the following formula (13):

[0186] (13);

[0187] In formula (13), Indicates the AU recognition result, Represents the Sigmoid activation function; represents the first refined text feature; Represents the first refined visual feature.

[0188] In some embodiments, the perception module may further include a first AU perception refinement module, a second AU perception refinement module, and a second prediction module, wherein the first AU perception refinement module is used to use the FE visual features as a query matrix, the first refined visual features as a key matrix and a value matrix, and process the FE visual features and the first refined visual features through an attention mechanism to obtain second refined visual features; the second AU perception refinement module is used to use the FE text features as a query matrix, the first refined text features as a key matrix and a value matrix, and process the FE text features and the first refined text features through an attention mechanism to obtain second refined text features; the second prediction module is used to obtain a FE recognition result according to the second refined visual features and the second refined text features;

[0189] The facial expression recognition result obtained according to the facial unit visual features, the facial expression visual features, the facial unit text features and the facial expression text features may include:

[0190] Taking the facial expression visual features as the query matrix and the first refined visual features as the key matrix and the value matrix, the facial expression visual features and the first refined visual features are processed through the attention mechanism to obtain the second refined visual features;

[0191] Taking the facial expression text features as the query matrix and the first refined text features as the key matrix and the value matrix, the facial expression text features and the first refined text features are processed through the attention mechanism to obtain the second refined text features;

[0192] A facial expression recognition result is obtained according to the second refined visual features and the second refined text features.

[0193] In this embodiment, the first refined visual features output by the first AU association modeling module and the FE visual features output by the FE classifier are input into the first AU perception refinement module. The first AU perception refinement module uses the first refined visual features as the key matrix and the value matrix, and the FE visual features as the query matrix, and combines the attention mechanism to obtain the second refined visual features, aiming to further refine the visual representation of FE by utilizing the potential correlation between FE and AU; the first refined text features output by the second AU association modeling module and the FE text features output by the text encoder are input into the second AU perception refinement module. The second AU perception refinement module uses the first refined text features as the value matrix and the key matrix, and the FE text features as the query matrix, and combines the attention mechanism to obtain the second refined text features, aiming to further refine the text representation of FE by utilizing the potential correlation between FE and AU; then, the second refined visual features and the second refined text features are input into the second prediction module, and the FE recognition result is obtained through the second prediction module. In this way, more detailed visual representation and text representation of FE can be captured, thereby improving the feature extraction capability of the second neural network model and improving the accuracy of the second neural network model on the FER task.

[0194] The second refined visual feature satisfies the following formula (14):

[0195] (14);

[0196] In formula (14), represents the second refined visual feature; Attn() represents the attention mechanism as shown in the above formula (9); represents the first refined visual feature; Represents FE visual features.

[0197] The second refined text feature satisfies the following formula (15):

[0198] (15);

[0199] In formula (15), represents the second refined text feature; Attn() represents the attention mechanism as shown in the above formula (9); represents the first refined text feature; Represents FE text features.

[0200] Obtaining the facial expression recognition result based on the second refined visual feature and the second refined text feature may include processing the product of the transposition of the second refined visual feature and the second refined text feature through a preset fourth activation function to obtain the facial expression recognition result.

[0201] The fourth activation function may be a Softmax activation function, but is not limited thereto.

[0202] The above facial expression recognition results satisfy the following formula (16):

[0203] (16);

[0204] In formula (16), represents the FE recognition result, represents the Softmax activation function, represents the second refined text feature, Represents the second refined visual feature.

[0205] In some embodiments, the facial emotion recognition method may further include:

[0206] The network parameters of the second neural network model are updated according to the facial expression recognition results and the facial unit recognition results.

[0207] In this embodiment, in order to improve the performance of the second neural network model, a loss function as shown in formula (4) is designed, and the parameters of other parts of the second neural network model except the text encoder are updated and optimized by gradient backpropagation.

[0208] In addition, the embodiment of the present application also provides a data augmentation device, which mainly includes:

[0209] A first acquisition module is used to acquire a plurality of facial sample images and a plurality of facial unlabeled images, each facial sample image is annotated with a corresponding facial unit label and a facial expression label;

[0210] A first processing module is used to train a pre-built basic model using a plurality of facial sample images to obtain a first neural network model and a confidence threshold set, wherein the confidence threshold set includes a confidence threshold of each facial unit label and a confidence threshold of each facial expression label;

[0211] A second processing module is used to input a plurality of face unlabeled images into the first neural network model to obtain a plurality of face pseudo-labeled images, and select a plurality of face augmented images from the plurality of face pseudo-labeled images according to a confidence threshold set, wherein each face pseudo-labeled image is annotated with a corresponding face unit pseudo-label and a facial expression pseudo-label;

[0212] The third processing module is used to obtain a facial data set according to multiple facial augmented images, a plurality of facial sample images and a text sample set, wherein the text sample set includes multiple text samples that have been masked and associated with facial units and multiple text samples associated with facial expressions.

[0213] The data augmentation device executes the data augmentation method. For specific limitations of the data augmentation device, please refer to the limitations of the data augmentation method above, which will not be repeated here.

[0214] Each module in the above data augmentation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0215] Finally, the embodiment of the present application also provides a facial emotion recognition device, which mainly includes:

[0216] A second acquisition module is used to acquire a facial image to be tested and text data as inputs of a second neural network model, wherein the text data includes text data associated with facial units after mask processing and text data associated with facial expressions;

[0217] A fourth processing module is used to perform facial recognition on the facial image to be tested through the second neural network model to obtain a facial unit recognition result and a facial expression recognition result;

[0218] The second neural network model is obtained by training a facial data set, and the facial data set is obtained by the above-mentioned data augmentation device. The second neural network model includes:

[0219] The basic model is used to perform encoding processing and linear processing on the facial image to be tested to obtain facial unit visual features and facial expression visual features;

[0220] A completion module is used to encode the text data associated with the facial unit to obtain the facial unit encoding features, and obtain the facial unit text features according to the facial unit encoding features and the facial unit visual features, and encode the text data associated with the facial expression to obtain the facial expression text features;

[0221] The perception module is used to obtain a facial unit recognition result based on the facial unit visual features and the facial unit text features, and to obtain a facial expression recognition result based on the facial unit visual features, the facial expression visual features, the facial unit text features and the facial expression text features.

[0222] The above-mentioned facial emotion recognition device executes the above-mentioned facial emotion recognition method. The specific limitations of the facial emotion recognition device can be found in the above limitations on the facial emotion recognition method, which will not be repeated here.

[0223] Each module in the above-mentioned facial emotion recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0224] In order to facilitate understanding of the data augmentation method, facial emotion recognition method and device proposed in the embodiment of the present application, the embodiment of the present application will be illustrated by an example below. The specific process of the embodiment of the present application to realize the joint training of FER tasks and AUD tasks and the simultaneous recognition of FE and AU includes the following steps S301-S307.

[0225] S301, obtaining a number of facial sample images through the Aff-Wild2 dataset, obtaining a number of facial unlabeled images through the VGGFace2 dataset, the IMDB-WIKI dataset and the CelebA dataset, and performing similarity transformation, size adjustment and enhancement processing on the number of facial sample images and the number of facial unlabeled images.

[0226] S302, first, input a number of facial sample images into a pre-built basic model, use these facial sample images to perform forward propagation on the basic model to train the basic model, and then obtain the activation probability of each FE label and the activation probability of each AU label; then, according to the total number of FE labels and the activation probability and true value of each FE label, combined with the above formula (1), calculate the cross entropy loss function value; at the same time, according to the total number of AU labels and the activation probability, true value and occurrence frequency of each AU label, combined with formulas (2)-(3), calculate the asymmetric loss function value and the dice loss function value, and then calculate the sum of the cross entropy loss function value, the asymmetric loss function value and the dice loss function value as the loss function value of the basic model; then, use the loss function value of the basic model in combination with the gradient backpropagation method to update the network parameters of the basic model; determine whether the basic model meets the preset end condition, if so, output the trained basic model as the first neural network model, otherwise return to the step of performing forward propagation on the pre-built basic model using a number of facial sample images to achieve cyclic training. Among them, the basic model mainly includes image encoder, FE classifier and AU classifier. FE classifier includes The AU classifier consists of a first linear layer in parallel. The image encoder is used to encode several images of size The facial sample images are encoded to obtain several visual embeddings with a length of 512; the first linear layer is used to process the several visual embeddings to obtain the activation probability of the FE label corresponding to the first linear layer; the second linear layer is used to process the several visual embeddings to obtain the activation probability of the AU label corresponding to the second linear layer.

[0227] S303, for each FE label, according to the FE prediction label and its prediction probability of the first neural network model on each facial sample image, combined with the above formulas (5)-(6), the confidence threshold of the FE label is obtained; for each AU label, according to the AU prediction label of each facial sample image and its positive sample prediction probability, combined with the above formula (7), the positive sample confidence threshold of the AU label is obtained, and according to the AU prediction label of each facial sample image and its negative sample prediction probability, combined with formula (8), the negative sample confidence threshold of the AU label is obtained.

[0228] S304, inputting a plurality of facial unlabeled images into the first neural network model, obtaining a plurality of facial pseudo-labeled images and pseudo-label data of each facial pseudo-labeled image through the first neural network model, wherein the pseudo-label data may include FE pseudo-labels, predicted probabilities of FE pseudo-labels, AU pseudo-labels and predicted probabilities of AU pseudo-labels, and FE pseudo-labels and AU pseudo-labels may be understood as the recognition results of the facial unlabeled images by the first neural network model. For a certain facial pseudo-labeled image, determining the type of the FE pseudo-label of the facial pseudo-labeled image, obtaining the confidence threshold of the FE label of the same type as the FE pseudo-label, and determining whether the predicted probability of the FE pseudo-label is greater than the confidence threshold of the FE label, if so, indicating that the FE pseudo-label is credible, otherwise indicating that the FE pseudo-label is not credible. For a certain face pseudo-label image, determine the type of AU pseudo-label of the face pseudo-label image and judge whether the AU pseudo-label is a positive sample or a negative sample. If the AU pseudo-label is a positive sample, obtain the positive sample confidence threshold of the AU label of the same type as the AU pseudo-label, and judge whether the predicted probability of the AU pseudo-label is greater than the positive sample confidence threshold of the AU label. If so, it means that the AU pseudo-label is a credible positive sample, otherwise it means that the AU pseudo-label is an unreliable positive sample; if the AU pseudo-label is a negative sample, obtain the negative sample confidence threshold of the AU label of the same type as the AU pseudo-label, and judge whether the predicted probability of the AU pseudo-label is greater than the negative sample confidence threshold of the AU label. If so, it means that the AU pseudo-label is a credible negative sample, otherwise it means that the AU pseudo-label is an unreliable negative sample. Based on this, select the face pseudo-label image with a credible FE pseudo-label and a credible AU pseudo-label as a face augmented image from several face pseudo-label images.

[0229] S305, using the "A photo of..." template in CLIP to construct multiple text samples associated with facial expressions. At the same time, construct multiple text samples associated with facial units based on FACS, and use at least one preset learnable vector to replace the adverbs and / or adjectives used to describe the degree of facial movement in each text sample associated with the facial unit to obtain multiple masked text samples associated with the facial unit. A text sample set is constructed by multiple masked text samples associated with the facial unit and multiple text samples associated with facial expressions.

[0230] S306, multiple facial augmented images, several facial sample images and text sample sets are integrated into a facial data set, the facial data set is processed based on Rand Augment, and the pre-built neural network model is trained using the processed facial data set to obtain a second neural network model. The second neural network model mainly includes a basic model, a completion module and a perception module, wherein the basic model is used to encode and linearly process the facial image to be tested to obtain AU visual features and FE visual features; the completion module is used to encode the text data associated with the AU to obtain AU encoding features, and obtain AU text features based on the AU encoding features and the AU visual features, and encode the text data associated with the FE to obtain FE text features; the perception module is used to obtain AU recognition results based on the AU visual features and the AU text features, and obtain FE recognition results based on the FE visual features, the AU visual features, the FE text features and the AU text features.

[0231] S307, obtaining the facial image to be tested and text data, where the text data includes text data associated with AU after mask processing and text data associated with FE, and inputting the facial image to be tested and the text data into the second neural network model, performing AUD and FER on the facial image to be tested through the second neural network model, and then obtaining FE recognition results and AU recognition results.

[0232] In summary, in terms of data augmentation, in order to address the defect of limited data sets, the embodiments of the present application utilize large-scale unlabeled facial images in combination with pseudo-labeling technology to generate a large number of pseudo-labeled facial images, which are all annotated with corresponding FE pseudo-labels and AU pseudo-labels, and screen these pseudo-labeled facial images based on category-specific confidence thresholds to obtain multiple high-confidence augmented facial images to ensure the high credibility of the pseudo-labels and reduce the noise of the pseudo-labels. Then, a facial dataset for training is constructed by using facial sample images annotated with FE labels and AU labels, facial augmented images annotated with FE pseudo-labels and AU pseudo-labels, and text samples used to describe AUs and FEs, which fully considers the intrinsic correlation between different activation intensities of AUs and the correlation between FE and AU, thereby effectively improving the generalization ability and recognition performance of the neural network model for FER tasks and AUD tasks.

[0233] In terms of facial emotion recognition, different from the existing method of treating FER and AUD as two independent tasks or only using the knowledge information of one of them to assist in identifying the other, the embodiment of the present application constructs a second neural network model to achieve joint training of FER and AUD, two highly related tasks, to achieve simultaneous recognition of emotion representations at two different granularity levels, FE and AU, thereby improving the accuracy of facial emotion recognition.

[0234] In terms of AU text description and FE text description, different from the existing use of digitally encoded One-hot tags as supervisory signals, the embodiment of the present application introduces text descriptions of FE and AU with richer semantics as supervisory signals, and makes full use of visual perception to explore the correlation between different AUs and the intrinsic correlation between FE and AU, and dynamically optimizes the text embedding generated by the text description, which not only overcomes the problem of the original text description being too rigid or absolute, but also improves the feature extraction capability of the second neural network model for the visual representation and text representation of FE and AU, thereby improving the recognition accuracy of the second neural network model in FER tasks and AUD tasks.

[0235] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0236] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data augmentation method, characterized in that: The following steps are involved: Acquire a plurality of facial sample images and a plurality of facial unlabeled images, each of the facial sample images being labeled with a corresponding facial unit label and a facial expression label; Using a plurality of the facial sample images to train a pre-built basic model, a first neural network model and a confidence threshold set are obtained, wherein the confidence threshold set includes a confidence threshold of each facial unit label and a confidence threshold of each facial expression label; Inputting a plurality of the facial unlabeled images into the first neural network model to obtain a plurality of facial pseudo-labeled images, and selecting a plurality of facial augmented images from the plurality of the facial pseudo-labeled images according to the confidence threshold set, each of the facial pseudo-labeled images being annotated with a corresponding facial unit pseudo-label and a facial expression pseudo-label; Obtaining a facial data set according to the plurality of facial augmented images, the plurality of facial sample images and a text sample set, wherein the text sample set includes a plurality of masked text samples associated with facial units and a plurality of text samples associated with facial expressions; Among them, the basic model includes an image encoder and a multi-cascade classifier connected in sequence, the multi-cascade classifier includes a facial expression classifier and a facial unit classifier connected in parallel, the facial expression classifier includes a plurality of first linear layers, and the plurality of first linear layers correspond one-to-one to the plurality of facial expression labels, and the facial unit classifier includes a plurality of second linear layers, and the plurality of second linear layers correspond one-to-one to the plurality of facial unit labels; the image encoder is used to encode a plurality of facial sample images to obtain a plurality of visual embeddings; each of the first linear layers is used to process a plurality of the visual embeddings in combination with a preset first activation function to obtain the activation probability of the facial expression label corresponding to each of the first linear layers; each of the second linear layers is used to process a plurality of the visual embeddings in combination with a preset second activation function to obtain the activation probability of the facial unit label corresponding to each of the second linear layers.

2. The data augmentation method according to claim 1, characterized in that: The method of using a plurality of facial sample images to train a pre-built basic model to obtain a first neural network model and a confidence threshold set includes: Performing forward propagation on a pre-built basic model using a plurality of facial sample images to obtain an activation probability of each facial unit label and an activation probability of each facial expression label; Obtaining a loss function value of the basic model according to the activation probability of each of the facial unit labels and the activation probability of each of the facial expression labels; Optimizing the network parameters of the basic model using the loss function value of the basic model; When the basic model does not meet the preset end condition, return to the step of using several facial sample images to perform forward propagation on the pre-built basic model until the basic model meets the end condition, and output the trained basic model as the first neural network model.

3. The data augmentation method according to claim 2, characterized in that: The step of obtaining the loss function value of the basic model according to the activation probability of each of the facial unit labels and the activation probability of each of the facial expression labels comprises: Obtaining a cross entropy loss function value according to the total number of the facial expression labels and the activation probability and true value of each of the facial expression labels; Obtaining an asymmetric loss function value and a dice loss function value according to the total number of the facial unit labels and the activation probability, true value, and occurrence frequency of each of the facial unit labels; The loss function value of the basic model is obtained according to the cross entropy loss function value, the asymmetric loss function value and the dice loss function value.

4. The data augmentation method according to claim 1, characterized in that: The method of using a plurality of facial sample images to train a pre-built basic model to obtain a first neural network model and a confidence threshold set includes: Inputting a plurality of the facial sample images into the first neural network model, obtaining a facial expression prediction label and a prediction probability thereof for each of the facial sample images and a facial unit prediction label and a positive sample prediction probability and a negative sample prediction probability thereof for each of the facial sample images, wherein the positive sample prediction probability is used to characterize the prediction probability that the facial unit prediction label is a positive sample, and the negative sample prediction probability is used to characterize the prediction probability that the facial unit prediction label is a negative sample; For each of the facial expression labels, according to the facial expression prediction label and its prediction probability of each of the facial sample images, a confidence threshold of the facial expression label is obtained; For each of the facial unit labels, a positive sample confidence threshold of the facial unit label is obtained according to the facial unit prediction label of each of the facial sample images and its positive sample prediction probability, and a negative sample confidence threshold of the facial unit label is obtained according to the facial unit prediction label of each of the facial sample images and its negative sample prediction probability, and then the positive sample confidence threshold and the negative sample confidence threshold of the facial unit label are used as the confidence threshold of the facial unit label.

5. The data augmentation method according to claim 4, characterized in that: The step of inputting the plurality of facial unlabeled images into the first neural network model to obtain a plurality of facial pseudo-labeled images, and selecting a plurality of facial augmented images from the plurality of facial pseudo-labeled images according to the confidence threshold set comprises: Inputting a plurality of the facial unlabeled images into the first neural network model, obtaining a plurality of facial pseudo-label images and pseudo-label data of each of the facial pseudo-label images, wherein the pseudo-label data includes facial unit pseudo-labels, predicted probabilities of the facial unit pseudo-labels, facial expression pseudo-labels, and predicted probabilities of the facial expression pseudo-labels; Selecting the facial pseudo-label image whose pseudo-label data satisfies the augmentation condition from among the plurality of facial pseudo-label images as the facial augmented image; Wherein, the amplification conditions include: The predicted probability of the facial expression pseudo-label of the facial pseudo-label image is greater than a confidence threshold of a facial expression label of the same type as the facial expression pseudo-label; The facial unit pseudo-label of the facial pseudo-label image is a positive sample and the predicted probability of the facial unit pseudo-label is greater than the positive sample confidence threshold of the facial unit label of the same type as the facial unit pseudo-label, or the facial unit pseudo-label of the facial pseudo-label image is a negative sample and the predicted probability of the facial unit pseudo-label is less than the negative sample confidence threshold of the facial unit label of the same type as the facial unit pseudo-label.

6. A facial emotion recognition method, characterized in that The following steps are involved: Acquire a facial image to be tested and text data as inputs of a second neural network model, wherein the text data includes text data associated with facial units after mask processing and text data associated with facial expressions; Performing facial recognition on the facial image to be tested by using the second neural network model to obtain a facial unit recognition result and a facial expression recognition result; The second neural network model is obtained by training a facial data set, and the facial data set is obtained by the data augmentation method according to any one of claims 1 to 5, and the second neural network model includes: A basic model is used to perform encoding processing and linear processing on the facial image to be tested to obtain facial unit visual features and facial expression visual features; A completion module is used to encode the text data associated with the facial unit to obtain facial unit encoding features, obtain facial unit text features based on the facial unit encoding features and the facial unit visual features, and encode the text data associated with the facial expression to obtain facial expression text features; A perception module is used to obtain the facial unit recognition result based on the facial unit visual features and the facial unit text features, and to obtain the facial expression recognition result based on the facial unit visual features, the facial expression visual features, the facial unit text features and the facial expression text features.

7. The facial emotion recognition method according to claim 6, characterized in that: The step of obtaining the facial unit text feature according to the facial unit coding feature and the facial unit visual feature comprises: The facial unit encoding features are used as a query matrix, and the facial unit visual features are used as a key matrix and a value matrix. The facial unit encoding features and the facial unit visual features are processed through an attention mechanism to obtain facial unit text features.

8. The facial emotion recognition method according to claim 6, characterized in that: The step of obtaining the facial unit recognition result according to the facial unit visual feature and the facial unit text feature comprises: Using the facial unit visual features as a query matrix, a key matrix, and a value matrix, the facial unit visual features are processed through an attention mechanism to obtain a first refined visual feature; Using the facial unit text features as a query matrix, a key matrix, and a value matrix, the facial unit text features are processed through an attention mechanism to obtain a first refined text feature; The facial unit recognition result is obtained according to the first refined visual feature and the first refined text feature.

9. The facial emotion recognition method according to claim 8, characterized in that: The step of obtaining the facial expression recognition result according to the facial unit visual feature, the facial expression visual feature, the facial unit text feature and the facial expression text feature comprises: Using the facial expression visual feature as a query matrix and the first refined visual feature as a key matrix and a value matrix, the facial expression visual feature and the first refined visual feature are processed by an attention mechanism to obtain a second refined visual feature; Using the facial expression text feature as a query matrix and the first refined text feature as a key matrix and a value matrix, the facial expression text feature and the first refined text feature are processed by an attention mechanism to obtain a second refined text feature; The facial expression recognition result is obtained according to the second refined visual features and the second refined text features.

Images