Facial expression and action unit detection method based on multi-task learning and automatic annotation

Abstract

The application provides an expression and action unit detection method based on multi-task learning and automatic labeling. The method adopts a multi-task learning model, can simultaneously perform face expression and action unit detection, and improves detection efficiency. In addition, the method deduces action unit labels by using expression detection results in the training process of the multi-task learning model, greatly reduces the dependence on manual labeling, and ensures the accuracy of the multi-task learning model for the face expression detection task and the action unit detection task based on the construction of the corresponding relationship between the expression and the action unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an expression and action unit detection method based on multi-task learning and automatic labeling. BACKGROUND

[0002] In existing face expression and action unit detection schemes, a large amount of manual labeling data is usually relied on to train a detection model, and especially for AU detection, due to complex and time-consuming labeling, the application range and effect of the model are often limited. In addition, the traditional method usually regards expression detection and AU detection as two independent tasks, and fails to fully utilize the inherent relationship between the two, resulting in low data utilization and limited generalization ability of the model. Therefore, how to effectively combine the relationship between face expression and AU and reduce the dependence on AU labeling has become a problem that needs to be solved in the industry. SUMMARY

[0003] The present application aims to provide an expression and action unit detection method based on multi-task learning and automatic labeling, which can realize effective expression detection and AU detection without real action unit labels.

[0004] The present application proposes the following technical solutions to achieve the above-mentioned purposes:

[0005] In a first aspect, a training method of a multi-task learning model is provided, and the method comprises:

[0006] Obtaining a face image sample, wherein the face image sample has an expression label;

[0007] Inputting the face image sample into a multi-task learning model, extracting action unit features of the face image sample by using an action unit detection branch of the multi-task learning model, obtaining an action unit detection result, and then obtaining an expression detection result by using an expression detection branch of the multi-task learning model according to the action unit features;

[0008] Determining a first loss function according to the expression detection result and the expression label;

[0009] Determining an action unit pseudo-label according to the expression detection result and a preset corresponding relationship between expressions and action units;

[0010] Determining a second loss function based on the action unit pseudo-label and the action unit detection result;

[0011] Updating parameters of the multi-task learning model by using the first loss function and the second loss function until a target multi-task learning model is obtained.

[0012] As an optional implementation of the method of the first aspect, the multi-task learning model comprises an input layer, a feature extraction layer, a facial action unit layer, a decoding layer and an output layer connected in sequence, the input layer, the feature extraction layer and the facial action unit layer constitute the action unit detection branch, and the decoding layer and the output layer constitute the expression detection branch.

[0013] The input layer is configured to map the facial image sample into an input format supported by the feature extraction layer.

[0014] The feature extraction layer is configured to extract facial features from the facial image sample input by the input layer.

[0015] The facial action unit layer is configured to extract the action unit features from the facial features.

[0016] The decoding layer is configured to capture weight information of the input action unit features in the current expression by a self-attention mechanism.

[0017] The output layer is configured to determine a probability distribution of the facial image sample in a preset expression category according to the weight information of the action unit features in the current expression output by the decoding layer.

[0018] As an optional implementation of the method of the first aspect, the multi-task learning model is updated by using the first loss function and the second loss function until a target multi-task learning model is obtained, and the method specifically comprises the following steps:

[0019] According to the first loss function and the second loss function, a total loss function is constructed.

[0020] The parameters of the multi-task learning model are updated by using the total loss function until a target multi-task learning model is obtained.

[0021] As an optional implementation of the method of the first aspect, the action unit pseudo label is determined according to the expression detection result and a preset correspondence relationship between expressions and action units, and the method specifically comprises the following steps:

[0022] The expression detection result is input into a pre-trained pseudo label generation model to obtain the action unit pseudo label.

[0023] As an optional implementation of the method of the first aspect, the training method of the pseudo label generation model specifically comprises the following steps:

[0024] A training sample is obtained, the training sample is facial expression type data, and the label of the training sample is an action unit label sequence corresponding to the facial expression type data.

[0025] training the generative model with the training samples until the pseudo-label generative model is obtained.

[0026] In a second aspect, an expression and action unit detection method based on multi-task learning and automatic labeling is provided, and the method comprises:

[0027] obtaining a target face image;

[0028] inputting the target face image into the multi-task learning model to obtain an expression detection result and an action detection result; the multi-task learning model is obtained by pre-training using the training method of the multi-task learning model.

[0029] In a third aspect, a face image action unit label automatic labeling method is provided, and the method comprises:

[0030] obtaining a target face image;

[0031] inputting the target face image into the multi-task learning model, and generating an action unit label for the target face image by an action unit detection branch of the multi-task learning model; the multi-task learning model is obtained by pre-training using the training method of the multi-task learning model.

[0032] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, when the computer program is executed on an electronic device, the electronic device executes the training method of the multi-task learning model, or executes the expression and action unit detection method based on multi-task learning and automatic labeling, or executes the face image action unit label automatic labeling method.

[0033] In a fifth aspect, an electronic device is provided, and the electronic device comprises:

[0034] at least one memory for storing a program;

[0035] at least one processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the training method of the multi-task learning model, or execute the expression and action unit detection method based on multi-task learning and automatic labeling, or execute the face image action unit label automatic labeling method.

[0036] Beneficial effects: the embodiment of the specification provides an expression and action unit detection method based on multi-task learning and automatic annotation. The method adopts a multi-task learning model, can simultaneously detect facial expressions and action units, and improves detection efficiency. In addition, the method deduces action unit labels by using expression detection results in the training process of the multi-task learning model, greatly reduces the dependence on manual annotation, and ensures the accuracy of the multi-task learning model for the facial expression detection task and the action unit detection task based on the construction of the corresponding relationship between the expressions and the action units.

[0037] The training method of the multi-task learning model, the automatic labeling method of the action unit label of the facial image, the computer readable storage medium and the electronic device described in the embodiment of the specification also have the beneficial effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the specification or the prior art, the drawings needed to be used in the embodiment or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0039] Figure 1 A structural schematic diagram of a multi-task learning model related to the embodiment.

[0040] Figure 2 A flowchart of a training method of a multi-task learning model related to the embodiment.

[0041] Figure 3 A corresponding relationship between an expression and an action unit related to the embodiment.

[0042] Figure 4 A flowchart of an expression and action unit detection method based on multi-task learning and automatic annotation related to the embodiment.

[0043] Figure 5 A flowchart of an automatic labeling method of an action unit label of a facial image related to the embodiment. DETAILED DESCRIPTION

[0044] First of all, it should be noted that the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0045] In order for those skilled in the technical field to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in combination with the drawings in the specification. Obviously, the described embodiments are only part of the embodiments of the specification, not all. Therefore, those skilled in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and brevity, the description below omits the description of well-known functions and structures.

[0046] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the specification in other embodiments. In some other embodiments, the steps included in the method can be more or less than described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; and multiple steps described in the specification can be combined into a single step for description in other embodiments.

[0047] For existing face expression and action unit (hereinafter referred to as AU) detection work, the current mainstream method is to train the model by relying on fine and tedious expression labeling and action unit AU labeling to achieve the purpose of detection. However, the labeling of action unit AU is quite tedious and complex, and the data set of labeled AU is very rare, which is usually difficult to obtain, and the accuracy of labeling is also difficult to guarantee. Inaccurate labeling often causes huge errors in subsequent model training, resulting in unsatisfactory results of the model in face expression and action unit detection.

[0048] Therefore, the present embodiment provides a multi-task learning model training method, an expression and action unit detection method based on multi-task learning and automatic labeling, a face image action unit label automatic labeling method, a computer readable storage medium and an electronic device.

[0049] The multi-task learning model training method, the expression and action unit detection method based on multi-task learning and automatic labeling, the face image action unit label automatic labeling method, the computer readable storage medium and the electronic device described in one or more embodiments of the specification will be further described in detail below in combination with the drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation on the embodiments of the specification.

[0050] Please refer to Figure 1 , Figure 1 A structural diagram of a multi-task learning model is provided in one or more embodiments of the specification. As shown in Figure 1As shown, the model includes an input layer, a feature extraction layer, a facial action unit layer, a decoder layer, and an output layer. Among them, the input layer, the feature extraction layer, and the facial action unit layer constitute an action unit detection branch, which is used for action unit detection on the input face image; the decoder layer and the output layer constitute an expression detection branch, which is used for expression detection based on the action unit detection result.

[0051] In the above multi-task learning model, the input layer is used to obtain the input face image and map the face image into an input format supported by the feature extraction layer. The feature extraction layer can be implemented by a convolutional neural network, which is specifically used to extract face features from the face image input by the input layer. The facial action unit layer is used to extract action unit features from the face features output by the feature extraction layer. The decoder layer is used to connect the facial action unit layer and the output layer. Due to the complexity of the relationship between expressions and action units (hereinafter referred to as AUs), complex expressions do not always correspond to AUs one by one, so a decoder layer is constructed between the facial action unit layer and the output layer. The action unit features of the facial action unit layer are input as the features of the output layer, and the weight information of each action unit feature in the current expression is determined by the self-attention mechanism in the decoder layer. The output layer obtains the weight information of the action unit features in the current expression according to the decoder layer, and finally obtains the probability distribution of the current expression on the preset expression type through linear transformation and softmax function. The output layer combines the self-attention mechanism and the prior knowledge of the relationship between AUs and expressions, which can better identify AUs and expressions.

[0052] For the above multi-task learning model, a training method of a multi-task learning model is proposed in one or more embodiments of the present specification, please refer to Figure 2 The method can include steps S200 to S210.

[0053] S200: Obtain face image samples.

[0054] Face images can be extracted from public datasets (such as FER-2013, AffectNet, etc.). Some of these data sets already have labels, and such data can be used directly. For data without expression labels, it can be paired with the corresponding expression label. The above expression label is used to represent the facial expression in the face image sample.

[0055] S202: input the face image sample into the multi-task learning model, use the action unit detection branch of the multi-task learning model to extract the action unit features of the face image sample, and obtain the action unit detection result, and then use the expression detection branch of the multi-task learning model to obtain the expression detection result according to the action unit features.

[0056] In some embodiments, a public facial expression dataset (such as FER-2013, AffectNet, etc.) can be utilized to train the multi-task learning model. The input of the multi-task learning model is a facial image sample, and the output includes the expression detection result and the action unit (AU) detection result of the face.

[0057] The action unit detection branch and the expression detection branch described above share the features extracted by the bottom-layer feature extraction network.

[0058] Specifically, the action unit detection branch can specifically be the input layer, the feature extraction layer, and the facial action unit layer in Figure 1 The input layer maps the facial image sample to the input supported by the feature extraction layer, the feature extraction layer is used to extract facial features from the facial image sample input by the input layer, and the facial action unit layer is used to extract multiple features related to facial action units from the facial features. Since each facial image sample contains multiple facial action units, the facial action unit layer will extract action unit features of multiple facial action units, such as eyebrow, lip corner, eye features, etc. An output layer can also be connected to the facial action unit layer to map the action unit features to specific action unit feature types, that is, the action unit detection result described above.

[0059] The expression detection branch can specifically be the decoding layer and the output layer in Figure 1 The decoding layer is used to capture the weight information of the input action unit features in the current expression through the self-attention mechanism. The output layer is used to determine the probability distribution of the facial image sample in the preset expression category according to the weight information of the action unit features output by the decoding layer in the current expression, so as to obtain the expression detection result.

[0060] S204: Determine a first loss function according to the expression detection result and the expression label.

[0061] The first loss function is constructed based on the gap between the expression detection result and the expression label. Expression detection is usually a multi-class classification problem (such as 7 basic expressions), and in this step, the cross-entropy loss (Categorical Cross-Entropy Loss) can be used to evaluate the difference between the action unit pseudo-label and the action unit detection result:

[0062] ;

[0063] Where M is the number of samples, K is the number of expression categories, is the true class label of the jth sample (using one-hot encoding), is the predicted probability of the kth category of the jth sample.

[0064] S206: Determine the action unit pseudo label according to the expression detection result and the preset correspondence between the expression and the action unit.

[0065] In this step, since the face image sample does not have a real action unit label, the action unit pseudo label corresponding to the face image sample under the current expression can be derived through the expression detection result. Here, the derivation can be achieved according to the pre-constructed correspondence between the expression and the action unit. For example, in the facial action coding system updated by Paul Ekman et al. in 2002, the facial action unit layer in the feature extraction layer (neural network) has 43 nodes, representing 43 facial action unit types, and the output layer (output facial expression) has 7 nodes, representing 7 facial expressions. In this step, the 7 facial expressions (happy, angry, sad, surprised, scared, disgusted, contemptuous) and the 43 facial action unit types related thereto can be associated in advance to obtain the above-mentioned correspondence between the expression and the action unit. For example, as shown in FIG. 6, which is a schematic diagram of the correspondence between the expression and the action unit, taking the expression of happiness as an example, if the expression detection result is happiness, then according to the pre-constructed correspondence between the expression and the action unit, the action unit pseudo label corresponding to the expression of happiness can be determined as AU6 and AU12, which represent cheek raise and mouth corner up, respectively. Figure 3 Figure 3 AU6 and AU12 are the action unit pseudo labels.

[0066] In some embodiments, machine learning, neural networks, etc. can also be used as the generation model. The labeled AU labels and expression labels can be used as a sample set to train the generation model in advance, so that the generation model can obtain the corresponding AU label based on the input expression information, and finally obtain a pseudo label generation model.

[0067] Specifically, a training sample can be obtained first, the training sample being a face expression type data, and the label of the training sample being an action unit label sequence corresponding to the face expression type data. Then the training sample is used to train the generation model until a pseudo label generation model is obtained.

[0068] In some embodiments, the multi-task learning model can also be preliminarily trained using an existing small labeled AU data set, and then migrated to a data set without AU labels for fine-tuning.

[0069] S208: Determine the second loss function based on the action unit pseudo label and the action unit detection result.

[0070] ​The action unit detection task can be regarded as a multi-label classification task because a face image can contain multiple action units simultaneously. The loss of each action unit can be calculated using binary cross-entropy loss (BCE):

[0071] ;

[0072] wherein, N is the number of samples, y i represents the action unit pseudo-label, p i represents the action unit detection result, represents the sigmoid function.

[0073] S210: Update the parameters of the multi-task learning model using the first loss function and the second loss function until a target multi-task learning model is obtained.

[0074] Specifically, a total loss function can be constructed according to the first loss function and the second loss function. For example, the first loss function and the second loss function are weighted and summed, so that the multi-task learning model optimizes AU detection and expression detection simultaneously. The expression of the total loss function is:

[0075] ;

[0076] wherein, α and β are weight coefficients for balancing the loss of AU detection and expression detection. Generally, α and β can be determined by experiments or cross-validation to obtain the optimal task performance.

[0077] Using the above total loss function, the parameters of the multi-task learning model are gradually updated by using the gradient descent method to reduce the error until a target multi-task learning model is obtained.

[0078] The above is a training method of a multi-task learning model according to the present embodiment. The method combines face expression detection and AU detection in a multi-task learning framework, jointly optimizes the relationship between the two, and reduces the demand for AU annotation through an automatic annotation mechanism, thereby reducing the cost and time of data preparation. Through multi-task learning and difference optimization, the relationship between expression and AU is fully utilized to improve the detection accuracy of the model. Since it does not rely on a large amount of AU annotation data, the method has good versatility and is suitable for different types of face data and application scenarios.

[0079] Based on the training method of the multi-task learning model described above, the present embodiment further provides an expression and action unit detection method based on multi-task learning and automatic annotation. Please refer to Figure 4The method comprises steps S400 to S402:

[0080] S400: Obtain a target face image.

[0081] S402: Input the target face image into the multi-task learning model to obtain an expression detection result and a motion detection result.

[0082] The multi-task learning model is obtained by using the training method of the multi-task learning model.

[0083] Based on the training method of the multi-task learning model, the embodiment further provides a face image motion unit label automatic labeling method, please refer to Figure 5 The method comprises steps S500 to S502:

[0084] S500: Obtain a target face image.

[0085] S502: Input the target face image into the multi-task learning model, and generate a motion unit label for the target face image by using the motion unit detection branch of the multi-task learning model.

[0086] The multi-task learning model is obtained by using the training method of the multi-task learning model.

[0087] Corresponding to the above method, the embodiment further provides a computer readable storage medium, which stores a computer program, when the computer program runs on an electronic device, the electronic device executes the training method of the multi-task learning model, or executes the expression and motion unit detection method based on multi-task learning and automatic labeling, or executes the face image motion unit label automatic labeling method.

[0088] In a typical configuration, the computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memories.

[0089] The memory can include non-permanent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0090] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage, quantum memory, graphene-based storage medium or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0091] Corresponding to the above method, the embodiment also provides an electronic device, comprising:

[0092] At least one memory for storing programs;

[0093] At least one processor for executing the programs stored in the memory, when the programs stored in the memory are executed, the processor is used to execute the training method of the multi-task learning model described above, or execute the expression and action unit detection method based on multi-task learning and automatic annotation described above, or execute the face image action unit label automatic annotation method described above.

[0094] At the hardware level, the electronic device includes a processor, a computer-readable storage medium, a memory, a data interface, a network interface, and of course other hardware required by the business. One or more embodiments of the present specification can be implemented in a software manner, such as reading the corresponding computer program from the computer-readable storage medium into the memory by the processor and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as logic devices or software and hardware combined manner, etc., that is, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic device.

[0095] The technical features of the above-described embodiments can be combined in any combination. In order to make the description concise, not all possible combinations of technical features in the above-described embodiments are described, however, as long as the combination of technical features does not exist contradictory, it should be considered as the scope of the present specification.

[0096] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A training method for a multi-task learning model, characterized in that, include: Obtain facial image samples, wherein the facial image samples have expression tags; The face image sample is input into a multi-task learning model, which includes an input layer, a feature extraction layer, a face action unit layer, a decoding layer and an output layer connected in sequence. The input layer, the feature extraction layer and the face action unit layer constitute an action unit detection branch, and the decoding layer and the output layer constitute an expression detection branch. The input layer maps the face image sample to the input format supported by the feature extraction layer. The feature extraction layer extracts face features from the face image sample input by the input layer. The face action unit layer extracts multiple face action unit action unit features from the face features. Based on the multiple face action unit action unit action unit features, the action unit detection result is obtained. The decoding layer uses a self-attention mechanism to capture the weight information of the input action unit features in the current expression. The output layer determines the probability distribution of the face image sample in the preset expression category based on the weight information of the action unit features output by the decoding layer in the current expression, and obtains the expression detection result. A first loss function is determined based on the expression detection results and the expression labels; Based on the expression detection results and the preset correspondence between expressions and action units, pseudo-labels for action units are determined; A second loss function is determined based on the pseudo-label of the action unit and the detection result of the action unit; The parameters of the multi-task learning model are updated using the first loss function and the second loss function until the target multi-task learning model is obtained.

2. The method according to claim 1, characterized in that, The multi-task learning model is updated using the first loss function and the second loss function until the target multi-task learning model is obtained, specifically including: Construct a total loss function based on the first loss function and the second loss function; The parameters of the multi-task learning model are updated using the total loss function until the target multi-task learning model is obtained.

3. The method according to claim 1, characterized in that, Based on the expression detection results and the preset correspondence between expressions and action units, pseudo-labels for action units are determined, specifically including: The expression detection results are input into a pre-trained pseudo-label generation model to obtain the pseudo-labels for the action units.

4. The method according to claim 3, characterized in that, The training method for the pseudo-label generation model specifically includes: Obtain training samples, wherein the training samples are facial expression type data, and the labels of the training samples are the action unit label sequences corresponding to the facial expression type data; The training samples are used to train the generation model until the pseudo-label generation model is obtained.

5. A method for detecting facial expression and action units based on multi-task learning and automated annotation, characterized in that, include: Acquire the target face image; The target face image is input into a multi-task learning model to obtain expression detection results and action detection results; The multi-task learning model is pre-trained using the method described in any one of claims 1 to 4.

6. A method for automatically labeling action units in facial images, characterized in that, include: Acquire the target face image; The target face image is input into a multi-task learning model, and action unit labels are generated on the target face image through the action unit detection branch of the multi-task learning model; the multi-task learning model is pre-trained using the method described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 4, or the method as described in claim 5, or the method as described in claim 6.

8. An electronic device, comprising: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1 to 4, or to perform the method as described in claim 5, or to perform the method as described in claim 6.

Images