Expression and action unit detection method based on multi-task learning and automatic labeling

Abstract

According to the expression and action unit detection method based on multi-task learning and automatic labeling, the multi-task learning model is adopted, the facial expression and action unit can be detected at the same time, and the detection efficiency is improved. Besides, in the training process of the multi-task learning model, an expression detection result is utilized to deduce an action unit label, the dependence on manual labeling is greatly reduced, and the accuracy of the multi-task learning model for a facial expression detection task and an action unit detection task is ensured based on the construction of the corresponding relation between the expression and the action unit.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an expression and action unit detection method based on multi-task learning and automatic labeling. Background Art

[0002] Existing facial expression and action unit detection solutions usually rely on a large amount of manually annotated data to train the detection model. Especially for AU detection, the annotation is complex and time-consuming, which often limits the application scope and effect of the model. In addition, traditional methods usually regard expression detection and AU detection as two independent tasks, failing to fully utilize the inherent relationship between the two, resulting in low data utilization and limited generalization ability of the model. Therefore, how to effectively combine the relationship between facial expressions and AUs and reduce dependence on AU annotation has become an urgent problem to be solved in the industry. Summary of the invention

[0003] Purpose of the invention: The present invention aims to propose an expression and action unit detection method based on multi-task learning and automatic labeling, which can achieve effective expression detection and AU detection without the need for real action unit labels.

[0004] Invention content: To achieve the above-mentioned purpose, the present invention proposes the following technical solutions: In a first aspect, a training method for a multi-task learning model is provided, the method comprising: Acquire a face image sample, wherein the face image sample has an expression label; Inputting the face image sample into the multi-task learning model, extracting the action unit features of the face image sample using the action unit detection branch of the multi-task learning model, and obtaining the action unit detection result, and then obtaining the expression detection result according to the action unit features using the expression detection branch of the multi-task learning model; Determine a first loss function according to the expression detection result and the expression label; Determining an action unit pseudo label according to the expression detection result and a preset correspondence between the expression and the action unit; Determine a second loss function based on the action unit pseudo-label and the action unit detection result; The first loss function and the second loss function are used to update the parameters of the multi-task learning model until a target multi-task learning model is obtained.

[0005] As an optional implementation manner of the method of the first aspect, the multi-task learning model includes an input layer, a feature extraction layer, a face action unit layer, a decoding layer and an output layer connected in sequence, the input layer, the feature extraction layer and the face action unit layer constitute the action unit detection branch, and the decoding layer and the output layer constitute the expression detection branch; The input layer is used to map the face image sample into an input format supported by the feature extraction layer; The feature extraction layer is used to extract facial features from the facial image samples input by the input layer; The face action unit layer is used to extract the action unit features from the face features; The decoding layer is used to capture the weight information of the input action unit features in the current expression through a self-attention mechanism; The output layer is used to determine the probability distribution of the face image sample on the preset expression category according to the weight information of the action unit feature in the current expression output by the decoding layer.

[0006] As an optional implementation manner of the method of the first aspect, using the first loss function and the second loss function to update the multi-task learning model until a target multi-task learning model is obtained, specifically including: Constructing a total loss function according to the first loss function and the second loss function; The total loss function is used to update the parameters of the multi-task learning model until a target multi-task learning model is obtained.

[0007] As an optional implementation manner of the method of the first aspect, determining the action unit pseudo label according to the expression detection result and the preset correspondence between the expression and the action unit specifically includes: The expression detection result is input into a pre-trained pseudo-label generation model to obtain the action unit pseudo-label.

[0008] As an optional implementation manner of the method of the first aspect, the training method of the pseudo-label generation model specifically includes: Acquire a training sample, wherein the training sample is facial expression type data, and the label of the training sample is an action unit label sequence corresponding to the facial expression type data; The training samples are used to train a generation model until the pseudo-label generation model is obtained.

[0009] In the second aspect, a method for detecting expression and action units based on multi-task learning and automatic annotation is provided, the method comprising: Get the target face image; The target face image is input into a multi-task learning model to obtain expression detection results and action detection results; the multi-task learning model is pre-trained using the above-mentioned multi-task learning model training method.

[0010] In a third aspect, a method for automatically labeling action units in facial images is provided, the method comprising: Get the target face image; The target face image is input into a multi-task learning model, and an action unit label is generated for the target face image through an action unit detection branch of the multi-task learning model; the multi-task learning model is pre-trained using the above-mentioned training method for the multi-task learning model.

[0011] In a fourth aspect, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device executes the above-mentioned training method of the multi-task learning model, or executes the above-mentioned expression and action unit detection method based on multi-task learning and automatic labeling, or executes the above-mentioned method for automatic labeling of action units of facial images.

[0012] In a fifth aspect, an electronic device is provided, including: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned training method of the multi-task learning model, or execute the above-mentioned expression and action unit detection method based on multi-task learning and automatic labeling, or execute the above-mentioned method for automatic labeling of action units of facial images.

[0013] Beneficial effects: The embodiments of this specification provide a method for detecting facial expressions and action units based on multi-task learning and automatic annotation. The method adopts a multi-task learning model and can detect facial expressions and action units at the same time, thereby improving the detection efficiency. In addition, the method uses the expression detection results to derive action unit labels during the training process of the multi-task learning model, greatly reducing the reliance on manual annotation, and based on the construction of the correspondence between expressions and action units, ensures the accuracy of the multi-task learning model for facial expression detection tasks and action unit detection tasks.

[0014] The training method of the multi-task learning model, the automatic labeling method of face image action unit labels, the computer-readable storage medium and the electronic device described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of the structure of a multi-task learning model involved in an embodiment.

[0017] Figure 2 The present invention is a flowchart of a training method for a multi-task learning model involved in an embodiment.

[0018] Figure 3 The figure is a schematic diagram of the correspondence between an expression and an action unit involved in an embodiment.

[0019] Figure 4 The present invention is a flowchart of a method for detecting expression and action units based on multi-task learning and automatic labeling according to an embodiment.

[0020] Figure 5 The present invention is a flowchart of a method for automatically labeling action units in facial images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.

[0022] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] For existing facial expression and action unit (AU) detection work, the current mainstream methods rely on detailed and cumbersome expression annotation and action unit AU annotation to train the model to achieve the purpose of detection. However, the annotation of action unit AU is quite cumbersome and complicated. There are few data sets with annotated AU, which are usually difficult to obtain, and the accuracy of the annotation is difficult to guarantee. Inaccurate annotation often causes huge errors in subsequent model training, resulting in unsatisfactory results in the model for facial expression and action unit detection.

[0025] In view of this, the present embodiment provides a training method for a multi-task learning model, a method for detecting expression and action units based on multi-task learning and automatic labeling, a method for automatically labeling action units in facial images, a computer-readable storage medium, and an electronic device.

[0026] The following will be combined with the drawings and specific embodiments of the specification to further describe in detail the training method of the multi-task learning model, the expression and action unit detection method based on multi-task learning and automatic labeling, the automatic labeling method of action units in facial images, the computer-readable storage medium and the electronic device described in one or more embodiments of this specification, but this detailed description does not constitute a limitation on the embodiments of this specification.

[0027] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the structure of a multi-task learning model proposed in one or more embodiments of this specification. Figure 1 As shown in the figure, the model includes an input layer, a feature extraction layer, a face action layer, a decoding layer (Decoder) and an output layer. The input layer, the feature extraction layer and the face action unit layer constitute the action unit detection branch, which is used to perform action unit detection on the input face image. The decoding layer and the output layer constitute the expression detection branch, which is used to perform expression detection based on the action unit detection results.

[0028] In the above multi-task learning model, the input layer is used to obtain the input face image and map the face image into the input format supported by the feature extraction layer. The feature extraction layer can be implemented by a convolutional neural network, which is specifically used to extract face features from the face image input by the input layer. The face action unit layer is used to extract action unit features from the face features output by the feature extraction layer. The decoding layer is used to connect the face action unit layer and the output layer. Due to the complexity of the relationship between expression and action unit (hereinafter referred to as AU), complex expressions do not always correspond to AU one-to-one. Therefore, a decoding layer is constructed between the face action unit layer and the output layer. The action unit features of the face action unit layer are used as the feature input of the output layer. The self-attention mechanism in the decoding layer determines the weight information of the action unit features of each AU in the current expression. According to the weight information of the action unit features output by the decoding layer in the current expression, the output layer finally obtains the probability distribution of the current expression on the preset expression type through linear transformation and softmax function. The output layer combines the self-attention mechanism and the prior knowledge of the relationship between AU and expression to better recognize AU and expression.

[0029] For the above multi-task learning model, one or more embodiments of this specification propose a training method for the multi-task learning model, please refer to Figure 2 , the method may include steps S200 to S210.

[0030] S200: Obtain a face image sample.

[0031] Face images can be extracted from public datasets (such as FER-2013, AffectNet, etc.). Some of these datasets already have labeled data, which can be used directly. For data without expression labels, they can be paired with corresponding expression labels. The above expression labels are used to characterize the facial expressions in face image samples.

[0032] S202: Input the face image sample into the multi-task learning model, use the action unit detection branch of the multi-task learning model to extract the action unit features of the face image sample, and obtain the action unit detection result, and then use the expression detection branch of the multi-task learning model to obtain the expression detection result according to the action unit features.

[0033] In some embodiments, a public facial expression dataset (such as FER-2013, AffectNet, etc.) can be used to train a multi-task learning model. The input of the multi-task learning model is a facial image sample, and the output includes facial expression detection results and action unit (AU) detection results.

[0034] The above-mentioned action unit detection branch and expression detection branch share the features extracted by the underlying feature extraction network.

[0035] Specifically, the action unit detection branch can be Figure 1 The input layer, feature extraction layer and face action unit layer in the face image sample are shown in Figure 1. The input layer maps the face image samples to the input supported by the feature extraction layer. The feature extraction layer is used to extract face features from the face image samples input by the input layer. The face action unit layer is used to extract multiple features related to the face action unit from the face features. Since each face image sample contains multiple face action units, the face action unit layer will extract action unit features of multiple face action units, such as eyebrows, lip corners, eye features, etc. An output layer can also be connected to the face action unit layer to map the action unit features to specific action unit feature types, which is the action unit detection result described above.

[0036] The expression detection branch can be specifically Figure 1 The decoding layer and output layer in the . The decoding layer is used to capture the weight information of the input action unit features in the current expression through the self-attention mechanism. The output layer is used to determine the probability distribution of the face image sample on the preset expression category according to the weight information of the action unit features output by the decoding layer in the current expression, so as to obtain the expression detection result.

[0037] S204: Determine a first loss function according to the expression detection result and the expression label.

[0038] The first loss function is constructed based on the gap between the expression detection results and the expression labels. Expression detection is usually a multi-category classification problem (such as 7 basic expressions). In this step, the cross entropy loss (Categorical Cross-Entropy Loss) can be used to evaluate the difference between the action unit pseudo label and the action unit detection result: ; Among them, M is the number of samples, K is the number of expression categories, is the true category label of the jth sample (using one-hot encoding), is the predicted probability of the kth category for the jth sample.

[0039] S206: Determine an action unit pseudo label according to the expression detection result and the preset correspondence between the expression and the action unit.

[0040] In this step, since the face image samples do not have real action unit labels, the action unit pseudo-labels that the face image samples should correspond to under the current expression can be derived from the expression detection results. The derivation here can be achieved based on the pre-constructed correspondence between expressions and action units. Taking the facial action coding system updated by Paul Ekman et al. in 2002 as an example, the facial action unit layer in the feature extraction layer (neural network) has 43 nodes, representing 43 facial action unit types, and the output layer (output facial expressions) has 7 nodes, representing 7 facial expressions. In this step, the 7 facial expressions (happiness, anger, sadness, surprise, fear, disgust, contempt) and the 43 related facial action unit types can be associated in advance to obtain the above-mentioned correspondence between expressions and action units. As Figure 3 The following is a schematic diagram of the corresponding relationship between an expression and an action unit. Taking the expression of happiness as an example, if the expression detection result is happiness, then Figure 3 It can be reversely deduced that there should be two AU outputs at this time: AU6 and AU12, which represent cheek lifting and mouth corner raising respectively. Then AU6 and AU12 are the above-mentioned action unit pseudo labels.

[0041] In some implementations, machine learning, neural networks, etc. can also be used as a generation model. The generation model can be trained in advance using labeled AU labels and expression labels as sample sets, so that the generation model can obtain the corresponding AU labels based on the input expression information, and finally obtain a pseudo-label generation model.

[0042] Specifically, a training sample may be obtained first, the training sample being human facial expression type data, the label of the training sample being an action unit label sequence corresponding to the human facial expression type data, and then the training sample is used to train the generation model until a pseudo-label generation model is obtained.

[0043] In some implementations, the multi-task learning model can be preliminarily trained using an existing small dataset with labeled AUs, and then migrated to a dataset without AU labels for fine-tuning.

[0044] S208: Determine a second loss function based on the action unit pseudo-label and the action unit detection result.

[0045] The action unit detection task can be regarded as a multi-label classification task, because a face image may contain multiple action units at the same time. The loss of each action unit can be calculated using Binary Cross-Entropy Loss (BCE): ; in,N is the sample size, y i represents the action unit pseudo label, p i represents the action unit detection result, Represents the sigmoid function.

[0046] S210: Using the first loss function and the second loss function, update the parameters of the multi-task learning model until a target multi-task learning model is obtained.

[0047] Specifically, the total loss function can be constructed based on the first loss function and the second loss function. For example, the first loss function and the second loss function are weighted and summed, so that the multi-task learning model can optimize AU detection and expression detection at the same time. The expression of the total loss function is: ; Among them, α and β are weight coefficients used to balance the losses of AU detection and expression detection. Usually, α and β can be determined by experiments or cross-validation to achieve the best task performance.

[0048] Using the above total loss function, the error is reduced by using the gradient descent method, and the parameters of the multi-task learning model are gradually updated until the target multi-task learning model is obtained.

[0049] The above is a training method for a multi-task learning model described in this embodiment. This method combines facial expression detection and AU detection in a multi-task learning framework, uses the relationship between the two for joint optimization, and reduces the need for AU annotation through an automated annotation mechanism, thereby reducing the cost and time of data preparation. Through multi-task learning and difference optimization, the relationship between expression and AU is fully utilized to improve the detection accuracy of the model. Since it does not rely on a large amount of AU annotation data, this method has good versatility and is suitable for different types of facial data and application scenarios.

[0050] Based on the training method of the multi-task learning model described above, this embodiment also provides a method for detecting expression and action units based on multi-task learning and automatic annotation. Figure 4 , the method comprises steps S400 to S402: S400: Acquire a target face image.

[0051] S402: Input the target face image into the multi-task learning model to obtain expression detection results and action detection results.

[0052] The above multi-task learning model is pre-trained using the above multi-task learning model training method.

[0053] Based on the training method of the multi-task learning model described above, this embodiment also provides a method for automatically labeling action units in face images. Figure 5 , the method comprises steps S500 to S502: S500: Acquire a target face image.

[0054] S502: Input a target face image into a multi-task learning model, and generate an action unit label for the target face image through an action unit detection branch of the multi-task learning model.

[0055] The above multi-task learning model is pre-trained using the above multi-task learning model training method.

[0056] Corresponding to the above method, this embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on an electronic device, the electronic device executes the above-mentioned training method of the multi-task learning model, or executes the above-mentioned expression and action unit detection method based on multi-task learning and automatic labeling, or executes the above-mentioned method for automatic labeling of action units in facial images.

[0057] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0058] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0059] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0060] Corresponding to the above method, this embodiment further provides an electronic device, including: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned training method of the multi-task learning model, or execute the above-mentioned expression and action unit detection method based on multi-task learning and automatic labeling, or execute the above-mentioned method for automatic labeling of action units of facial images.

[0061] At the hardware level, the electronic device includes a processor, a computer-readable storage medium, a memory, a data interface, a network interface, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as a processor reading a corresponding computer program from a computer-readable storage medium into a memory and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0062] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A training method for a multi-task learning model, characterized in that: include: Acquire a face image sample, wherein the face image sample has an expression label; Inputting the face image sample into the multi-task learning model, extracting the action unit features of the face image sample using the action unit detection branch of the multi-task learning model, and obtaining the action unit detection result, and then obtaining the expression detection result according to the action unit features using the expression detection branch of the multi-task learning model; Determine a first loss function according to the expression detection result and the expression label; Determining an action unit pseudo label according to the expression detection result and a preset correspondence between the expression and the action unit; Determine a second loss function based on the action unit pseudo-label and the action unit detection result; The first loss function and the second loss function are used to update the parameters of the multi-task learning model until a target multi-task learning model is obtained.

2. The method according to claim 1, characterized in that The multi-task learning model comprises an input layer, a feature extraction layer, a face action unit layer, a decoding layer and an output layer connected in sequence, wherein the input layer, the feature extraction layer and the face action unit layer constitute the action unit detection branch, and the decoding layer and the output layer constitute the expression detection branch; The input layer is used to map the face image sample into an input format supported by the feature extraction layer; The feature extraction layer is used to extract facial features from the facial image samples input by the input layer; The face action unit layer is used to extract the action unit features from the face features; The decoding layer is used to capture the weight information of the input action unit features in the current expression through a self-attention mechanism; The output layer is used to determine the probability distribution of the face image sample on the preset expression category according to the weight information of the action unit feature in the current expression output by the decoding layer.

3. The method according to claim 1, characterized in that: Using the first loss function and the second loss function, updating the multi-task learning model until a target multi-task learning model is obtained, specifically including: Constructing a total loss function according to the first loss function and the second loss function; The total loss function is used to update the parameters of the multi-task learning model until a target multi-task learning model is obtained.

4. The method according to claim 1, characterized in that Determining an action unit pseudo label according to the expression detection result and the preset correspondence between the expression and the action unit, specifically includes: The expression detection result is input into a pre-trained pseudo-label generation model to obtain the action unit pseudo-label.

5. The method according to claim 1, characterized in that The training method of the pseudo-label generation model specifically includes: Acquire a training sample, wherein the training sample is facial expression type data, and the label of the training sample is an action unit label sequence corresponding to the facial expression type data; The training samples are used to train a generation model until the pseudo-label generation model is obtained.

6. Expression and action unit detection method based on multi-task learning and automatic annotation, characterized in that: include: Get the target face image; Inputting the target face image into a multi-task learning model to obtain expression detection results and action detection results; The multi-task learning model is pre-trained using the method described in any one of claims 1 to 5.

7. A method for automatically labeling action units in facial images, characterized in that: include: Get the target face image; The target face image is input into a multi-task learning model, and an action unit label is generated for the target face image through the action unit detection branch of the multi-task learning model; the multi-task learning model is pre-trained using the method described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on an electronic device, the electronic device executes the method according to any one of claims 1 to 5, or executes the method according to claim 6, or executes the method according to claim 7.

9. An electronic device, comprising: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 5, or, execute the method according to claim 6, or, execute the method according to claim 7.

Images