Facial motion unit recognition method and device and model training method and device

Abstract

A method for training a facial action unit (FAU) recognition model includes: inputting a facial image into a parallel first backbone network and a second backbone network to obtain its first viewpoint features and second viewpoint features, respectively; the facial image includes accurately labeled facial images, inaccurately labeled facial images, or unlabeled facial images; a first clean network and a first noisy network predict corresponding first accurate labels and first inaccurate labels based on the first viewpoint features, respectively; a second clean network and a second noisy network predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features, respectively; or the first clean network predicts corresponding first accurate labels and first pseudo-labels based on the first viewpoint features, and the second clean network predicts corresponding second accurate labels and second pseudo-labels based on the second viewpoint features. This method can utilize accurately labeled datasets, inaccurately labeled datasets, and unlabeled datasets to train a facial action unit (FAU) recognition model with higher accuracy and stronger generalization performance.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for facial action unit recognition, and a method and apparatus for model training. Background Technology

[0002] In 1978, researchers proposed the Facial Action Coding System (FACS), which encodes facial movements into Action Units (AUs) to more accurately describe subtle changes in facial actions. FACS contains nearly a hundred AUs, encoding not only the face but also head posture. By identifying the degree of activation of different AUs, facial movements can be described and further applied to fields such as emotion recognition and lie detection. Since AU annotation requires extensive training of annotators, and annotating a single image often consumes significant time and effort, obtaining large amounts of AU annotation data requires substantial human, material, and financial resources. AU recognition algorithms can effectively solve these problems.

[0003] Current facial action unit (AMU) recognition algorithms can be broadly categorized into four types based on the characteristics of their training data: supervised learning, weakly supervised learning, semi-supervised learning, and self-supervised learning. The first type, supervised learning, primarily utilizes facial images and AMU annotations. These methods often incorporate local facial region features (such as regions of interest, ROIs), relationships between AMUs, temporal information of AMUs, expressions, optical flow, and feature points for modeling. While these methods offer good interpretability, they are prone to overfitting on small datasets due to insufficient data, resulting in poor model generalization. The second type, weakly supervised learning, utilizes annotations related to AMU recognition tasks or noisy AMU labels as weak annotations. These annotations are often readily available, such as expressions and feature points, allowing the use of large-scale unlabeled data. The third type, semi-supervised learning, trains using a small amount of labeled data and a large amount of unlabeled data. These algorithms typically exhibit better generalization. The fourth type of self-supervised learning algorithm mainly utilizes large-scale unlabeled datasets. By mining information within the unlabeled data, such as temporal information, optical flow, and contrast information between faces, a pre-trained model with good representational ability can be obtained. Then, the pre-trained model can be used for fine-tuning in downstream tasks to achieve better results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to solve the problem of how to train a facial action unit (AU) recognition model with higher accuracy and stronger generalization performance by simultaneously utilizing accurate labeled datasets, inaccurate labeled datasets with noisy annotations, or unlabeled datasets when the existing AU datasets are relatively small.

[0005] To achieve the above objectives, this invention proposes a method for training a facial action unit recognition model, comprising:

[0006] The facial images are input into a first backbone network and a second backbone network in parallel to obtain the first viewpoint features and the second viewpoint features of the facial images, respectively, wherein the facial images include accurately labeled facial images and inaccurately labeled facial images;

[0007] A first clean network and a first noisy network predict corresponding first accurate labels and first inaccurate labels based on the accurately labeled facial image and the inaccurately labeled facial image, respectively. A second clean network and a second noisy network predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively.

[0008] The above-mentioned training method for facial action unit recognition models also includes:

[0009] The predicted mean of the first inaccurate label and the second inaccurate label obtained by the first noise network and the second noise network is linearly combined with the inaccurate label of the inaccurately labeled facial image to obtain a new inaccurate label for retraining the first noise network and the second noise network.

[0010] To achieve the above objectives, the present invention also proposes a method for training a facial action unit recognition model, comprising:

[0011] The facial images are input into a first backbone network and a second backbone network in parallel to obtain the first viewpoint features and the second viewpoint features of the facial images, respectively, wherein the facial images include accurately labeled facial images and unlabeled facial images;

[0012] The first clean network predicts a first accurate label and a first pseudo label based on the first viewpoint features of the accurately labeled facial image and the unlabeled facial image, and the second clean network predicts a second accurate label and a second pseudo label based on the second viewpoint features of the accurately labeled facial image and the unlabeled facial image.

[0013] The above-mentioned training method for the facial action unit recognition model includes using accurate label loss to constrain the first clean network and the second clean network, and using inaccurate label loss to constrain the first noisy network and the second noisy network.

[0014] The accurate label loss is:

[0015]

[0016] The inaccurate labeling loss is:

[0017]

[0018] in, Indicates the first One perspective, Indicates the first The accurate label for each AU Indicates the first Inaccurate labels for each AU. This indicates that a clean network is important for the first The prediction results for each AU, Indicates the noise network for the first The prediction results for each AU, and This represents the parameters used for data balancing.

[0019] The above-mentioned training method for the facial action unit recognition model uses orthogonal loss to constrain the weights of the first clean network and the second clean network, and uses consistency loss to constrain the prediction results of the first clean network and the second clean network.

[0020] The orthogonal loss is:

[0021]

[0022] in, Indicates from the A clean network from a third perspective for the first The weight of each AU;

[0023] The consistency loss is:

[0024]

[0025] in, Indicates the predicted probability The entropy.

[0026] In the above-mentioned training method for the facial action unit recognition model, the first backbone network and the second backbone network adopt the ResNet34 architecture, and the first clean network, the first noisy network, the second clean network and the second noisy network adopt fully connected networks.

[0027] To achieve the above objectives, the present invention also proposes a facial action unit recognition model training device, comprising:

[0028] The feature acquisition unit is used to input facial images into a first backbone network and a second backbone network in parallel to obtain first viewpoint features and second viewpoint features of the facial images, wherein the facial images include accurately labeled facial images and inaccurately labeled facial images.

[0029] The label prediction unit is used for a first clean network and a first noisy network to predict corresponding first accurate labels and first inaccurate labels based on the accurately labeled facial image and the inaccurately labeled facial image, respectively; and for a second clean network and a second noisy network to predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively.

[0030] The aforementioned facial action unit recognition model training device further includes:

[0031] The retraining unit is used to linearly combine the predicted mean of the first inaccurate label and the second inaccurate label obtained by the first noise network and the second noise network with the inaccurate label of the inaccurately labeled face image to obtain a new inaccurate label for retraining the first noise network and the second noise network.

[0032] To achieve the above objectives, the present invention also proposes a facial action unit recognition model training device, comprising:

[0033] The feature acquisition unit is used to input facial images into a first backbone network and a second backbone network in parallel to obtain first viewpoint features and second viewpoint features of the facial images, wherein the facial images include accurately labeled facial images and unlabeled facial images.

[0034] The label prediction unit is used by a first clean network to predict corresponding first accurate labels and first pseudo labels based on the first viewpoint features of the accurately labeled facial image and the unlabeled facial image, and by a second clean network to predict corresponding second accurate labels and second pseudo labels based on the second viewpoint features of the accurately labeled facial image and the unlabeled facial image.

[0035] To achieve the above objectives, the present invention also proposes a facial action unit recognition method, comprising:

[0036] The facial image to be identified is input into a first backbone network and a second backbone network in parallel to obtain the first viewpoint features and the second viewpoint features of the facial image to be identified, respectively.

[0037] The first viewpoint features and the second viewpoint features are respectively input into the first clean network and the second clean network trained by the above-mentioned facial action unit recognition model training method to predict the first accurate label and the second accurate label.

[0038] The mean of the first accurate label and the second accurate label is transformed into a corresponding probability value by using the sigmoid function. It is then determined whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

[0039] To achieve the above objectives, the present invention also provides a facial motion unit recognition device, comprising:

[0040] The feature acquisition unit is used to input the face image to be identified into the parallel first backbone network and second backbone network to obtain the first viewpoint features and the second viewpoint features of the face image to be identified, respectively.

[0041] The label prediction unit is used to input the first viewpoint features and the second viewpoint features into the first clean network and the second clean network trained by the above-mentioned facial action unit recognition model training method to predict the first accurate label and the second accurate label.

[0042] The state discrimination unit is used to transform the mean of the first accurate label and the second accurate label into a corresponding probability value through the sigmoid function, and determine whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

[0043] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described facial action unit recognition model training method and / or the above-described facial action unit recognition method.

[0044] To achieve the above objectives, the present invention also proposes an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the above-described facial motion unit recognition model training method and / or the above-described facial motion unit recognition method are implemented.

[0045] As can be seen from the above scheme, the advantages of this invention are: the technical solution proposed in this invention can utilize accurate labeled data, inaccurate labeled data, and unlabeled labeled data for training, achieving better generalization performance and higher recognition accuracy. The main framework of the technical solution of this invention is based on a regularized collaborative training network. Two perspectives are used to learn mutually independent features. Through the constraint of perspective consistency, inaccurate labeled data and unlabeled labeled data can be utilized. In addition, each perspective also uses a noisy network as a regularization term to constrain the clean network, preventing the clean network from overfitting and improving the model's generalization performance. Furthermore, based on the retraining method, the noisy network is retrained using a linear combination of the prediction results of the noisy network after model convergence and the inaccurate labels, further improving the facial action unit recognition accuracy after model convergence. Attached Figure Description

[0046] Figure 1A This is a flowchart of a facial action unit recognition model training method according to an embodiment of the present invention.

[0047] Figure 1B A flowchart illustrating a facial action unit recognition model training method according to another embodiment of the present invention.

[0048] Figure 2A For the corresponding Figure 1A A schematic diagram of the framework of the facial action unit recognition model.

[0049] Figure 2B For the corresponding Figure 1B A schematic diagram of the framework of the facial action unit recognition model.

[0050] Figure 3A This is a block diagram of a facial motion unit recognition model training device according to an embodiment of the present invention.

[0051] Figure 3B This is a block diagram of a facial motion unit recognition model training device according to another embodiment of the present invention.

[0052] Figure 4 This is a flowchart of a facial motion unit recognition method according to an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of the framework of a facial action unit recognition model according to an embodiment of the present invention.

[0054] Figure 6 This is a block diagram of a facial action unit recognition model according to an embodiment of the present invention.

[0055] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0056] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0057] This invention proposes a training method for a facial action unit (AMU) recognition model based on weakly supervised learning. This method utilizes a small amount of accurately labeled data, a large amount of inaccurately labeled data, and unlabeled data as training sets. Although the inaccurately labeled data contains noise, it still contains useful information that can be extracted to improve the model's recognition accuracy, accelerate convergence, and enhance generalization ability. Unlike existing deep learning-based supervised learning methods, the proposed method can utilize a large amount of noisy data to extract useful information. Existing supervised learning methods mainly use AU-related annotations such as expressions and feature points as additional information. Some methods also consider incorporating temporal or optical flow information and combining it with local facial information for training. Because existing AU datasets are relatively small, these methods often suffer from overfitting and have poor generalization performance. The proposed method mainly utilizes the effective information in large-scale noisy data, which can better improve the network's generalization performance.

[0058] In this invention, the training data may be, for example, the EmotioNet dataset used in natural scenes, but this invention is not limited to this and other types of datasets may also be used. The training dataset consists of three parts: one part is facial image data with manually annotated AU labels, referred to as accurately annotated facial images, which have accurate labels; another part is facial images without labels, referred to as unlabeled images, which do not have labels; and the third part can be obtained, for example, by using a network (e.g., ResNet34) pre-trained on the accurately annotated facial image dataset to annotate the unlabeled images, referred to as inaccurately annotated facial images, which have inaccurate labels. Here, there is no limitation on the method of obtaining inaccurately annotated facial images; they can be obtained by a pre-trained model or from manually annotated noisy labeled images. Preferably, facial recognition and five-point (two corners of the eyes, the tip of the nose, and two corners of the mouth) localization can be performed on the RGB facial image using a publicly available facial recognition engine, and the facial region can be cropped out and the facial image can be saved as, for example, 256×256. In addition, during the training process, the facial image is augmented to improve the generalization performance of the model, including randomly cropping the image to, for example, 224×224, and randomly horizontally flipping and randomly grayscale the image.

[0059] See Figure 1A and Figure 2AAs shown, an embodiment of the present invention proposes a facial action unit recognition model training method 100, including steps S110-S120.

[0060] Step S110: Input the facial image into the parallel first backbone network and second backbone network to obtain the first viewpoint features and second viewpoint features of the facial image, wherein the facial image includes accurately labeled facial images and inaccurately labeled facial images.

[0061] The overall network structure of the model includes two perspectives. For each perspective, there is a backbone network, a noisy network, and a clean network. Specifically, the first perspective of the model network includes a first backbone network, a first clean network, and a first noisy network, while the second perspective includes a second backbone network, a second clean network, and a second noisy network. The first and second backbone networks adopt the ResNet34 architecture, and the first, second, first, and second noisy networks, as well as the first and second clean networks, are all fully connected networks. Preferably, the specific structure of the network is shown in Table 1 below, but this invention is not limited thereto.

[0062] Table 1

[0063]

[0064] The obtained training data (facial images) are input into two parallel backbone networks (a first backbone network and a second backbone network) to obtain features of the facial images from two perspectives (first perspective features and second perspective features), wherein the facial images include accurately labeled facial images and inaccurately labeled facial images.

[0065] In step S120, the first clean network and the first noisy network predict the corresponding first accurate label and the first inaccurate label based on the accurately labeled face image and the inaccurately labeled face image, respectively. The second clean network and the second noisy network predict the corresponding second accurate label and the second inaccurate label based on the second viewpoint features of the accurately labeled face image and the inaccurately labeled face image, respectively.

[0066] In this embodiment, the noisy network is used to learn the mapping of image features to inaccurate labels and serves as a regularization term to prevent the clean network from overfitting; the clean network is used to learn the mapping of image features to accurate labels and is used for the final image classification.

[0067] Specifically, from a first perspective, the features of accurately labeled facial images and inaccurately labeled facial images obtained through the first backbone network are respectively input into the first clean network and the first noisy network to predict the corresponding first accurate label and the first inaccurate label; from a second perspective, the features of accurately labeled facial images and inaccurately labeled facial images obtained through the second backbone network are respectively input into the second clean network and the second noisy network to predict the corresponding second accurate label and the second inaccurate label.

[0068] Furthermore, in embodiments of the present invention, during model training, preferably, accurate label loss is used to constrain the clean network, and inaccurate label loss is used to constrain the noisy network; to ensure that the two perspectives learn mutually independent features, orthogonal loss is used to constrain the weight parameters of the clean network; to ensure that the prediction results of the two perspectives are consistent, consistency loss is used to constrain the prediction results of the clean network in both perspectives. Specifically, as shown below:

[0069] The first clean network and the second clean network are constrained using accurate label loss, and the first noisy network and the second noisy network are constrained using inaccurate label loss.

[0070] The accurate labeling loss is:

[0071]

[0072] The inaccurate labeling loss is:

[0073]

[0074] in, Indicates the first One perspective, Indicates the first The accurate label for each AU Indicates the first Inaccurate labels for each AU. This indicates that a clean network is important for the first The prediction results for each AU, Indicates the noise network for the first The prediction results for each AU, and This represents the parameters used for data balancing.

[0075] Orthogonal loss is used to constrain the weights of the first and second clean networks, and consistency loss is used to constrain the prediction results of the first and second clean networks.

[0076] The orthogonal loss is:

[0077]

[0078] in, Indicates from the A clean network from a third perspective for the first The weight of each AU;

[0079] The consistency loss is:

[0080]

[0081] in, Indicates the predicted probability The entropy.

[0082] Furthermore, in an embodiment of the present invention, preferably, after network convergence, the predicted values ​​of inaccurate labels and the inaccurate labels are linearly fused, and the newly generated fused labels are used to retrain the two noisy networks. In other words, the original inaccurate labels are replaced with new inaccurate labels, that is, the predicted mean of the first and second inaccurate labels obtained by the first and second noisy networks is linearly combined with the inaccurate labels of the inaccurately labeled facial images to obtain new inaccurate labels for retraining the first and second noisy networks. An example is as follows:

[0083] For an inaccurately labeled facial image from a noisy dataset Its inaccurate label can be represented as New inaccurate labels used to train noisy networks It can be expressed as a linear combination of the inaccurate labels and the mean of the noisy network predictions from both perspectives, as shown in the following formula:

[0084]

[0085] in, This represents the mean of the network predictions from two perspectives of noise. Indicates inaccurate balance labeling and The weight is usually set to 0.5, but it is not limited to this. The new inaccurate labels are turned into "hard labels" by passing the threshold of 0.5.

[0086] In addition, the The AU prediction probability generated by the noisy network from each perspective can be expressed as follows:

[0087]

[0088] in, This indicates element-wise addition. and They represent the first time. Features of clean and noisy networks from different perspectives.

[0089] See Figure 1B and Figure 2B As shown, based on the same inventive concept, another embodiment of the present invention proposes a facial action unit recognition model training method 100', including: steps S110'-S120'.

[0090] Step S110': Input the facial image into the parallel first backbone network and second backbone network to obtain the first viewpoint features and second viewpoint features of the facial image respectively, wherein the facial image includes accurately labeled facial images and unlabeled facial images;

[0091] In step S120', the first clean network predicts the corresponding first accurate label and first pseudo label based on the first viewpoint features of the accurately labeled facial image and the unlabeled facial image, and the second clean network predicts the corresponding second accurate label and second pseudo label based on the second viewpoint features of the accurately labeled facial image and the unlabeled facial image.

[0092] The overall network structure of the model includes two perspectives. Each perspective contains a backbone network and a clean network. Specifically, the first perspective of the model network includes a first backbone network and a first clean network, and the second perspective of the model network includes a second backbone network and a second clean network. The first and second backbone networks adopt the ResNet34 architecture, and both the first and second clean networks are fully connected networks. The specific structure of the network can be referred to Table 1 above.

[0093] Furthermore, in this embodiment, the clean network is used to learn the mapping from accurately labeled facial image features to accurate labels and from unlabeled facial image features to pseudo-labels, and the accurate labels are used for the final image classification.

[0094] Furthermore, in this embodiment, during model training, preferably, accurate label loss is used to constrain the clean network; to ensure that the two perspectives learn independent features, orthogonal loss is used to constrain the weight parameters of the clean network; and to ensure that the prediction results of the two perspectives are consistent, consistency loss is used to constrain the prediction results of the clean network in both perspectives. The specific loss function can be found in the above embodiment.

[0095] See Figure 3A As shown, based on the same inventive concept, an embodiment of the present invention proposes a facial action unit recognition model training device 200, comprising:

[0096] Feature acquisition unit 210 is used to input facial images into a first backbone network and a second backbone network in parallel to obtain first viewpoint features and second viewpoint features of the facial images respectively, wherein the facial images include accurately labeled facial images and inaccurately labeled facial images.

[0097] The label prediction unit 220 is used for a first clean network and a first noisy network to predict corresponding first accurate labels and first inaccurate labels based on the accurately labeled facial image and the inaccurately labeled facial image, respectively, and for a second clean network and a second noisy network to predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively.

[0098] In one embodiment, the facial action unit recognition model training device 200 further includes:

[0099] The retraining unit 230 is used to linearly combine the predicted mean of the first inaccurate label and the second inaccurate label obtained by the first noise network and the second noise network with the inaccurate label of the inaccurately labeled face image to obtain a new inaccurate label for retraining the first noise network and the second noise network.

[0100] See Figure 3B As shown, based on the same inventive concept, an embodiment of the present invention proposes a facial action unit recognition model training device 200', comprising:

[0101] The feature acquisition unit 210' is used to input the facial image into a parallel first backbone network and a second backbone network to obtain the first viewpoint features and the second viewpoint features of the facial image, wherein the facial image includes an accurately labeled facial image and an unlabeled facial image.

[0102] The label prediction unit 220' is used for a first clean network to predict a first accurate label and a first pseudo label based on the first viewpoint features of the accurately labeled facial image and the unlabeled facial image, and a second clean network to predict a second accurate label and a second pseudo label based on the second viewpoint features of the accurately labeled facial image and the unlabeled facial image.

[0103] The facial action unit recognition model was trained using the above method. The application of this model will now be described. It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the methods, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] See Figure 4 and Figure 5As shown, based on the same inventive concept, an embodiment of the present invention proposes a facial action unit recognition method 300, including steps S310-S330.

[0105] Step S310: Input the face image to be recognized into the parallel first backbone network and second backbone network to obtain the first viewpoint features and second viewpoint features of the face image to be recognized, respectively.

[0106] The facial image to be identified is an unlabeled facial image.

[0107] Step S320: Input the first viewpoint features and the second viewpoint features into the first clean network and the second clean network trained by the facial action unit recognition model training methods 100 and 100' of the above embodiments to predict the first accurate label and the second accurate label.

[0108] Step S330: The mean of the first accurate label and the second accurate label is transformed into the corresponding probability value by the sigmoid function. It is determined whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

[0109] Specifically, the mean of the prediction results of the clean network from two perspectives (first accurate label and second accurate label) is transformed into the corresponding probability value through the sigmoid function. When the probability value is greater than or equal to 0.5, it is determined that the AU is activated; otherwise, it is determined to be suppressed. However, the present invention does not limit the specific value of the threshold, which can be adjusted according to the actual situation.

[0110] See Figure 6 As shown, based on the same inventive concept, an embodiment of the present invention provides a facial motion unit recognition device 400, comprising:

[0111] The feature acquisition unit 410 is used to input the face image to be identified into a parallel first backbone network and a second backbone network to obtain the first viewpoint features and the second viewpoint features of the face image to be identified, respectively.

[0112] The label prediction unit 420 is used to input the first viewpoint features and the second viewpoint features into the first clean network and the second clean network trained by the facial action unit recognition model training methods 100 and 100' of the above embodiments to predict the first accurate label and the second accurate label.

[0113] The state discrimination unit 430 is used to transform the mean of the first accurate label and the second accurate label into the corresponding probability value through the sigmoid function, and determine whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

[0114] See Figure 7 As shown, based on the same inventive concept, embodiments of the present invention also propose an electronic device 500. The electronic device 500 is, for example, but not limited to, a personal computer (PC), smartphone, tablet computer, personal digital assistant (PDA), mobile internet device (MID), etc., and includes a processor 510 and a memory 520. The processor 510 and the memory 520 are electrically connected directly or indirectly to realize data transmission or interaction. The facial motion unit recognition model training devices 200, 200' and / or the facial motion unit recognition model 400 include at least one software module that can be stored in the memory 520 or embedded in the operating system (OS) of the electronic device 500 in the form of software or firmware. The processor 510 executes executable modules stored in the memory 520, such as software functional modules and computer programs included in the facial motion unit recognition model training devices 200 and 200', to implement the facial motion unit recognition model training methods 100 and 100', or the software functional modules and computer programs included in the facial motion unit recognition model 400, to implement the facial motion unit recognition method 300. Upon receiving an execution instruction, the processor 510 executes the computer program.

[0115] Based on the same inventive concept, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the facial action unit recognition model training methods 100, 100' and / or the facial action unit recognition method 300 of the above embodiments. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0116] In summary, the technical solution proposed in this invention involves labeling unlabeled facial image data to obtain inaccurately labeled facial image data. Then, the unlabeled, inaccurately labeled, and accurately labeled data are fed into two parallel networks for collaborative training. These two networks learn different features from the data from two different perspectives, significantly improving the model's generalization performance. To ensure that the features learned from the two perspectives are independent, orthogonal loss is used to constrain the weights of the fully connected network. To ensure consistency in the judgment results from the two perspectives, consistency loss is used to constrain the prediction results of the fully connected networks from both perspectives. For each perspective, a backbone network is first used to extract features. Then, the features extracted from each perspective are passed through a noisy network and a clean network. These two networks are used to learn inaccurate and accurate labels, respectively. The noisy network can be used as a regularization term to constrain the clean network, thereby preventing overfitting. After the network converges, the reliability of the network prediction results is relatively high. At this time, the combination of the noisy network prediction results and the inaccurate labels can better cancel out the noise in the inaccurate labels. Therefore, this invention fuses the mean of the two perspectives of the noisy network prediction results with the inaccurate labels, for example, by linear weighting, and then updates the noisy network with the new inaccurate labels after fusion, thereby achieving the effect of further improving accuracy.

[0117] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for training a facial action unit recognition model, characterized in that, include: The facial images are input into a first backbone network and a second backbone network in parallel to obtain the first viewpoint features and the second viewpoint features of the facial images, respectively, wherein the facial images include accurately labeled facial images and inaccurately labeled facial images; The first clean network and the first noisy network predict corresponding first accurate labels and first inaccurate labels based on the first viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively. The second clean network and the second noisy network predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively.

2. The facial action unit recognition model training method as described in claim 1, characterized in that, Also includes: After the network converges, the predicted mean of the first inaccurate label and the second inaccurate label obtained by the first noise network and the second noise network is linearly combined with the inaccurate label of the inaccurately labeled face image to obtain a new inaccurate label for retraining the first noise network and the second noise network.

3. The facial action unit recognition model training method as described in any one of claims 1-2, characterized in that, The first clean network and the second clean network are constrained using accurate label loss, and the first noisy network and the second noisy network are constrained using inaccurate label loss. The accurate label loss is: The inaccurate labeling loss is: in, Indicates the first One perspective, Indicates the first The accurate label for each AU Indicates the first Inaccurate labels for each AU. This indicates that a clean network is important for the first The prediction results for each AU, Indicates the noise network for the first The prediction results for each AU, and This represents the parameters used for data balancing.

4. The facial action unit recognition model training method as described in claim 3, characterized in that, Orthogonal loss is used to constrain the weights of the first clean network and the second clean network, and consistency loss is used to constrain the prediction results of the first clean network and the second clean network. The orthogonal loss is: in, Indicates from the A clean network from a third perspective for the first The weight of each AU; The consistency loss is: in, Indicates the predicted probability The entropy.

5. The facial action unit recognition model training method as described in claim 1, characterized in that, The first backbone network and the second backbone network adopt the ResNet34 architecture, and the first clean network, the first noisy network, the second clean network and the second noisy network adopt the fully connected network.

6. A facial motion unit recognition model training device, characterized in that, include: The feature acquisition unit is used to input facial images into a first backbone network and a second backbone network in parallel to obtain first viewpoint features and second viewpoint features of the facial images, wherein the facial images include accurately labeled facial images and inaccurately labeled facial images. The label prediction unit is used for a first clean network and a first noisy network to predict corresponding first accurate labels and first inaccurate labels based on the accurately labeled facial image and the inaccurately labeled facial image, respectively; and for a second clean network and a second noisy network to predict corresponding second accurate labels and second inaccurate labels based on the second viewpoint features of the accurately labeled facial image and the inaccurately labeled facial image, respectively.

7. The facial motion unit recognition model training device as described in claim 6, characterized in that, Also includes: The retraining unit is used to linearly combine the predicted mean of the first inaccurate label and the second inaccurate label obtained by the first noise network and the second noise network with the inaccurate label of the inaccurately labeled face image to obtain a new inaccurate label for retraining the first noise network and the second noise network.

8. A method for facial action unit recognition, characterized in that, include: The facial image to be identified is input into a first backbone network and a second backbone network in parallel to obtain the first viewpoint features and the second viewpoint features of the facial image to be identified, respectively. The first viewpoint features and the second viewpoint features are respectively input into the first clean network and the second clean network trained by the facial action unit recognition model training method as described in any one of claims 1-5 to predict the first accurate label and the second accurate label. The mean of the first accurate label and the second accurate label is transformed into a corresponding probability value by using the sigmoid function. It is then determined whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

9. A facial motion unit recognition device, characterized in that, include: The feature acquisition unit is used to input the face image to be identified into the parallel first backbone network and second backbone network to obtain the first viewpoint features and the second viewpoint features of the face image to be identified, respectively. A label prediction unit is used to input the first viewpoint features and the second viewpoint features into the first clean network and the second clean network trained by the facial action unit recognition model training method as described in any one of claims 1-5, respectively, to predict the first accurate label and the second accurate label. The state discrimination unit is used to transform the mean of the first accurate label and the second accurate label into a corresponding probability value through the sigmoid function, and determine whether the probability value is greater than or equal to a threshold. If so, the facial action unit corresponding to the face image to be identified is determined to be activated; otherwise, it is determined to be suppressed.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the facial action unit recognition model training method as described in any one of claims 1-5 and / or the facial action unit recognition method as described in claim 8.

11. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, implements the facial action unit recognition model training method as described in any one of claims 1-5 and / or the facial action unit recognition method as described in claim 8.

Images