A facial action unit recognition method based on global-local cooperative regularization

Abstract

This invention relates to the field of image processing technology, specifically to a face action unit recognition method based on global-local collaborative regularization. The method uses labeled face training data input into a deep neural network to extract high-level semantic feature representations. A global branch network is constructed to model the dependencies between different face action units globally. A local branch network is constructed to model the external relationships between face action units in local face regions, and the intra-regional connections between face action units within local face regions, thereby learning discriminative region features related to face action units. A consistent collaborative regularization mechanism is designed to ensure that the features learned by the global and local branches output consistent prediction results after passing through different multi-label classifiers. This invention generates a face action unit recognition network through repeated iterative iterations, generating optimal network parameters and improving the accuracy of face action unit recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a face action unit recognition method based on global-local collaborative regularization. Background Technology

[0002] Facial expressions typically reflect a person's psychological state and emotional information. Automated facial expression analysis technology has a wide range of needs and applications, attracting significant attention in scenarios such as auxiliary diagnosis and treatment of physiological and mental illnesses, fatigue driving detection, lie detection in criminal investigations, and human-computer interaction. Therefore, achieving automatic facial expression recognition and analysis has important practical significance and social value.

[0003] In 1978, American psychologists Ekman et al. proposed the Facial Action Coding System (FACS) to describe the subtle movements of facial muscles and defined a series of facial action units (AUs) with specific semantics. For example, AU1 represents raising the inner eyebrow, and AU12 represents stretching the corners of the mouth. Since any facial expression can be represented as a combination of several AUs, AUs provide a more objective, accurate, and fine-grained description than the coarse-grained classification of the six basic facial expressions. The goal of AU recognition is to identify the state of several AUs, i.e., whether they are activated or inactive, from a facial image. AU and its combinations are more helpful in accurately analyzing facial expressions and understanding individual emotions.

[0004] However, existing AU (Activity Unit) labeled datasets are limited and lack diversity. AUs exhibit significant intra-class variation among different individuals, and the same AU can show varying intensities across different faces. Therefore, learning discriminative AU features is extremely challenging, and existing face action unit recognition models suffer from low accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a face action unit recognition method based on global-local collaborative regularization, which aims to solve the problem of low recognition accuracy of existing face action unit recognition models.

[0006] To achieve the above objectives, this invention provides a face action unit recognition method based on global-local collaborative regularization, comprising the following steps:

[0007] Step 1: Obtain a training set of face images labeled with facial action units and perform preprocessing operations;

[0008] Step 2: Construct a feature extractor to extract high-level semantic feature representations related to human face;

[0009] Step 3: Construct a global-local dual-branch network to model the relationships between facial action units of different granularities;

[0010] Step 4: Design a consistent collaborative regularization constraint mechanism to collaboratively train the global-local dual-branch network and optimize the network model;

[0011] Step 5: Train the face action unit recognition network using backpropagation of errors, continuously update the network model, converge to generate the optimal network parameters, and obtain the best face action unit recognition network model.

[0012] Step 6: Input the face image to be recognized into the converged face action unit recognition model for testing, and obtain the face action unit recognition result.

[0013] Optionally, the preprocessing operations in step 1 include alignment operations, data augmentation operations consisting of random cropping and random horizontal flipping, image augmentation, and normalization.

[0014] Optionally, the feature extractor in step 2 consists of a ResNet34 neural network, used to extract 512-dimensional feature representations.

[0015] Optionally, the global-local dual-branch network includes a global branch network and a local branch network. The global branch network models the relationship between pixels across regions, realizes the interaction between features of different face action units (AUs) in the global scope, and captures the global context information of different AUs. The local branch network obtains high-dimensional feature representations by region, which are respectively fed into three independent face action unit classifiers to obtain prediction results for different region blocks. The prediction results of the local branch network are then combined to obtain the final prediction results of the local branch network.

[0016] Optionally, the global branch network uses a multi-label cross-entropy classification loss function for AU identification, as shown in the following formula:

[0017]

[0018] in, This represents the classification loss function for the global branch, where C represents the number of AU classes. Let p represent the prediction result of the i-th AU in the global branch network, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data.

[0019] Optionally, the local branch network uses a multi-label cross-entropy classification loss function for AU identification, as shown in the following formula:

[0020]

[0021] in, This represents the classification loss function for local branches, where C represents the number of AU classes. Let p represent the prediction result of the local branch for the i-th AU, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data.

[0022] Optionally, for the prediction results p of the global-local dual-branch network output. g and p l The Jensen-Shannon divergence is used for consistency regularization constraints, as shown in the following formula:

[0023]

[0024] Where, L con The consistent collaborative regularization loss function is represented by C; C represents the number of AU categories. This represents the prediction result of the global branch network for the i-th AU; while Let H(p) represent the prediction result of the local branch network for the i-th AU; where H(p) is calculated as follows:

[0025] H(p)=-(plogp+(1-p)log(1-p)).

[0026] Optionally, the overall loss function of the face action unit recognition network model is expressed by the following formula:

[0027]

[0028] Here, α = 120 is a hyperparameter used to balance the weight of the latter loss.

[0029] Optionally, in step 5, the overall loss function of the face action unit recognition network model is calculated, the network model is continuously updated, the optimal network parameters are generated, and the network error is eventually converged to the optimal range, thereby obtaining the optimal face action unit recognition network model.

[0030] Optionally, in step 6, the test sample is input into the converged face action unit recognition model, and finally the classifier is used to obtain the face action unit recognition result.

[0031] This invention provides a face action unit recognition method based on global-local collaborative regularization. It uses labeled face training data input into a deep neural network to extract high-level semantic feature representations. A global branch network is constructed to model the dependencies between different face action units globally. A local branch network is constructed to model the external relationships between face action units in local face regions, and the intra-regional connections between face action units within a local face region, thereby learning discriminative regional feature representations related to face action units. A consistent collaborative regularization mechanism is designed to ensure that the feature representations learned by the global and local branches output consistent prediction results after passing through different multi-label classifiers. This invention uses a collaborative learning strategy for model training, iteratively generating the face action unit recognition network, using error backpropagation to train the network, continuously updating the network model, generating optimal network parameters, extracting robust sentiment representations, and outputting reliable multi-label prediction results, thus improving the robustness of the model and the accuracy of face action unit recognition. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the steps of a face action unit recognition method based on global-local collaborative regularization according to the present invention.

[0034] Figure 2 This is an overall architecture diagram of the face action unit recognition model based on global-local collaborative regularization in an embodiment of the present invention. Detailed Implementation

[0035] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0036] Please see Figure 1 This invention provides a face action unit recognition method based on global-local collaborative regularization, comprising the following steps:

[0037] S1: Obtain a training set of face images labeled with facial action units and perform preprocessing operations;

[0038] S2: Construct a feature extractor to extract high-level semantic feature representations related to human face;

[0039] S3: Construct a global-local dual-branch network to model the relationships between facial action units of different granularities;

[0040] S4: Design a consistent collaborative regularization constraint mechanism to collaboratively train the global-local dual-branch network and optimize the network model;

[0041] S5: Train the face action unit recognition network using error backpropagation, continuously update the network model, converge to generate the optimal network parameters, and obtain the best face action unit recognition network model.

[0042] S6: Input the face image to be recognized into the converged face action unit recognition model for testing, and obtain the face action unit recognition result.

[0043] The following description, in conjunction with specific embodiments and execution steps, illustrates the process:

[0044] Step S1: Obtain the training set DISFA of face images with facial action unit annotations, and perform preprocessing operations on the training set, including alignment operations, data augmentation operations consisting of random cropping and random horizontal flipping to augment the images, and normalization processing.

[0045] Step S2: Construct a feature extractor composed of ResNet34; input each face image in the face image training set into the feature extractor to extract high-level semantic feature representations related to facial information;

[0046] Step S3: Construct a global-local dual-branch network to model the relationship between facial action units at different granularities, so as to learn discriminative facial action unit feature representations and output the recognition prediction results of the dual-branch network.

[0047] like Figure 2 The diagram shows a global-local dual-branch network provided in an embodiment of the present invention, specifically including:

[0048] (1) The global branch network mainly consists of two parts: a global interaction module and a multi-label classifier. The global interaction module includes a dynamic position bias module (DPB), a multi-head self-attention module (MHSA), a multilayer perceptron module (MLP), a normalization layer (LN), and a global average pooling layer (GAP). By modeling the relationships between pixels across regions, it realizes the interaction between features of different AUs globally, capturing the global contextual information of different AUs, enabling the global branch network to better perceive and learn the inherent regional dependencies of AUs. Specifically, the high-dimensional feature F output by the feature extractor is input into the global interaction module to obtain the output feature representation Z. g See formulas (1) and (2):

[0049] M g =MHSA(LN(F)+DPB)+F (1)

[0050] Z g =GAP[MLP(LN(M g (2)

[0051] DPB comes from CrossFormer (CrossFormer: A versatile vision transformer hinging on cross-scale attention).

[0052] Next, Z g The prediction result p is fed into the face action unit classifier and outputs the global branch network. g Therefore, the global branch network uses the multi-label cross-entropy classification loss function for AU identification, as shown in formula (3):

[0053]

[0054] in, This represents the classification loss function for the global branch, where C represents the number of AU classes. Let p represent the prediction result of the i-th AU in the global branch network, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data.

[0055] (2) The local branch network mainly consists of four parts: (i) region partitioning module, (ii) intrinsic feature relationship module, (iii) region interaction module, and (iv) three multi-label classifiers.

[0056] The region segmentation module includes a spatial transformation network (STN) for obtaining three predefined ROI regions corresponding to the original face image from the high-dimensional feature F representation. (High-dimensional feature representation of a portion of the face) (High-dimensional feature representation of the central region of the face) (High-dimensional feature representation of the lower part of the face), where the transformation matrix of STN is shown in formula (4):

[0057]

[0058] in, Let x and y represent the scaling factors of the horizontal and vertical axes of the i-th region, respectively. These represent the translation coefficients of the i-th region, which can be calculated using the corresponding top-left and bottom-right coordinates.

[0059] The intrinsic feature relationship module first utilizes a self-attention mechanism. The self-attention layer obtains Q (query), K (key), and V (value) through three 1x1 convolutional layers. A softmax layer then dynamically aggregates local region features, adding the features of the original segmented regions to the dynamically aggregated features to further enhance the AU feature representation of each local region. This is followed by global average pooling (GAP) and a linear layer consisting of a 3x3 convolutional layer, a normalization layer, and a ReLU activation function to obtain the enhanced region features. The feature output based on the intrinsic feature relationship module is O. u , O m , O l For simplicity, the intrinsic feature relationship module is shown in formula (5).

[0060] O u =ReLU(BN(Conv) 1×1 (GAP(softmax(Q T K)V+F u (5)

[0061] Similarly, we can obtain O m With O l .

[0062] The region interaction module, composed of a dynamic graph convolutional neural network, aims to model the dependencies between AU features in different regions, thereby providing crucial complementary information for different AU feature representations. Specifically, it uses the output features O of the intrinsic feature relationship module... u O m O l By splicing, we get O = [O uO m O l O serves as the input to the region interaction module, i.e., as the input node of the graph convolutional neural network. The correlation matrix A is dynamically constructed based on the features of the input nodes, reflecting the topological relationships between each node. Finally, the feature representation output by the region interaction module is Z = [Z u Z m Z l See formula (6):

[0063] Z=Leaky ReLU[O+GCN(O,A)] (6)

[0064] Where GCN(·) represents a graph convolutional neural network layer consisting of a one-dimensional convolution with a 1x1 kernel and Leaky ReLU, A = Sigmoid(Conv1(O)) represents the dynamic correlation matrix, and Conv1 represents a one-dimensional convolution with a 1x1 kernel.

[0065] The multi-label classifier in the local branch network will convert the feature representations Z of the three regions output by the region interaction module into the feature representations Z of the three regions. u Z m Z l The data are fed into three independent face action unit classifiers to obtain prediction results p for different regions. u ,p m ,p l Then p u ,p m ,p l The final prediction result p of the local branch network is obtained by splicing the results. l Therefore, the local branch network uses the multi-label cross-entropy classification loss function for AU identification, as shown in formula (7):

[0066]

[0067] in, This represents the classification loss function for local branches, where C represents the number of AU classes. Let p represent the prediction result of the local branch for the i-th AU, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data.

[0068] Step S4: Design a consistent collaborative regularization constraint mechanism to collaboratively train the global-local dual-branch network, that is, for the prediction result p output by the global-local dual-branch network. g and p lJensen-Shannon divergence is used for consistency regularization constraints, which force the same face image to output consistent prediction results after passing through a common feature extractor, global-local dual-branch network and multi-label classifier. Through the mutual constraint mechanism, the network is prevented from learning bias information, as shown in formula (8):

[0069]

[0070] Where, L con The consistent collaborative regularization loss function is represented by C; C represents the number of AU categories. This represents the prediction result of the global branch network for the i-th AU; while H(p) represents the prediction result of the local branch network for the i-th AU; where H(p) = -(plogp + (1-p)log(1-p)).

[0071] The overall loss function of the face action unit recognition model based on global-local collaborative regularization is given by formula (9):

[0072]

[0073] Here, α = 120 is a hyperparameter used to balance the weight of the latter loss.

[0074] Step S5: Input the face images in the training set into the face action unit recognition network, use backpropagation of error to train the neural network, continuously update the network model, generate the optimal network parameters, and finally converge the network error to the optimal range, thereby obtaining the best face action unit recognition model.

[0075] Step S6: Input the face image to be recognized into the converged face action unit recognition model for testing, and obtain accurate face action unit recognition results.

[0076] Furthermore, to verify the feasibility of the method, this invention trained and tested a network model for facial action unit recognition based on global-local co-regularization on the DISFA dataset. The recognition performance of the identity-independent three-fold cross-validation model on eight AUs (AU1 (inner eyebrow lift), AU2 (outer eyebrow lift), AU4 (eyebrow convergence and downward pressure), AU6 (cheek lift), AU9 (nose wrinkle), AU12 (corner of mouth lift), AU25 (lip separation), AU26 (chin drop)) was assessed. DISFA is the dataset proposed in the paper "Disfa: A spontaneous facial action intensity database [J]. IEEE Transactions on Affective Computing, 2013, 4(2): 151-160". The experimental equipment consisted of a CPU 12700 / 12700KF, 32GB RAM, and an NVIDIA GeForce RTX 3090Ti. For facial action unit recognition, which is essentially a binary classification task for a single facial action unit (activated or inactive), the average recognition accuracy (F1-score) was used. This is used to measure the final recognition performance. Indicates recall rate, TP represents the number of correctly predicted positive samples, FP represents the number of incorrectly predicted positive samples, and FP represents the number of incorrectly predicted negative samples.

[0077] The proposed face action unit recognition method based on global-local collaborative regularization (this invention) is compared with five state-of-the-art methods, including Emotion-Aware Contrast Learning Network (EmoCo), Uncertain Graph Neural Network (UGN-B), Hybrid Message Passing Network (HMP-PS), Uncertainty Weighted Multi-Task Learning Network (RA-UML), and Semi-Supervised Contrast Learning Network (SSCL). The results are shown in Table 1, with the best recognition result (F1-score, %) marked in bold. This invention's face action unit recognition method based on global-local collaborative regularization surpasses current existing face action unit recognition methods.

[0078] Table 1: Quantitative experimental results of each method on the DISFA dataset

[0079]

[0080] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A face action unit recognition method based on global-local collaborative regularization, characterized in that, Includes the following steps: Step 1: Obtain a training set of face images labeled with facial action units and perform preprocessing operations; Step 2: Construct a feature extractor to extract high-level semantic feature representations related to human face; Step 3: Construct a global-local dual-branch network to model the relationships between facial action units of different granularities; The global-local dual-branch network includes a global branch network and a local branch network. The global branch network models the relationship between pixels across regions, realizes the interaction between different AU features in the global scope, and captures the global context information of different AUs. The AU is the Action Unit, i.e., the face action unit. The local branch network obtains high-dimensional feature representations by region, which are fed into three independent face action unit classifiers to obtain prediction results for different region blocks. The prediction results of the local branch network are then combined to obtain the final prediction results of the local branch network. The global branch network uses a multi-label cross-entropy classification loss function for AU identification, as shown in the following formula: in, This represents the classification loss function for the global branch, where C represents the number of AU classes. Let p represent the prediction result of the i-th AU in the global branch network, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data; The local branch network uses a multi-label cross-entropy classification loss function for AU identification, as shown in the following formula: in, This represents the classification loss function for local branches, where C represents the number of AU classes. Let p represent the prediction result of the local branch for the i-th AU, and p i b represents the true label value of the i-th AU. c This represents a balancing parameter used to balance the sample weights in each batch of training data; Step 4: Design a consistent collaborative regularization constraint mechanism to collaboratively train the global-local dual-branch network and optimize the network model; Step 5: Train the face action unit recognition network using backpropagation of errors, continuously update the network model, converge to generate the optimal network parameters, and obtain the best face action unit recognition network model. Step 6: Input the face image to be recognized into the converged face action unit recognition network model for testing, and obtain the face action unit recognition results.

2. The face action unit recognition method based on global-local collaborative regularization as described in claim 1, characterized in that, The preprocessing operations in step 1 include alignment, data augmentation using a combination of random cropping and random horizontal flipping, image augmentation, and normalization.

3. The face action unit recognition method based on global-local collaborative regularization as described in claim 2, characterized in that, The feature extractor in step 2 consists of a ResNet34 neural network, used to extract 512-dimensional feature representations.

4. The face action unit recognition method based on global-local collaborative regularization as described in claim 3, characterized in that, For the prediction result p of the global-local dual-branch network output g and p l The Jensen-Shannon divergence is used for consistency regularization constraints, as shown in the following formula: Where, L con The consistent collaborative regularization loss function is represented by C; C represents the number of AU categories. This represents the prediction result of the global branch network for the i-th AU; while Let H(p) represent the prediction result of the local branch network for the i-th AU; where H(p) is calculated as follows: H(p)=-(plogp+(1-p)log(1-p)).

5. The face action unit recognition method based on global-local collaborative regularization as described in claim 4, characterized in that, The overall loss function of the face action unit recognition network model is expressed by the following formula: Here, α = 120 is a hyperparameter used to balance the weight of the latter loss.

6. The face action unit recognition method based on global-local collaborative regularization as described in claim 5, characterized in that, In step 5, the overall loss function of the face action unit recognition network model is calculated, the network model is continuously updated, the optimal network parameters are generated, and the network error is eventually converged to the optimal range, thereby obtaining the optimal face action unit recognition network model.

7. The face action unit recognition method based on global-local collaborative regularization as described in claim 6, characterized in that, In step 6, the test sample is input into the converged face action unit recognition model, and finally the classifier is used to obtain the face action unit recognition result.

Images