A method for facial action unit recognition based on CLIP and multimodal mask cue learning

Abstract

The present invention discloses a method for facial action unit recognition based on CLIP and multimodal mask prompt learning, comprising the following steps: Step 1: designing a multimodally shared AU prompt (AU-prompt) and an attention mask based on the AU prior position, wherein the AU-prompt contains multiple learnable tokens, and the attention mask is designed according to the prior positions of facial muscle changes of different AUs to obtain a specific AU attention mask; Step 2: during the multimodal prompt learning process, the attention mask of the specific AU is introduced into different layers of the CLIP model text and image encoder for prompt learning, and the AU local detail features and the global correlation features between different AUs are learned at the same time to obtain local features and global features; Step 3: fusing the local features and the global features, and calculating the loss with the text features, finally achieving AU recognition. The present invention can effectively identify facial action units when the labeled data is limited.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial action unit recognition, and in particular relates to a facial action unit recognition method based on CLIP and multimodal mask prompt learning. Background Art

[0002] Facial expression recognition has widespread applications in fields such as affective computing and human-computer interaction. Facial action units (AUs) are the fundamental units that describe facial expressions, and accurately identifying AUs is crucial for understanding a person's emotional state. However, existing AU recognition methods typically require large amounts of annotated data, a process that is time-consuming, expensive, and error-prone, limiting further improvements in AU recognition performance.

[0003] Due to the local nature of AUs, many methods attempt to use regional facial features for AU recognition. PAttNet crops the facial region of a specific AU and uses different CNNs to encode it, weighting each block through an attention mechanism for AU recognition. GLEE-Net designs a local branch that focuses on specific local facial regions. JAA-Net jointly performs AU recognition and facial feature point recognition, and the predicted feature points are used to calculate the attention map for each AU. However, such methods ignore some dependencies between different facial muscles. For example, happiness can usually be represented by a combination of AU6 (cheek lift), AU12 (pull the corners of the mouth), and AU25 (slightly open lips). Summary of the Invention

[0004] In order to overcome the above problems in the prior art, the purpose of the present invention is to provide a method for facial action unit recognition based on CLIP and multimodal mask prompt learning, which can effectively recognize facial action units when the labeled data is limited.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for facial action unit recognition based on CLIP and multimodal mask cue learning, comprising the following steps:

[0007] Step 1: Design a multimodal shared AU prompt (AU-prompt) and attention mask based on the AU prior positions. The AU-prompt contains multiple learnable tokens, and the attention mask is designed based on the prior positions of facial muscle changes of different AUs to obtain a specific AU attention mask.

[0008] Step 2: During the multimodal cue learning process, the attention mask of a specific AU is introduced into different layers of the text and image encoders of the CLIP model for cue learning. The local detail features of the AU and the global correlation features between different AUs are learned simultaneously to obtain local features and global features.

[0009] Step 3: Fuse local features and global features, calculate the loss with text features, and finally achieve AU recognition.

[0010] The multimodally shared AU prompt (AU-prompt) in step 1 is shared between the text and visual branches of the CLIP model, and is respectively subjected to a linear mapping function to obtain the prompt of the text branch and the prompt of the visual branch. The prompt of the text branch and the prompt of the visual branch are respectively spliced ​​with the text features and the image features, and input into the transformer for encoding;

[0011] Set the text linear mapping function to F t (·) The visual linear mapping function is F v (·), then the text branch and the visual branch They are defined as:

[0012]

[0013]

[0014] Among them: AU-prompt P of each layer i Contains z learnable tokens, which are represented as: P i ={p1,p2,...,p z}, where i is the number of layers of transformer, d t is the dimension of the text word vector feature, d v is the dimension of image features.

[0015] CLIP's text encoder is a Transformer model that uses the self-attention mechanism to capture word dependencies and contextual information. The image encoder is a Vision Transformer architecture with powerful feature extraction capabilities.

[0016] If the text encoder T has G transformer layers During prompt fine-tuning, learnable text is only added to the first J layers (J≤G) And combined with the previous layer text feature word vector W i-1 Splicing is sent to the transformer block (Ti ), the text input of each layer of transformer can be obtained as:

[0017]

[0018] Subsequent layers of the text encoder no longer add learnable prompts and only use the previous layer and word vector W j-1 The output of is calculated as follows:

[0019]

[0020] The same image encoder V has G transformer layers Then the input of each layer of transformer in the image encoder can be obtained as:

[0021]

[0022] where c i-1 is the feature vector that captures the image information of the previous layer, E i-1 is the feature vector of the previous layer image, is the hint vector added by the previous layer in the vision branch.

[0023] The attention mask obscures the non-AU area. The attention mask is generated by masking the facial muscle change area of ​​a specific AU, encoding the AU area as 1 and the non-AU area as 0; and making the mask completely cover the specific AU area under different postures.

[0024] The multimodal mask prompt learning in step 2 is specifically as follows:

[0025] The attention mask is concatenated with the image features in the first few layers of the visual transformer. The attention mechanism is used to make the model focus only on the information of the unobstructed areas in the image. The model focuses only on the specific areas of the AU in the first few layers of the visual transformer, thereby learning the local features of the AU. The mask is removed in the subsequent layers, and the global information is used to improve the effect of AU recognition.

[0026] Set M i-1 is the AU attention mask introduced in the previous layer of the image encoder V, then the input and output of the attention mask layer are expressed as:

[0027]

[0028] At the Lth layer, we get the feature vector c that can capture the local information of AU LAfter the L layer, the local attention mask is removed and prompt fine-tuning is performed under the global attention mask to improve the effect of AU recognition by using global information:

[0029]

[0030] Process the image features E of the previous layer in the subsequent transformer layer j-1 And the output information of the prompt Extract global features of AU:

[0031]

[0032] The feature fusion method in step 3 includes: splicing local features and global features; projecting the spliced ​​features into a common vector space of visual text through a linear mapping function, and using a binary cross entropy loss function. The loss function is calculated by comparing the probability value output by the model with the binary value (0 or 1) of the true label, and using the result as the loss value. The cross entropy loss of the AU's input label y (negative class is 0, positive class is 1) and the AU's predicted positive class probability p is:

[0033]

[0034] Beneficial effects of the present invention:

[0035] The present invention improves the AU recognition accuracy for AU datasets: through multimodal mask prompt learning, AU local features and global correlation features can be effectively learned, thereby improving the accuracy of AU recognition.

[0036] Reduce the need for labeled data: The present invention can use language descriptions and image information to improve the robustness and accuracy of AU recognition. The fusion of multimodal data enables the model to better understand and recognize facial action units, and even in the case of insufficient labeled data, it can still achieve a high recognition accuracy. Reduced labeling costs. Strong generalization ability: The CLIP model has strong generalization ability and can adapt to different AU recognition tasks. The present invention uses contrastive learning to perform pre-training on large-scale graphic and text data, which can effectively align visual information and text information, thereby achieving zero-sample or few-sample learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a model structure diagram of the present invention.

[0038] Figure 2 It is the attention mask distribution map of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to the accompanying drawings.

[0040] This paper proposes an AU recognition method based on CLIP and multimodal mask prompt learning. The model structure diagram is as follows: Figure 1 As an example, a multimodal masked prompt learning method is used to design an AU-prompt for each independent AU in the AU recognition task, shared by both the text and vision branches. To extract local information about the AU, a corresponding local attention mask is designed for the AU-prompt based on the prior positions of different AUs and applied to the first few layers of the visual transformer. This allows the AU-prompt to be locally fine-tuned for specific AUs. Subsequent layers of the model leverage inter-AU connections to extract features from global information to enhance AU recognition accuracy, ultimately combining the locally extracted features with the globally extracted features. During the fine-tuning process, only the prompt is learned, while the rest of the model is frozen, ensuring the model's generalization capabilities. The AU-prompt is specifically introduced, and the method of the present invention is explained in detail from both the text and vision branches.

[0041] The specific steps are as follows:

[0042] Step 1: In order to learn the feature information of different AUs without destroying the parameters of the model itself, the present invention introduces a specific prompt for each AU and fine-tunes the prompt of the CLIP model.

[0043] Specifically, the first J layers of the transformer in the model introduce a learnable AU-prompt P i , AU-prompt P of each layer i Contains z learnable tokens, which are represented as: P i ={p1,p2,...,p z}, where i is the number of layers of transformer. i The text and visual branches are shared, and the text branch is obtained by linear mapping functions Ft(·) and Fv(·) respectively. and the visual branch

[0044]

[0045]

[0046] Where dt is the dimension of the text word vector feature, and dv is the dimension of the image feature.

[0047] The present invention uses attention mask to capture the local features of each AU. Masking is performed based on the muscle change area of ​​each AU. In order to overcome the problem of large variations in head postures of different individuals in the dataset, the mask is designed so that it can completely cover the specific AU area under different postures. For example, when fine-tuning the visual prompt for AU1 (inner eyebrow raised), AU6 (cheek raised) in the middle half of the face, and AU12 (lip raised) in the lower half of the face, their mask positions are as follows: Figure 2 As shown, red represents the AU area, which is coded as 1, and other non-AU areas are coded as 0.

[0048] Step 2: The text encoder T has G transformer layers During prompt fine-tuning, learnable text is only added to the first J layers (J≤G) And combined with the previous layer text feature word vector W i-1 Splicing is sent to the transformer block (T i ):

[0049]

[0050] [·,·] is a concatenation operation. In particular, W0 generates text description features by segmenting the input information "An image of a face {with / without}AUnumberpresent" and then projects it. number is the number of different AUs, N is the number of word vectors, and dt is the dimension of the text word vector feature. After the Jth layer, no learnable prompt is added, and only the output of the prompt of the previous layer is used:

[0051]

[0052] Finally, the text embedding corresponding to the class token of the last Transformer block is projected into the common vector space of visual text through the projection layer to obtain the final text features y1, y2.

[0053] The image encoder V has G transformer layers For the input image It is divided into m image blocks of fixed size and projected onto Where H and W are the height and width of the image respectively, and dv is the dimension of the image feature. Similar to the text encoder, the visual branch of CLIP also converts z learnable It is concatenated with the image vector E0 as the input of the image encoder, and a learnable prompt is added only in the first J layers of the transformer. The specific calculation process is as follows:

[0054]

[0055] The AU attention mask M of each layer is obtained by the above method i , in the process of visual prompt fine-tuning, the first L layers are passed through M i The non-AU area is blocked so that the attention mechanism in the transformer block only focuses on the information of the unblocked area in the image. Local prompt fine-tuning is performed in this area to reduce the influence of other areas and effectively learn the local features of the AU:

[0056]

[0057] At the Lth layer, we obtain the feature vector cL that can capture the local information of the AU. After the Lth layer, we remove the local attention mask and perform prompt fine-tuning under the global attention mask to improve the effect of AU recognition by using global information:

[0058]

[0059] Each subsequent layer processes the image of the previous layer and the output information of prompt:

[0060]

[0061] The above formula is used to obtain the feature vector cG of the global information of the image in the last layer of the transformer. Finally, cL and cG are fused and projected into the common vector space of visual text to obtain the final image feature x.

[0062] Step 3: For classification, the text prompt is a manually designed class label y∈{with,without} with two categories, and the temperature parameter τ is used to calculate the prediction corresponding to the image I with the highest cosine similarity score (sim(·))

[0063]

[0064] AU recognition is a classic binary classification problem, which involves assigning an object to one of two classes. For AU recognition, it can be divided into two categories: presence and absence.

[0065] The present invention uses a binary cross entropy loss function, which is calculated by comparing the probability value output by the model with the binary value (0 or 1) of the true label and using the result as the loss value. The cross entropy loss of the output label l (which can be 0 or 1) and the predicted probability p is defined as:

[0066]

[0067] In Prompt Learning, the present invention performs regional learning by controlling the predefined attention masks of different AU prior positions to capture the local features of AUs. At the same time, in order to make full use of the intrinsic connections between AUs, global information is further incorporated to improve AU recognition performance.

Claims

1. A method for facial action unit recognition based on CLIP and multimodal mask cue learning, characterized in that The following steps are included: Step 1: Design a multimodal shared AU prompt and attention mask based on the AU prior positions. The AU-prompt contains multiple learnable tokens, and the attention mask is designed based on the prior positions of facial muscle changes of different AUs to obtain a specific AU attention mask. Step 2: During the multimodal cue learning process, the attention mask of a specific AU is introduced into different layers of the text and image encoders of the CLIP model for cue learning. The local detail features of the AU and the global correlation features between different AUs are learned simultaneously to obtain local features and global features. Step 3: Fuse local features and global features, calculate the loss with text features, and finally realize AU recognition; CLIP's text encoder uses the Transformer model based on the self-attention mechanism, and the image encoder is the VisionTransformer architecture; If the text encoder T has G transformer layers During prompt fine-tuning, learnable text is only added to the first J layers (J≤G) And combined with the previous layer text feature word vector W i-1 Splicing is sent to the transformer block (T i ), the text input of each layer of transformer can be obtained as: Subsequent layers of the text encoder no longer add learnable prompts and only use the previous layer and word vector W j-1 The output of is calculated as follows: The same image encoder V has G transformer layers Then the input of each layer of transformer in the image encoder can be obtained as: where c i-1 is the feature vector that captures the image information of the previous layer, E i-1 is the feature vector of the previous layer image, is the hint vector added by the previous layer in the visual branch; The multimodal mask prompt learning in step 2 is specifically as follows: The attention mask is concatenated with the image features in the first few layers of the visual transformer. The attention mechanism is used to make the model focus only on the information of the unoccluded areas in the image. The model focuses only on the specific areas of the AU in the first few layers of the visual transformer, thereby learning the local features of the AU. The mask is removed in the subsequent layers, and the global information is used to improve the effect of AU recognition. Set M i-1 is the AU attention mask introduced in the previous layer of the image encoder V, then the input and output of the attention mask layer are expressed as: At the Lth layer, the feature vector c that captures the local information of AU is obtained L , after the L layer, the local attention mask is removed, prompt fine-tuning is performed under the global attention mask, and the global information is used to improve the effect of AU recognition: Process the image features E of the previous layer in the subsequent transformer layer j-1 And the output information of the prompt Extract global features of AU:

2. The method for facial action unit recognition based on CLIP and multimodal mask prompt learning according to claim 1, characterized in that: The multimodally shared AU prompt (AU-prompt) in step 1 is shared between the text and visual branches of the CLIP model, and is respectively subjected to a linear mapping function to obtain the prompt of the text branch and the prompt of the visual branch. The prompt of the text branch and the prompt of the visual branch are respectively spliced ​​with the text features and the image features, and input into the transformer for encoding; Set the text linear mapping function to F t (·) The visual linear mapping function is F v (·), then promptP of the text branch i t And promptP of the visual branch i v They are defined as: Among them: AU-promptP of each layer i Contains z learnable tokens, which are represented as: P i ={p1,p2,...,p z }, where i is the number of layers of transformer, d t is the dimension of the text word vector feature, d v is the dimension of image features.

3. The method for facial action unit recognition based on CLIP and multimodal mask prompt learning according to claim 2, characterized in that: The attention mask obscures the non-AU area. The attention mask is generated by masking the facial muscle change area of ​​a specific AU, encoding the AU area as 1 and the non-AU area as 0; and making the mask completely cover the specific AU area under different postures.

4. The method for facial action unit recognition based on CLIP and multimodal mask prompt learning according to claim 1, characterized in that: The feature fusion method in step 3 includes: splicing local features and global features; projecting the spliced ​​features into a common vector space of visual text through a linear mapping function, and using a binary cross entropy loss function. The loss function is calculated by comparing the probability value output by the model with the binary value (0 or 1) of the true label, and using the result as the loss value. The cross entropy loss of the AU's input label y (negative class is 0, positive class is 1) and the AU's predicted positive class probability p is:

Images