Facial action unit recognition method based on CLIP and multi-mode mask prompt learning

Abstract

The invention discloses a facial action unit identification method based on CLIP and multi-mode mask prompt learning. The method comprises the following steps: 1, carrying out facial action unit identification on the basis of the CLIP and the multi-mode mask prompt learning; the method comprises the steps that 1, a multi-modal shared AU prompt (AU-prompt) and an attention mask are designed based on an AU prior position, the AU-prompt comprises a plurality of learnable tokens, the attention mask is designed according to prior positions of different AU facial muscle changes, and a specific AU attention mask is obtained; 2, in a multi-mode prompt learning process, introducing an attention mask of a specific AU into different layers of a CLIP model text and an image encoder for prompt learning, and learning local detail features of the AU and global association features among different AUs to obtain local features and global features; and step 3, fusing the local features and the global features, calculating loss with the text features, and finally realizing AU identification. According to the method, the facial action unit can be effectively recognized under the condition that the labeled data is limited.

Description

Technical Field

[0001] The invention belongs to the technical field of facial action unit recognition, and in particular relates to a facial action unit recognition method based on CLIP and multimodal mask prompt learning. Background Art

[0002] Facial expression recognition has wide applications in the fields of affective computing and human-computer interaction. Facial action units (AUs) are the basic units for describing facial expressions, and accurate AU recognition is crucial for understanding a person's emotional state. However, existing AU recognition methods usually require a large amount of annotated data, and the annotation process is time-consuming, expensive, and error-prone, which limits the further improvement of AU recognition performance.

[0003] Due to the local characteristics of AU, many methods try to use regional facial features for AU recognition. PAttNet crops the facial area of ​​a specific AU and uses different CNNs for encoding, and weights each block through the attention mechanism for AU recognition. GLEE-Net designs a local branch that focuses on a specific local facial area. JAA-Net jointly performs AU recognition and facial feature point recognition, and the predicted feature points are used to calculate the attention map of each AU. However, such methods ignore some dependencies between different facial muscles. For example, happiness can usually be represented by a combination of AU6 (cheek lift), AU12 (pull the corners of the mouth), and AU25 (lips slightly open). Summary of the invention

[0004] In order to overcome the above problems in the prior art, the purpose of the present invention is to provide a method for facial action unit recognition based on CLIP and multimodal mask prompt learning, which can effectively recognize facial action units when the labeled data is limited.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for facial action unit recognition based on CLIP and multimodal mask cue learning, comprising the following steps;

[0007] Step 1: Design a multimodal shared AU prompt (AU-prompt) and attention mask based on the AU prior position. The AU-prompt contains multiple learnable tokens. The attention mask is designed according to the prior positions of facial muscle changes of different AUs to obtain a specific AU attention mask.

[0008] Step 2: In the process of multimodal cue learning, the attention mask of a specific AU is introduced into different layers of the text and image encoders of the CLIP model for cue learning. At the same time, the local detail features of the AU and the global correlation features between different AUs are learned to obtain local features and global features.

[0009] Step 3: Fuse local features and global features, and calculate the loss with text features to finally achieve AU recognition.

[0010] In step 1, the multimodally shared AU prompt (AU-prompt) is shared in the text and visual branches of the CLIP model, and the prompt of the text branch and the prompt of the visual branch are obtained through linear mapping functions respectively. The prompt of the text branch and the prompt of the visual branch are respectively concatenated with the text features and the image features, and input into the transformer for encoding;

[0011] Set the text linear mapping function to F t (·) The visual linear mapping function is F v (·), then the text branch And the visual branch They are defined as:

[0012]

[0013]

[0014] Among them: AU-prompt P of each layer i Contains z learnable tokens, which are represented by: P i ={p1,p2,...,p z}, where i is the number of transformer layers, d t is the dimension of the text word vector feature, d v is the dimension of image features.

[0015] CLIP's text encoder is a Transformer model. Through the self-attention mechanism, the model can capture the dependencies and contextual information between words. The image encoder is a Vision Transformer architecture with powerful feature extraction capabilities.

[0016] If the text encoder T has G transformer layers During prompt fine-tuning, only the first J layers (J ≤ G) are added with learnable text. And combined with the previous layer text feature word vector W i-1 Concatenate and feed into transformer block (Ti ), the text input of each layer of transformer can be obtained as:

[0017]

[0018] Subsequent layers of the text encoder no longer add learnable prompts, only the previous layer is used and word vector W j-1 The output of is calculated as follows:

[0019]

[0020] The same image encoder V has G transformer layers Then the input of each layer of transformer in the image encoder can be obtained as:

[0021]

[0022] where c i-1 is the feature vector that captures the image information of the previous layer, E i-1 is the feature vector of the previous layer image, is the hint vector added by the previous layer in the vision branch.

[0023] The attention mask covers the non-AU area, and the generation method of the attention mask includes: masking according to the facial muscle change area of ​​the specific AU, encoding the AU area as 1 and encoding the non-AU area as 0; making the mask completely cover the specific AU area under different postures.

[0024] The step 2 of multimodal mask prompt learning is specifically as follows:

[0025] In the first few layers of the visual transformer, the attention mask is concatenated with the image features. The attention mechanism is used to make the model focus only on the information of the unoccluded areas in the image, so that the model only focuses on the specific areas of the AU in the first few layers of the visual transformer, thereby learning the local features of the AU. The mask is removed in the next few layers, and the global information is used to improve the effect of AU recognition.

[0026] Set M i-1 is the AU attention mask introduced in the previous layer of the image encoder V, then the input and output of the attention mask layer are expressed as:

[0027]

[0028] At the Lth layer, we get the feature vector c that can capture the local information of the AU LAfter the L layer, the local attention mask is removed, and prompt fine-tuning is performed under the global attention mask to improve the effect of AU recognition by using global information:

[0029]

[0030] The image features E of the previous layer are processed in the subsequent transformer layer j-1 And the output information of the prompt Extract global features of AU:

[0031]

[0032] The feature fusion method of step 3 includes: splicing local features and global features; projecting the spliced ​​features to the common vector space of visual text through a linear mapping function, and using a binary cross entropy loss function. The calculation method of the loss function is to compare the probability value output by the model with the binary value (0 or 1) of the true label, and use the result as the loss value. The cross entropy loss of the AU input label y (negative class is 0, positive class is 1) and the AU predicted positive class probability p is:

[0033]

[0034] Beneficial effects of the present invention:

[0035] The present invention improves the AU recognition accuracy for AU data sets: through multimodal mask prompt learning, AU local features and global correlation features can be effectively learned, thereby improving the AU recognition accuracy.

[0036] Reduce the need for labeled data: The present invention can use language descriptions and image information to improve the robustness and accuracy of AU recognition. The fusion of multimodal data enables the model to better understand and recognize facial action units, and even in the case of insufficient labeled data, it can still achieve a high recognition accuracy. Reduced labeling costs. Strong generalization ability: The CLIP model has strong generalization ability and can adapt to different AU recognition tasks. The present invention uses contrastive learning to perform pre-training on large-scale graphic data, which can effectively align visual information and text information, thereby achieving zero-sample or few-sample learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a model structure diagram of the present invention.

[0038] Figure 2 It is the attention mask distribution map of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0040] This paper proposes an AU recognition method based on CLIP and multimodal mask prompt learning. Figure 1 As shown in the figure, for example, a multimodal mask prompt learning method is used to design an AU-prompt shared by the text and visual branches for each independent AU in the AU recognition task. In order to extract the local information of the AU, a corresponding local attention mask is designed for the AU-prompt according to the prior positions of different AUs and applied to the first few layers of the visual transformer, so that the AU-prompt is locally fine-tuned for the specific AU. The following layers of the model use the connection between AUs to extract features from the global information to enhance the accuracy of AU recognition, and finally combine the locally extracted features with the globally extracted features. During the fine-tuning process, only the prompt is learned, while the rest of the model is frozen, ensuring the generalization ability of the model itself. The AU-prompt will be specifically introduced and the method of the present invention will be explained in detail from the two branches of text and vision.

[0041] The specific steps are as follows:

[0042] Step 1: In order to learn the feature information of different AUs without destroying the parameters of the model itself, the present invention introduces a specific prompt for each AU and fine-tunes the prompt of the CLIP model.

[0043] Specifically, the first J layers of the transformer in the model introduce a learnable AU-prompt P i , each layer's AU-prompt P i Contains z learnable tokens, which are represented by: P i ={p1,p2,...,p z}, where i is the number of layers of the transformer. i The text and visual branches are shared, and the text branch is obtained by linear mapping functions Ft(·) and Fv(·) respectively. And the visual branch

[0044]

[0045]

[0046] Where dt is the dimension of the text word vector feature, and dv is the dimension of the image feature.

[0047] The present invention uses attention mask to capture the local features of each AU. Masking is performed based on the muscle change area of ​​each AU. In order to overcome the problem of large changes in head postures of different individuals in the data set, the mask is designed so that it can completely cover the specific AU area under different postures. For example, when fine-tuning the visual prompt for the upper half of the face AU1 (inner eyebrow raised), the middle half of the face AU6 (cheek raised), and the lower half of the face AU12 (lip raised), their mask positions are as follows: Figure 2 As shown, red represents the AU area, which is encoded as 1, and other non-AU areas are encoded as 0.

[0048] Step 2: The text encoder T has G transformer layers During prompt fine-tuning, only the first J layers (J ≤ G) are added with learnable text. And combined with the previous layer text feature word vector W i-1 Concatenate and feed into transformer block (T i ):

[0049]

[0050] [·,·] is a concatenation operation. In particular, W0 generates text description features and projects the input information "An image of a face {with / without}AUnumberpresent" by segmenting the words. number is the number of different AUs, N is the number of word vectors, and dt is the dimension of the text word vector feature. After the Jth layer, no learnable prompt is added, and only the output of the previous layer prompt is used:

[0051]

[0052] Finally, the text embedding corresponding to the class token of the last Transformer block is projected into the common vector space of visual text through the projection layer to obtain the final text features y1, y2.

[0053] The image encoder V has G transformer layers For the input image It is divided into m image patches of fixed size and projected onto Where H and W are the height and width of the image, respectively, and dv is the dimension of the image feature. Similar to the text encoder, the visual branch of CLIP also transforms z learnable It is concatenated with the image vector E0 as the input of the image encoder, and a learnable prompt is added only in the first J layers of the transformer. The specific calculation process is as follows:

[0054]

[0055] The AU attention mask M of each layer is obtained by the above method i , in the process of visual prompt fine-tuning, the first L layers are passed through M i The non-AU area is blocked so that the attention mechanism in the transformer block only focuses on the information of the unblocked area in the image, and local prompt fine-tuning is performed in this area to reduce the influence of other areas and effectively learn the local features of the AU:

[0056]

[0057] At the Lth layer, the feature vector cL that can capture the local information of the AU is obtained. After the Lth layer, the local attention mask is removed, and prompt fine-tuning is performed under the global attention mask to improve the effect of AU recognition by using global information:

[0058]

[0059] Each subsequent layer processes the image and prompt output of the previous layer:

[0060]

[0061] The above formula is used to obtain the feature vector cG of the global information of the image in the last layer of the transformer. Finally, cL and cG are fused and projected into the common vector space of visual text to obtain the final image feature x.

[0062] Step 3: For classification, the text prompt is a manually designed class label y∈{with,without} with two categories, and the temperature parameter τ is used to calculate the prediction corresponding to the image I with the highest cosine similarity score (sim(·))

[0063]

[0064] AU recognition is a classic binary classification problem, which means assigning an object to one of two classes. For AU recognition, it can be divided into two categories: presence and absence.

[0065] The present invention adopts a binary cross entropy loss function, which is calculated by comparing the probability value output by the model with the binary value (0 or 1) of the true label, and using the result as the loss value. The cross entropy loss of the output label l (which can be 0 or 1) and the predicted probability p is defined as:

[0066]

[0067] In Prompt Learning, the present invention captures the local features of AUs by controlling the predefined attention masks of different AU prior positions for regional learning. At the same time, in order to make full use of the intrinsic connections between AUs, the global information is further integrated to improve the AU recognition performance.

Claims

1. A method for facial action unit recognition based on CLIP and multimodal mask cue learning, characterized in that: The steps include: Step 1: Design a multimodal shared AU prompt (AU-prompt) and attention mask based on the AU prior position. The AU-prompt contains multiple learnable tokens. The attention mask is designed according to the prior positions of facial muscle changes of different AUs to obtain a specific AU attention mask. Step 2: In the process of multimodal cue learning, the attention mask of a specific AU is introduced into different layers of the text and image encoders of the CLIP model for cue learning. At the same time, the local detail features of the AU and the global correlation features between different AUs are learned to obtain local features and global features. Step 3: Fuse local features and global features, and calculate the loss with text features to finally achieve AU recognition.

2. The method for facial action unit recognition based on CLIP and multimodal mask cue learning according to claim 1, characterized in that: In step 1, the multimodally shared AU prompt (AU-prompt) is shared in the text and visual branches of the CLIP model, and the prompt of the text branch and the prompt of the visual branch are obtained through linear mapping functions respectively. The prompt of the text branch and the prompt of the visual branch are respectively concatenated with the text features and the image features, and input into the transformer for encoding; Set the text linear mapping function to F t (·) The visual linear mapping function is F v (·), then the prompt P of the text branch i t And the visual branch prompt P i v They are defined as: Among them: AU-prompt P of each layer i Contains z learnable tokens, which are represented by: P i ={p1,p2,...,p z }, where i is the number of transformer layers, d t is the dimension of the text word vector feature, d v is the dimension of image features.

3. The method for facial action unit recognition based on CLIP and multimodal mask prompt learning according to claim 2, characterized in that: CLIP's text encoder is a Transformer model through the self-attention mechanism, and the image encoder is the Vision Transformer architecture; If the text encoder T has G transformer layers During prompt fine-tuning, only the first J layers (J ≤ G) are added with learnable text. And combined with the previous layer text feature word vector W i-1 Concatenate and feed into transformer block (T i ), the text input of each layer of transformer can be obtained as: Subsequent layers of the text encoder no longer add learnable prompts, only the previous layer is used and word vector W j-1 The output of is calculated as follows: The same image encoder V has G transformer layers Then the input of each layer of transformer in the image encoder can be obtained as: where c i-1 is the feature vector that captures the image information of the previous layer, E i-1 is the feature vector of the previous layer image, is the hint vector added by the previous layer in the vision branch.

4. The method for facial action unit recognition based on CLIP and multimodal mask cue learning according to claim 3, characterized in that: The attention mask covers the non-AU area, and the generation method of the attention mask includes: masking according to the facial muscle change area of ​​the specific AU, encoding the AU area as 1 and encoding the non-AU area as 0; making the mask completely cover the specific AU area under different postures.

5. The method for facial action unit recognition based on CLIP and multimodal mask cue learning according to claim 3, characterized in that: The step 2 of multimodal mask prompt learning is specifically as follows: In the first few layers of the visual transformer, the attention mask is concatenated with the image features. The attention mechanism is used to make the model focus only on the information of the unoccluded areas in the image, so that the model only focuses on the specific areas of the AU in the first few layers of the visual transformer, thereby learning the local features of the AU. The mask is removed in the next few layers, and the global information is used to improve the effect of AU recognition.

6. The method for facial action unit recognition based on CLIP and multimodal mask cue learning according to claim 3, characterized in that: Set M i-1 is the AU attention mask introduced in the previous layer of the image encoder V, then the input and output of the attention mask layer are expressed as: At the Lth layer, we get the feature vector c that can capture the local information of the AU L After the L layer, the local attention mask is removed, and prompt fine-tuning is performed under the global attention mask to improve the effect of AU recognition by using global information: The image features E of the previous layer are processed in the subsequent transformer layer j-1 And the output information of the prompt Extract global features of AU:

7. The method for facial action unit recognition based on CLIP and multimodal mask cue learning according to claim 1, characterized in that: The feature fusion method of step 3 includes: splicing local features and global features; projecting the spliced ​​features to the common vector space of visual text through a linear mapping function, and using a binary cross entropy loss function. The calculation method of the loss function is to compare the probability value output by the model with the binary value (0 or 1) of the true label, and use the result as the loss value. The cross entropy loss of the AU input label y (negative class is 0, positive class is 1) and the AU predicted positive class probability p is:

Images