Face action unit intensity recognition method and system based on cross attention fusion

Abstract

The invention discloses a face action unit intensity recognition method and system based on cross attention fusion. In time domain analysis, an ordered regression neural network based on a visual self-attention model is constructed and is used for estimating the appearance sequence of facial expressions. In spatial domain analysis, a region of interest (ROI) neural network based on face landmark points is constructed for estimating the intensity of each AU from a specified local region. In addition, a cross-attention module is proposed to explore associations between global semantics from a time domain and local semantics from a spatial domain. According to the method, the organization process of training data is simplified, the calculation complexity of the model is also reduced, redundant information in space is eliminated to a certain extent, semantic association between local ROIs is also explored, and therefore the intensity recognition capacity of all AUs is effectively enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and more specifically, relates to a method and system for recognizing the intensity of facial action units based on cross-attention fusion. Background Art

[0002] AU (Action Unit) is a core component in the field of facial expression recognition and analysis. It was created by human facial expression scientists Paul Ekman and Wallace V. Friesen in the 1970s. AU is mainly used to describe and quantify the degree of activity of muscle groups in different parts of human facial expressions. These activities involve subtle changes in facial areas such as eyebrows, eyes, nose, and mouth. Each action unit in the face is assigned a unique code. For example, AU2 represents the raising of the outer eyebrows; AU26 represents the drooping of the lower jaw, etc. AU intensity is quantified into 6 ordered levels (0-5, 0 represents the weakest intensity) during annotation, and annotations are provided.

[0003] The Transformer model initially achieved great success in the field of natural language processing (NLP), such as the BERT and GPT models. In the field of computer vision (CV), the ViT model (Vision Transformer) maintains spatial information by dividing the input image into small blocks and converting it into a vector form, combined with position encoding. The encoder of the ViT model is one of its core structures, which mainly includes several key parts: layer normalization (Layer Norm), multi-head attention mechanism (Multi-Head Attention), random inactivation layer (Dropout) and multi-layer perceptron layer (MLP). The multi-layer stacked encoder is called the Encoder Block layer of the ViT model. Layer Norm stabilizes model training and accelerates convergence through normalization technology. The Multi-Head Attention mechanism allows the model to capture the complex relationship between the parts of the image and understand the local features and overall structural information. Dropout, as a regularization method, prevents overfitting by randomly discarding neurons. MLP extracts features independently for each position to enhance the model's expressiveness. These components work together to improve the ViT model's ability to process and understand image content.

[0004] Facial landmarks refer to a series of coordinate points marked on key parts of the face, which are used to accurately depict and analyze facial structure and dynamics. Among them, the 68-point landmark is a widely used standard configuration, which covers detailed location information of areas such as eyes, eyebrows, nose, lips and facial contours. These points can not only help identify individual features, but also support a variety of functions such as facial expression analysis, posture correction, virtual reality interaction, etc., playing a core role in face recognition, medical plastic surgery, entertainment technology and other fields. Through advanced technologies such as deep learning, modern landmark detection systems can efficiently and accurately locate these 68 key points, greatly improving the accuracy and speed of facial information processing.

[0005] In the past, many methods for studying AU intensity also started from temporal and spatial features, such as using dynamic graph convolution (DGCNN) or graph convolutional neural network (GCN) to consider the co-occurrence characteristics between AUs, and some modeling data based on the occurrence relationship of AUs, using a self-supervised method to predict the intensity of AUs. Or using convolutional neural network (CNN) to establish a time module to specifically extract the temporal features of AUs. However, existing AU intensity recognition methods have the disadvantages of complex training and insufficient accuracy. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a facial action unit intensity recognition method and system based on cross-attention fusion, which can effectively improve the intensity recognition ability of AU.

[0007] To achieve the above object, according to one aspect of the present invention, a method for recognizing the intensity of facial action units based on cross-attention fusion is provided, wherein facial action unit intensity recognition is performed based on a trained deep learning model, and the training of the deep learning model comprises the steps of:

[0008] Collect sample pairs from a training image sequence, each sample pair comprising two image frame samples with different intensity values, mark intensity labels on the facial action units contained in the two image frame samples in the sample pair, input one image frame sample in the sample pair as a first image frame sample into a first branch of a timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the first image frame sample, input the other image frame sample in the sample pair as a second image frame sample into a second branch of the timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the second image frame sample;

[0009] Cutting the ROI area of ​​the first image frame sample in the sample pair, sending the ROI area cut out of the first image frame sample to the spatial analysis module, and obtaining the second intensity prediction value of the facial action unit contained in the first image frame sample;

[0010] A sorting loss is calculated according to the image intensity sorting values ​​of the two image frame samples, a temporal branch loss is calculated according to the first intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the two image frame samples, and a spatial branch loss is calculated according to the second intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the first image frame samples. The sum of the sorting loss, temporal branch loss, and spatial branch loss of all sample pairs is used as the loss function of the deep learning model, and the deep learning model is iteratively trained based on the loss function.

[0011] Further, after the first branch of the temporal analysis module obtains the temporal coding features of the first image frame sample and the spatial analysis module obtains the spatial coding features of the first image frame sample, a cross attention weight is calculated based on the temporal coding features and the spatial coding features of the first image frame sample;

[0012] The first branch of the timing analysis module outputs a first intensity prediction value and a picture intensity ranking value of a facial action unit contained in the first image frame sample based on the timing coding feature of the first image frame sample and the cross attention weight;

[0013] The spatial analysis module outputs a second intensity prediction value of a facial action unit contained in the first image frame sample based on the spatial encoding features and cross-attention weights of the first image frame sample.

[0014] Furthermore, the facial action unit intensity recognition based on the trained deep learning model includes the steps of:

[0015] Acquire the image frame to be identified, input the image frame to be identified into the timing analysis module, crop the ROI area from the image frame to be identified, input the ROI area of ​​the image frame to be identified into the spatial analysis module, and use the output features of the timing analysis module as the recognition strength of the image frame to be identified.

[0016] Furthermore, the timing analysis module is a twin network, and parameters of the first branch and the second branch of the timing analysis module are shared.

[0017] Furthermore, each branch of the timing analysis module includes a ViT network, a global average pooling layer and two fully connected layers, the first output of the ViT network of the timing analysis module is connected to the input of the global average pooling layer, the output of the global average pooling layer is connected to a fully connected layer, and the second output of the ViT network of the timing analysis module is connected to another fully connected layer. The spatial analysis module includes a ViT network, a global average pooling layer and a fully connected layer, the output of the ViT network of the spatial analysis module is connected to the input of the global average pooling layer, and the output of the global average pooling layer is connected to the fully connected layer. The ViT network of the first branch of the timing analysis module is connected to the ViT network of the spatial analysis module through a cross attention module.

[0018] Furthermore, the calculation formula of the sorting loss is:

[0019]

[0020] in, represents the sorting loss, max represents the maximum value operation, Represents the picture intensity ranking value of the first image frame sample in the nth sample pair, represents the picture intensity ranking value of the second image frame sample in the nth sample pair, and N represents the number of sample pairs.

[0021] Furthermore, the calculation formula of the timing branch loss is:

[0022]

[0023] in, represents the timing branch loss, represents the first intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, represents the intensity label vector of the face action unit of the first image frame sample in the nth sample pair, represents the first intensity prediction vector of the face action unit of the second image frame sample in the nth sample pair, The intensity label vector representing the facial action unit of the second image frame sample in the nth sample pair;

[0024] The calculation formula of spatial branch loss is:

[0025]

[0026] in, represents the spatial branch loss, represents the second intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, The second intensity label vector representing the facial action unit of the first image frame sample in the nth sample pair, where N represents the number of sample pairs.

[0027] Furthermore, the sum of the intensities of the facial action units contained in the image frame sample is taken as the intensity value of the image frame sample.

[0028] Furthermore, the method of collecting sample pairs from the training image sequence includes the steps of: dividing the entire aligned face image sequence into a plurality of monotonically increasing and monotonically decreasing training image sequences according to the intensity increase or decrease trend of each image sample frame, and performing fixed-interval sampling on each training image sequence to obtain sample pairs.

[0029] According to another aspect of the present invention, a facial action unit intensity recognition system based on cross-attention fusion is provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the methods.

[0030] In general, the above technical solutions conceived by the present invention have beneficial effects compared with the prior art:

[0031] 1. The present invention uses a single timing analysis module to extract global expression timing information to assist the intensity estimation task of multiple AUs. Compared with building a timing analysis module for each AU, this not only simplifies the organization process of training data, but also reduces the computational complexity of the model. At the same time, the globality of timing information is also used to supplement the spatial semantics outside the ROI.

[0032] 2. The present invention uses a feature point-guided ROI-based spatial analysis module to extract local spatial information of each AU. This model eliminates redundant spatial information to a certain extent and explores the semantic association between local ROIs, thereby enhancing the semantic representation ability of each AU. The ROI-based spatial analysis module uses the same network structure as the temporal analysis module, which is also conducive to the integration of the temporal analysis module and the spatial analysis module.

[0033] 3. The present invention uses cross attention to fuse spatial semantics and temporal semantics. By exploring the association between global semantics and local semantics, and the association between spatial semantics and temporal semantics, the performance of AU intensity estimation is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the working principle of the method for recognizing the strength of facial action units according to an embodiment of the present invention;

[0035] Figure 2Schematic diagram of the working principle of the timing analysis module of the facial action unit intensity recognition method according to an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the working principle of the spatial analysis module of an embodiment of the present invention.

[0037] Figure 4 4 is a schematic diagram of ROI clipping according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0039] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. "Multiple" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0040] Unless otherwise specified, "plurality" means two or more.

[0041] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0042] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] The embodiments of the present invention provide a method and system for recognizing facial action unit intensity based on cross-attention fusion, which are described below respectively.

[0045] like Figure 1 As shown, a facial action unit intensity recognition method based on cross-attention fusion in an embodiment of the present invention performs facial action unit intensity recognition based on a trained deep learning model.

[0046] The training of a deep learning model consists of the following steps:

[0047] Collect sample pairs from a training image sequence, each sample pair comprising two image frame samples with different intensity values, mark intensity labels on the facial action units contained in the two image frame samples in the sample pair, input one image frame sample in the sample pair as a first image frame sample into a first branch of a timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the first image frame sample, input the other image frame sample in the sample pair as a second image frame sample into a second branch of the timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the second image frame sample;

[0048] Cutting the ROI area of ​​the first image frame sample in the sample pair, sending the ROI area cut out of the first image frame sample to the spatial analysis module, and obtaining the second intensity prediction value of the facial action unit contained in the first image frame sample;

[0049] A sorting loss is calculated according to the image intensity sorting values ​​of the two image frame samples, a temporal branch loss is calculated according to the first intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the two image frame samples, and a spatial branch loss is calculated according to the second intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the first image frame samples. The sum of the sorting loss, temporal branch loss, and spatial branch loss of all sample pairs is used as the loss function of the deep learning model, and the deep learning model is iteratively trained based on the loss function.

[0050] For the timing analysis module, it is necessary to construct data sample pairs and send them into the timing analysis module.

[0051] Further, before the start of model training, image preprocessing is performed on each frame of the training image sequence, including: face detection and face alignment. The image preprocessing outputs an aligned face image sequence with the same size and consistent binocular coordinates, and its size is 224×224×3. The aligned face images are respectively input into the temporal analysis module and the spatial analysis module.

[0052] Further, the sum of the face action unit intensities included in the image frame sample is used as the intensity value of the image frame sample.

[0053] The specific steps for constructing the sample pairs are as follows:

[0054] (1) Calculate the expression intensity of the image frame by frame, and the calculation formula can be expressed as:

[0055]

[0056] where Y represents the expression intensity of the image, and y k represents the intensity of the k-th AU, and the value range of k is {1, 2,..., K}.

[0057] (2) According to the calculation results of step (1), select the segments where the images are monotonically increasing or decreasing from the image sequence. For the convenience of subsequent unified processing, reverse the order of the monotonically decreasing segments to obtain segments where the image intensity is monotonically increasing.

[0058] (3) From each segment obtained in step (2), sample multiple sample pairs. The sample pairs are represented by the binary group (i, j), and the collected sample pairs satisfy i < j. According to monotonicity, it can be known that Y i < Y j , where Y i and Y j respectively represent the image expression intensities of the i-th and j-th frames of the segment.

[0059] Preferably, the sample pairs are sampled at a fixed interval of 3 frames, that is, j = i + 3.

[0060] Input the constructed sample pairs into the temporal analysis module.

[0061] The temporal analysis module is a siamese network, and the parameters of the first branch and the second branch of the temporal analysis module are shared.

[0062] The specific steps for training the siamese network are as follows:

[0063] (1) Input the constructed sample pairs into the two branches of the siamese network. For example Figure 2As shown in Figure 1, the twin network consists of two branches with exactly the same structure, named the first branch and the second branch. The first branch is used to process sample i in the sample pair (i, j), and the second branch is used to process sample j in the sample pair (i, j). Since the two branches of the twin network share parameters, the pictures sent to the two branches will be processed exactly the same. The total intensity value output by the first branch is p1, and the prediction vector of each AU intensity is output as The total strength value output by the second branch is p2, and the prediction vector of each AU strength is output as

[0064] Furthermore, the branch is composed of ViT, global average pooling, FC_P and FC_Q, where FC_P and FC_Q represent two independent fully connected layers, followed by a number 1 or 2 to distinguish the first branch from the second branch.

[0065] Furthermore, ViT has two types of outputs, class token and patch token. The class token output by the first branch is connected to the input of FC_P1, and FC_P1 outputs p1; the class token output by the second branch is connected to the input of FC_P2, and FC_P2 outputs p2. The patch token output by the first branch is connected to the input of global average pooling, and the output of global average pooling is connected to the input of FC_Q1, and FC_Q1 outputs The patch token output by the second branch is connected to the input of the global average pooling, and the output of the global average pooling is connected to the input of FC_Q2.

[0066] (2) Calculate the sorting loss and train the model using the gradient descent algorithm to minimize the sorting loss. The specific calculation formula is:

[0067]

[0068] Where N represents the size of a training mini-batch, max(·,·) is the maximum value function, and the superscript n in p1 and p2 represents the nth sample pair in the training mini-batch. Represents the picture intensity ranking value of the first image frame sample in the nth sample pair, Represents the picture intensity ranking value of the second image frame sample in the nth sample pair.

[0069] (3) Calculate the AU intensity estimation loss, and train the model through the gradient descent algorithm to minimize the AU intensity estimation loss. It means that the calculation formula is:

[0070]

[0071] Among them, N represents the size of a training mini-batch, that is, the number of sample pairs, and the superscript n represents the nth sample pair in the training mini-batch. represents the first intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, represents the intensity label vector of the face action unit of the first image frame sample in the nth sample pair, represents the first intensity prediction vector of the face action unit of the second image frame sample in the nth sample pair, The intensity label vector representing the facial action unit of the second image frame sample in the nth sample pair. Computes the squared l2 norm of a vector.

[0072] For the training data processing of the spatial analysis module, the image frame needs to be cut into ROI areas. The number of cut ROI areas is related to the number of AU labels and the type of AU given in the training data. Specifically, each AU cuts the corresponding ROI area. However, due to the symmetry of the human face, each AU can cut two ROI areas on the face. Some AUs with similar positions can also share a ROI area.

[0073] The selection of ROI is based on the facial landmark points and the inner eye distance. The inner eye distance is calculated by the distance between the two nearest landmark points around the eyes. The positions of different ROIs are found according to the correspondence between AU and facial landmark points.

[0074] In one embodiment, the ROI area is located according to the facial feature points, and the specific cropped ROI area is as follows: Figure 4 shown.

[0075] Furthermore, the model structure adopted by the spatial analysis module is also the ViT model. According to the number and type of AU labels given in the data set, each image is cut into several ROI areas.

[0076] Cut the ROI area of ​​an image frame sample in the sample pair, and input all the cut areas into the spatial branch to estimate the AU intensity. The network structure of the spatial branch is as follows: Figure 3 As shown in the right branch, it includes ViT_R, global average pooling layer, and fully connected layer FC_R. The specific connection method is that the patch token output of ViT_R is connected to the input of the global average pooling layer, and the output of the global average pooling layer is connected to the input of the fully connected layer FC_R. The fully connected layer FC_R outputs the strength estimation vector of the spatial branch.

[0077] Calculate the AU intensity regression loss, and train the model through the gradient descent algorithm to minimize the AU intensity regression loss. It means that the calculation formula is:

[0078]

[0079] Where N represents the size of a training mini-batch, and the superscript n represents the nth sample in the training mini-batch. represents the second intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, The second intensity label vector representing the facial action unit of the first image frame sample in the nth sample pair.

[0080] Preferably, the structures of ViT_R and FC_R in the spatial branch are consistent with those of ViT_1 and FC_Q1 in the twin network, but the values ​​and training of the network parameters are independent.

[0081] Furthermore, the ViT network of the first branch of the temporal analysis module is connected to the ViT network of the spatial analysis module through a cross attention module.

[0082] Further, after the first branch of the temporal analysis module obtains the temporal coding features of the first image frame sample and the spatial analysis module obtains the spatial coding features of the first image frame sample, a cross attention weight is calculated based on the temporal coding features and the spatial coding features of the first image frame sample;

[0083] The first branch of the timing analysis module outputs a first intensity prediction value and a picture intensity ranking value of a facial action unit contained in the first image frame sample based on the timing coding feature of the first image frame sample and the cross attention weight;

[0084] The spatial analysis module outputs a second intensity prediction value of a facial action unit contained in the first image frame sample based on the spatial encoding features and cross-attention weights of the first image frame sample.

[0085] In other words, the query value of ViT of a branch of the temporal analysis module is selected and passed to the spatial analysis module as the query value of the spatial analysis module ViT; the query value in the spatial analysis module ViT is passed to the temporal analysis module as the query value of the temporal analysis module ViT, and the attention weight is calculated based on the exchanged query value. The specific exchange process is as follows:

[0086] Define the image frame i to get X after passing through the Encoder Block layer of the temporal branch ViT model B, the dimension size is 197×768. The ROI area of ​​the image frame i is defined and sent to the spatial branch and passed through the Encoder Block layer to obtain X R , whose dimension is 19×768. Then the formula of the cross attention model is:

[0087]

[0088] X ′ B =Attention T (Q R ,K B ,V B )

[0089] X ′ R =Attention ROI (Q B ,K R ,V R )

[0090] Q B , K B and V B The eigenvectors X are B The linear change of Q R , K R and V R The eigenvectors X are R The linear change of and are the parameters to be learned by the model, and their dimensions are all 768 × 768. d represents the dimensions of the query vector and the key vector. The purpose is to make the order of magnitude of the dot product not too large, thereby avoiding the gradient vanishing problem of the Softmax function. After the cross attention module, the feature X is obtained. ′ B With feature X ′ R , and the feature dimensions of the two branches remain unchanged.

[0091] Furthermore, feature X ′ B Enter the timing branch, feature X ′ R The FC-Q1 layer in the temporal branch FC layer and the FC-R layer in the spatial branch FC layer output the AU intensity estimation values ​​of the temporal branch and the spatial branch respectively.

[0092] Furthermore, facial action unit intensity recognition is performed based on the trained deep learning model, including the steps of: acquiring an image frame to be recognized, inputting the image frame to be recognized into a timing analysis module, cropping an ROI area from the image frame to be recognized, inputting the ROI area of ​​the image frame to be recognized into a spatial analysis module, and using the output features of the timing analysis module as the recognition intensity of the image frame to be recognized.

[0093] In the process of model testing, there is no need to construct sample pairs. The image frame k of the test set is sent to the time branch, and the ROI area cropped from the image frame k is sent to the space branch. Since the space branch is used to assist in enhancing the performance of the model and selects local features, the final statistical model result is the result of the time branch.

[0094] The DISFA dataset is used for training and testing. The DISFA dataset contains 27 videos of 27 different subjects, each with 4844 frames. These videos are from 27 subjects (12 females and 15 males). There are 12 AUs (1, 2, 4, 5, 6, 9, 12, 15, 17, 20, 25, and 26) whose intensity labels are annotated by an expert FACS encoder. Following a three-fold cross-validation scheme, 18 subjects are used for training and 9 subjects are used for testing.

[0095] 1. Experimental evaluation indicators

[0096] The performance of deep learning models is evaluated using intraclass correlation (ICC) and mean absolute error (MAE). ICC measures the consistency of ratings by comparing the differences between different ratings of the same subject to the total differences across all ratings and all subjects. ICC measures the intra-group consistency of each intensity level, and they are all numbers from 0 to 1, the larger the better; MAE is a positive number that measures the deviation between the predicted value and the actual value, and the smaller the value, the better. The ICC(3,1) variant is used. Mean absolute error is a common metric used in regression problems to measure the average difference between the prediction and the target.

[0097] 2. Train the intensity regression model and test it on the test set.

[0098] The training and testing process uses three-fold cross validation, and the average of the three-fold test results is taken as the final result. While ensuring that the training and testing methods and data sets are exactly the same, the results are compared with other model results. The results in the DISFA data set are shown in Table 1:

[0099] Table 1

[0100]

[0101]

[0102] The results show that the method disclosed in the present invention is superior to other existing methods, indicating that the present invention can effectively utilize temporal information and spatial information to enhance the results of facial AU intensity estimation.

[0103] It also verifies whether each module has an effect during the experiment. During the verification process, the ICC and MAE indicators of the model with only the time module, only the space module, and both the time module and the space module are tested. The results are shown in Table 2:

[0104] Table 2

[0105] Time module Space Module ICC MAE × × 0.632 0.210 × √ 0.646 0.209 √ × 0.642 0.205 √ √ 0.651 0.204

[0106] The model works better when adding time branch and space branch separately than when using only ViT model, and the model works better when adding time branch and space branch at the same time than adding time branch or space branch alone.

[0107] A facial action unit intensity recognition system based on cross-attention fusion according to an embodiment of the present invention comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes any step of a facial action unit intensity recognition method based on cross-attention fusion.

[0108] The implementation principle and technical effect of the facial action unit intensity recognition system based on cross-attention fusion in the embodiment of the present invention are the same as those of the above method and will not be repeated here.

[0109] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A facial action unit intensity recognition method based on cross-attention fusion, It is characterized in that The intensity of facial action units is recognized based on the trained deep learning model, wherein the training of the deep learning model comprises the following steps: Collect sample pairs from a training image sequence, each sample pair comprising two image frame samples with different intensity values, mark intensity labels for facial action units contained in the two image frame samples in the sample pair, input one image frame sample in the sample pair as a first image frame sample into a first branch of a timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the first image frame sample, input the other image frame sample in the sample pair as a second image frame sample into a second branch of the timing analysis module, obtain a first intensity prediction value and a picture intensity ranking value of the facial action unit contained in the second image frame sample; Cutting the ROI area of ​​the first image frame sample in the sample pair, sending the ROI area cut out of the first image frame sample to the spatial analysis module, and obtaining the second intensity prediction value of the facial action unit contained in the first image frame sample; A sorting loss is calculated according to the image intensity sorting values ​​of the two image frame samples, a temporal branch loss is calculated according to the first intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the two image frame samples, and a spatial branch loss is calculated according to the second intensity prediction values ​​and the corresponding intensity labels of the facial action units contained in the first image frame samples. The sum of the sorting loss, temporal branch loss, and spatial branch loss of all sample pairs is used as the loss function of the deep learning model, and the deep learning model is iteratively trained based on the loss function.

2. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that After the first branch of the temporal analysis module obtains the temporal coding features of the first image frame sample and the spatial analysis module obtains the spatial coding features of the first image frame sample, a cross attention weight is calculated based on the temporal coding features and the spatial coding features of the first image frame sample; The first branch of the timing analysis module outputs a first intensity prediction value and a picture intensity ranking value of a facial action unit contained in the first image frame sample based on the timing coding feature of the first image frame sample and the cross attention weight; The spatial analysis module outputs a second intensity prediction value of a facial action unit contained in the first image frame sample based on the spatial encoding features and cross-attention weights of the first image frame sample.

3. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that The method of performing facial action unit intensity recognition based on the trained deep learning model includes the following steps: Acquire the image frame to be identified, input the image frame to be identified into the timing analysis module, crop the ROI area from the image frame to be identified, input the ROI area of ​​the image frame to be identified into the spatial analysis module, and use the output features of the timing analysis module as the recognition strength of the image frame to be identified.

4. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that The timing analysis module is a twin network, and parameters of a first branch and a second branch of the timing analysis module are shared.

5. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 4, It is characterized in that Each branch of the timing analysis module includes a ViT network, a global average pooling layer and two fully connected layers. The first output of the ViT network of the timing analysis module is connected to the input of the global average pooling layer, the output of the global average pooling layer is connected to a fully connected layer, and the second output of the ViT network of the timing analysis module is connected to another fully connected layer. The spatial analysis module includes a ViT network, a global average pooling layer and a fully connected layer. The output of the ViT network of the spatial analysis module is connected to the input of the global average pooling layer, and the output of the global average pooling layer is connected to the fully connected layer. The ViT network of the first branch of the timing analysis module is connected to the ViT network of the spatial analysis module through a cross attention module.

6. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that The calculation formula of sorting loss is: in, represents the sorting loss, max represents the maximum value operation, Represents the picture intensity ranking value of the first image frame sample in the nth sample pair, represents the picture intensity ranking value of the second image frame sample in the nth sample pair, and N represents the number of sample pairs.

7. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that The calculation formula of timing branch loss is: in, represents the timing branch loss, represents the first intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, represents the intensity label vector of the face action unit of the first image frame sample in the nth sample pair, represents the first intensity prediction vector of the face action unit of the second image frame sample in the nth sample pair, A strength label vector representing the facial action unit of the second image frame sample in the nth sample pair; The calculation formula of spatial branch loss is: in, represents the spatial branch loss, represents the second intensity prediction vector of the face action unit of the first image frame sample in the nth sample pair, The second intensity label vector representing the facial action unit of the first image frame sample in the nth sample pair, where N represents the number of sample pairs.

8. The method for recognizing facial action unit intensity based on cross-attention fusion as claimed in claim 1, It is characterized in that The sum of the intensities of the facial action units contained in the image frame sample is taken as the intensity value of the image frame sample.

9. The method for recognizing facial action unit intensity based on cross attention fusion as claimed in claim 1, It is characterized in that The method of collecting sample pairs from the training image sequence includes the steps of: dividing the entire aligned face image sequence into a plurality of monotonically increasing and monotonically decreasing training image sequences according to the intensity increase or decrease trend of each image sample frame, and sampling each training image sequence at fixed intervals to obtain sample pairs.

10. A facial action unit intensity recognition system based on cross-attention fusion, It is characterized in that The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 9.

Images