Dynamic continuous facial action unit intensity estimation method and system based on multi-scale graph structure, medium and equipment

Abstract

The invention relates to the field of computer vision, and discloses a dynamic continuous face action unit intensity estimation method and system based on a multi-scale image structure, a medium and equipment, and the method comprises the steps: carrying out the feature extraction of an input face video frame sequence, and obtaining a multi-scale space-time difference feature; generating corresponding multi-scale AU node features based on the multi-scale space-time difference features, and dynamically generating an AU graph structure under each scale by using a dynamic graph constructor in combination with frame-level correlation and semantic constraints; performing weighted fusion on the node features of the multi-scale AU graph, and realizing adaptive fusion of different scale node features through a learnable gating coefficient to obtain fused AU feature representation; and inputting the fused AU features into a time sequence encoder, modeling the AU time sequence relationship in the video sequence, and mapping the time sequence features into the intensity value of each target AU. According to the invention, continuous, accurate and time sequence consistency estimation of each AU intensity in the face video can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, medium, and device for estimating the intensity of dynamic continuous facial action units based on a multi-scale graph structure. Background Technology

[0002] With the increasing integration of emotion-sensing technologies into fields such as healthcare, education, and human-computer interaction, the demand for high-precision and robust facial expression analysis technologies is growing. Facial expression analysis plays a crucial role in applications such as remote patient monitoring, intelligent tutoring systems, and emotional robots. One of the core tasks of facial motion analysis is to identify the activation state and intensity of specific facial muscle groups (action units, AUs) based on the Facial Action Coding System (FACS), which is essential for a refined understanding of facial behavior.

[0003] However, facial expression and motion analysis (AU) in unconstrained videos still faces multiple challenges: facial movements are subtle and diverse, spatiotemporal dependencies are complex, and consistency of prediction results across the temporal dimension is crucial. Most existing AU analysis methods are based on static images, modeling dependencies only within a single frame. To more accurately locate AU-related regions, some studies have divided the face into multiple local regions or guided feature extraction based on facial keypoints. While these strategies improve performance to some extent, they rely on fixed region divisions and are susceptible to keypoint detection errors, especially under significant head pose changes. More critically, frameworks based on static spatial features struggle to capture the dynamic changes in facial expressions over time, thus limiting their ability to model AU evolution and co-occurrence activation patterns. Temporal consistency between adjacent video frames can enhance the stability of AU recognition and improve the robustness of the analysis. Therefore, recent research has begun to introduce temporal modeling into AU recognition tasks, but many methods remain insufficiently sensitive to subtle temporal changes or fail to distinguish the differences in the contribution of actions at different spatial scales to different AUs. Effective AU analysis should be able to adaptively capture facial dynamics across multiple spatial scales, identifying both significant muscle movements and subtle local changes. Unlike existing multi-scale methods that typically employ a uniform graph structure, scale-specific graph structures can model AU dependencies at a finer granularity, resulting in higher adaptability and accuracy.

[0004] Compared to AU detection, AU intensity estimation is more challenging, requiring precise capture of continuous temporal changes in facial muscles. This task not only determines the presence or absence of AUs but also estimates subtle changes in intensity throughout the entire process, from the onset of an expression to its peak and decay. This demands a model that is highly sensitive to local appearance changes and possesses accurate facial expression dynamics modeling capabilities. However, AU intensity annotation is complex and prone to introducing noise. Most existing datasets use discrete level annotations (e.g., 0–5 levels). While this simplifies the natural, continuous changes in facial expressions, it is susceptible to label noise and increased intra-class variability due to the subjective differences of the annotators, leading to overfitting and decreased cross-individual generalization ability. To overcome these issues, the FEAFA series datasets introduce continuous AU intensity annotation based on floating-point numbers, laying the data foundation for fine-grained facial expression dynamics analysis.

[0005] In summary, this study proposes a continuous facial action unit intensity estimation method that can fully integrate multi-scale features and dynamic graph structure information, and possess long-term temporal modeling capabilities. This method has significant research value and broad social application prospects for improving the accuracy of fine-grained expression analysis and promoting the development of fields such as emotion computing and rehabilitation assessment. Especially in scenarios such as medical rehabilitation, mental health monitoring, and human-computer interaction, it can provide highly reliable technical support for the accurate quantification and dynamic tracking of facial movement changes. Therefore, there is an urgent need for a facial action unit intensity estimation method that can efficiently model multi-scale spatial relationships and long-term temporal dependencies to meet the current pressing needs of intelligent interactive applications for high accuracy and generalization capabilities. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a method, system, medium, and device for estimating the intensity of dynamic continuous facial action units based on a multi-scale graph structure. This method can simultaneously capture multi-scale spatial dependence and long-range temporal information, enabling continuous, accurate, and temporally consistent estimation of the intensity of each action unit (AU) in a face video.

[0007] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: a dynamic continuous facial action unit intensity estimation method based on a multi-scale graph structure, comprising: extracting features from an input facial video frame sequence to obtain multi-scale spatiotemporal difference features; generating corresponding multi-scale AU node features based on the multi-scale spatiotemporal difference features, and combining frame-level correlation and semantic constraints, dynamically generating AU graph structures at each scale using a dynamic graph builder; performing weighted fusion of the node features of the multi-scale AU graph, and achieving adaptive fusion of node features at different scales through learnable gating coefficients to obtain the fused AU feature representation; inputting the fused AU features into a temporal encoder to model the temporal relationship of AUs in the video sequence, and mapping the temporal features to the intensity values ​​of each target AU.

[0008] Furthermore, feature extraction is performed on the input face video frame sequence to obtain multi-scale spatiotemporal difference features, including: using ResNet-50 as the backbone network to extract hierarchical facial feature maps from shallow to deep layers; For each scale of facial feature sequence, the temporal difference between adjacent frames is calculated to capture fine-grained dynamic changes, and the dynamic features at different scales are processed with uniform resolution within the temporal window through two-dimensional convolution and three-dimensional average pooling operations. For each scale of dynamic features, a spatial attention map is generated using a convolution-based attention weight generation module, and scale weights are calculated to represent the importance of the dynamic features at that scale to the overall representation. Static global features are extracted from the final layer of the backbone network and mapped to a unified representation space through a fully connected layer to ensure consistency with the multi-scale dynamic features in terms of dimension and semantic space. The static global features are added element-wise to the weighted multi-scale dynamic features to obtain multi-scale features that integrate global appearance information and dynamic changes.

[0009] Furthermore, based on multi-scale spatiotemporal difference features, corresponding multi-scale AU node features are generated. Combining frame-level correlation and semantic constraints, a dynamic graph builder is used to dynamically generate AU graph structures at each scale, including: For multi-scale features extracted from shallow to deep, the differences between shallow local texture and high-level semantic representation are utilized to construct the corresponding AU feature distribution and dynamic association structure at each scale. From the feature map at each scale, the corresponding AU node features are extracted by N parallel AU feature extractors; By introducing a learnable AU correlation matrix, matrix multiplication is performed on the features of AU nodes to enhance the structured representation of nodes and capture co-occurrence patterns and dependencies between AUs. At each scale, a frame-level AU dependency adjacency matrix is ​​dynamically constructed based on AU node features. The adjacency matrix is ​​obtained by projecting node features into query vectors and key vectors respectively, and using a multi-head scaling dot product attention mechanism to calculate the similarity between nodes. The frame-level adjacency matrix sequence is input into the temporal smoothing module based on the ConvGRU convolutional gated recurrent unit, and the hidden state is updated through local two-dimensional convolution to obtain a temporally smoothed adjacency matrix. The smoothed adjacency matrix is ​​normalized to obtain a directed graph structure with normalized transition probabilities. The AU node features are then updated by graph propagation based on the directed graph to capture the subtle and time-evolving dependencies between AUs.

[0010] Furthermore, the node features of the multi-scale AU graph are weighted and fused. Learnable gating coefficients are used to achieve adaptive fusion of node features at different scales, resulting in a fused AU feature representation, including: For each scale of AU node features, a learnable gating function is set to control the information flow of features at that scale during the fusion process; the gating function consists of a linear transformation and a sigmoid activation order, and outputs a gating coefficient vector with the same dimension as the input features. The AU node features at the corresponding scale are weighted element-wise by gating coefficients to suppress irrelevant features and enhance key information. The gated weighted AU node features at each scale are summed across all scales to obtain the fused AU node feature representation.

[0011] Furthermore, the timing encoder consists of bidirectional Mamba modules used to capture long-term timing dependencies.

[0012] Furthermore, the fused AU features are input into the temporal encoder to model the temporal relationships of AUs in the video sequence, including: The fused frame-by-frame AU node features are input into a multi-layer bidirectional Mamba encoder to model the feature sequence of each AU in the time dimension. In each Mamba encoding block, the input features are normalized by RMSNorm and mapped to a higher-dimensional latent space through a linear transformation. A one-dimensional convolution is applied to the mapped feature sequence to introduce local contextual information. A dual-stream coding strategy is adopted, in which one stream processes the feature sequence in the forward time direction, and the other stream processes the feature sequence in the reverse time direction. Then, the output of the reverse stream is reversed again in time direction and added element by element to the output of the forward stream to achieve symmetrical bidirectional temporal context modeling.

[0013] Furthermore, the temporal features are mapped to the intensity values ​​of each target AU, including: Establish a set of learnable prototype vectors that correspond one-to-one with the target AU. The prototype vectors are used to represent the feature directions of each target AU. The AU node features and prototype vectors are subjected to L2 normalization to ensure numerical stability and geometric alignment. The cosine similarity is calculated based on the normalized AU node features and the corresponding AU prototype vector, and a learnable bias term is introduced to obtain the intensity estimate of each target AU in each frame. The prototype vector is shared across all frames to guide the model to learn a consistent semantic direction for the same AU in the temporal dimension, thereby improving the generalization ability across time frames.

[0014] Secondly, the technical solution adopted by this invention is as follows: a dynamic continuous facial action unit intensity estimation system based on a multi-scale graph structure, comprising: a feature extraction module, which extracts features from the input face video frame sequence to obtain multi-scale spatiotemporal difference features; an AU graph generation module, which generates corresponding multi-scale AU node features based on the multi-scale spatiotemporal difference features, and dynamically generates AU graph structures at each scale by combining frame-level correlation and semantic constraints using a dynamic graph builder; an adaptive fusion module, which performs weighted fusion of node features of the multi-scale AU graph, and achieves adaptive fusion of node features at different scales through learnable gating coefficients to obtain the fused AU feature representation; and an intensity acquisition module, which inputs the fused AU features into a temporal encoder, models the temporal relationship of AUs in the video sequence, and maps the temporal features to the intensity values ​​of each target AU.

[0015] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0016] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0017] The present invention has the following advantages due to the adoption of the above technical solutions: 1. This invention can make full use of multi-scale facial features and characterize the time-varying dependency relationship between AUs through a scale-specific dynamic graph structure, which has stronger adaptability and robustness compared with fixed graph structure methods.

[0018] 2. This invention employs a gated fusion mechanism to achieve dynamic weighted combination of multi-scale features, effectively balancing the contribution of different scales to the final prediction, thereby improving the accuracy of AU intensity estimation.

[0019] 3. This invention, combined with a bidirectional temporal modeling module, can capture the continuous changing trend of AU intensity in video sequences and generate smooth prediction results with temporal consistency.

[0020] In summary, this invention can be widely applied to scenarios such as affective computing, human-computer interaction, and medical rehabilitation assessment, meeting the application requirements of high precision and strong generalization ability, and has significant scientific research value and social benefits. Attached Figure Description

[0021] Figure 1This is an overall flowchart of the dynamic continuous facial motion unit intensity estimation method in this embodiment of the invention; Figure 2 This is a detailed flowchart of the dynamic continuous facial motion unit intensity estimation method in an embodiment of the present invention; Figure 3 This is a feature visualization example for AU intensity estimation in an embodiment of the present invention, showing the spatial and temporal feature focus areas of the input face sequence data at different network depths. Detailed Implementation

[0022] To address the limitations of existing AU intensity estimation methods, such as limited multi-scale feature interaction, weak adaptability to dynamic graph structures, and insufficient capture of temporal dependencies, this invention provides a dynamic continuous facial action unit intensity estimation method based on a multi-scale graph structure. The method includes: preprocessing facial expression videos by face detection and alignment to retain video frame sequences with detectable face regions; extracting multi-scale features from the video frame sequences using a deep convolutional neural network to obtain multi-scale features that integrate global face information and dynamic differences at different scales; generating AU nodes for each scale feature, combining feature similarity and learnable parameters to generate the AU graph structure at that scale, and performing frame-level temporal smoothing to obtain AU node features at different levels and their dynamic relationships; using a gated fusion mechanism to adaptively weight and merge the multi-scale AU node features to obtain a fused temporal feature sequence; using a bidirectional temporal feature extractor to perform long-range temporal modeling on the fused features to generate enhanced temporal AU representations; and inputting the enhanced temporal features into an AU regression module to output the intensity estimates of each AU over continuous time.

[0023] This invention uses convolutional neural networks and dynamic graph neural networks for multi-scale feature extraction and structure construction, and combines a gating weighting mechanism to achieve adaptive fusion of multi-scale graph features. In terms of temporal modeling, it combines ConvGRU and bidirectional Mamba structures to efficiently capture the local and global temporal dependencies of AU sequences, thereby improving the ability to model subtle dynamic changes in long-sequence facial expressions. Based on prototype learning, it performs regression prediction to achieve accurate estimation of continuous AU intensity.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] In one embodiment of the present invention, a dynamic continuous facial action unit (AU) intensity estimation method based on a multi-scale graph structure is provided. The aim is to estimate the intensity values ​​of the facial action units of the person in the video frame by frame after inputting a video of facial expression actions to be detected. During the training phase, the input to this method is a sequence of pre-processed facial expression action video frames and their corresponding AU intensity value labels for each frame. During the testing phase, the input to this method is the facial expression action video, and the output is the AU intensity estimation value corresponding to each video frame.

[0027] Specifically, in this embodiment, as follows: Figure 1 , Figure 2 As shown, the dynamic continuous AU intensity estimation method includes the following steps: 1) Perform preprocessing such as face detection and alignment on facial expression videos to retain video frame sequences in which face regions can be detected; extract features from the input face video frame sequences to obtain multi-scale spatiotemporal difference feature representations; 2) Generate corresponding multi-scale AU node features based on multi-scale spatiotemporal difference features, and combine frame-level correlation and semantic constraints to dynamically generate AU graph structures at each scale using a dynamic graph builder; 3) Weighted fusion of node features in multi-scale AU graphs is performed. Learnable gating coefficients are used to achieve adaptive fusion of node features at different scales to obtain the fused AU feature representation. 4) Input the fused AU features into the temporal encoder to model the temporal relationship of AUs in the video sequence and map the temporal features to the intensity values ​​of each target AU. The temporal encoder is composed of bidirectional Mamba modules to capture long-term temporal dependencies; each AU corresponds to a learnable prototype vector.

[0028] In step 1) above, the facial expression video is preprocessed with face detection and alignment, and the T-frame video frame sequence with detectable face regions is retained. Specifically, OpenCV is used to extract video frames, and Dlib is used for face detection, alignment and cropping. The image sequence with effective face regions is used as the model input, and the AU intensity value label corresponding to each frame of the facial expression video is read.

[0029] In this embodiment, the FEAFA dataset (FEAFA: A well-annotated dataset for facial expression analysis and 3D facial animation) was used for training and validation. This dataset is the first to provide fine-grained AU intensity labels, providing 24 redefining AU floating-point intensity values ​​([0,1]) for each frame of all collected video data. The dataset contains 123 video sequences collected from 122 Asian subjects of different genders and ages. All videos were filmed in natural scenes, totaling 99,356 frames with complete manual annotation. During testing, facial videos could be captured using any ordinary RGB camera, or pre-recorded videos of facial expressions and movements could be uploaded.

[0030] In step 1) above, feature extraction is performed on the input face video frame sequence to obtain multi-scale spatiotemporal difference features, including the following steps: 1.1) ResNet-50 is used as the backbone network to extract hierarchical facial feature maps from shallow to deep layers.

[0031] Specifically, different semantic levels are extracted from the input video frame sequence from the backbone network. The feature representations are denoted as follows: The shallow features retain fine-grained texture variations, while the deep features contain abstract semantic information.

[0032] 1.2) Facial feature sequences at each scale Calculate the temporal difference between adjacent frames respectively. This method captures fine-grained dynamic changes and uses 2D convolution and 3D average pooling operations to perform uniform resolution processing on dynamic features at different scales within a temporal window. Indicates the first Frame of face image in the first Scale characteristics of the layer The total number of frames in the input face image video; Represents the real number field; Indicates the first The number of channels for layer-scale features; Indicates the first The height of layer-scale features; Indicates the first Width of layer-scale features.

[0033] Specifically, scale-dependent 2D convolution and 3D average pooling are used to perform uniform resolution alignment and dynamic difference aggregation within a time window (k frames before and after) to obtain a scale-specific dynamic feature sequence. The calculation formula is as follows:

[0034] in, The kernel size and stride are The average 3D pooling layer kernel size is ; This indicates the height of the feature scale of the 4th layer, which is the deepest layer. This indicates the width of the 4th layer, which is the deepest layer of scale features; Indicates the first Frame in Dynamic characteristics of layers; This represents the feature dimension.

[0035] 1.3) For the dynamic features at each scale, a spatial attention map is generated using a convolution-based attention weight generation module, and the scale weight is calculated to represent the importance of the dynamic features at that scale to the overall representation.

[0036] Specifically, a lightweight attention weighting mechanism is introduced to weight the dynamic features at each scale. Calculate spatial attention weights The importance of dynamic features at different scales is adaptively assigned, and its calculation formula is as follows:

[0037] 1.4) Extract static global features from the final layer of the backbone network and map them to a unified representation space through a fully connected layer to ensure consistency with the multi-scale dynamic features in terms of dimension and semantic space.

[0038] Specifically, projection is performed through fully connected layers to obtain a unified representation that is compatible with multi-scale dynamic features in both dimensionality and semantic space. .

[0039] 1.5) Static global features The elements are added to the weighted multi-scale dynamic features to obtain multi-scale features that integrate global appearance information and dynamic changes.

[0040] Specifically, static global features The scale-aware fusion features are obtained by fusing the weighted dynamic features element by element. The calculation formula is as follows:

[0041] In this embodiment, ResNet-50 is used as the backbone network to perform four-layer multi-scale feature extraction on the input 16-frame image sequence. For the difference sequence at each scale, resolution alignment and dynamic difference aggregation are performed within a time window of 5 frames before and after the difference sequence. After projecting the global static features and dynamic features onto a unified semantic space dimension, i.e. Then, a weighted merging process is performed.

[0042] In step 2) above, corresponding multi-scale AU node features are generated based on multi-scale spatiotemporal difference features. Combining frame-level correlation and semantic constraints, a dynamic graph builder is used to dynamically generate the AU graph structure at each scale, including the following steps: 2.1) For the multi-scale features extracted from shallow to deep, the differences between shallow local texture and high-level semantic representation are used to construct the corresponding AU feature distribution and dynamic association structure at each scale.

[0043] 2.2) Extract the corresponding AU node features from the feature map at each scale using N parallel AU feature extractors.

[0044] Specifically, the fusion features for each scale ,pass A parallel AU linear transformer is used to extract the corresponding AU node features, resulting in node features at each scale. ,in, Indicates the first The AU in the first Feature nodes of the layer Indicates the number of AUs.

[0045] 2.3) By introducing a learnable AU correlation matrix, matrix multiplication is performed on the features of AU nodes to enhance the structured representation ability between nodes and capture the co-occurrence patterns and dependencies between AUs.

[0046] 2.4) At each scale, a frame-level AU dependency adjacency matrix is ​​dynamically constructed based on the AU node features. The adjacency matrix is ​​obtained by projecting the node features into query vectors and key vectors respectively, and using a multi-head scaling dot product attention mechanism to calculate the similarity between nodes.

[0047] Specifically, for node features at each scale Project them respectively into query vectors With key vector The similarity between AU nodes is calculated using a multi-head scaling dot product attention mechanism to obtain the attention adjacency matrix for each frame. The calculation process is as follows:

[0048]

[0049] in, Indicates the number of attention heads; It is a single-head dimension; , These represent linear transformations that map node features to query embeddings and key embeddings, respectively. For the first Frame in The attention adjacency matrix of the layer; Indicates the first Frame number Layer A query vector with attention heads; Indicates the first Frame number Layer The key vector of the attention head. 2.5) Input the frame-level adjacency matrix sequence into the temporal smoothing module based on the ConvGRU convolutional gated recurrent unit, and update the hidden state through local two-dimensional convolution to obtain a temporally smoothed adjacency matrix.

[0050] Specifically, to maintain temporal consistency in the relationships between AU nodes, a lightweight smoothing mechanism based on ConvGRU is adopted, which smooths the frame-level adjacency matrix at each scale. Inputting the ConvGRU module yields the updated hidden state:

[0051] in, That is, the adjacency matrix after time-series smoothing. .

[0052] 2.6) For the smoothed adjacency matrix Graph normalization is performed to obtain a directed graph structure with normalized transition probabilities, and AU node features are then analyzed based on the directed graph. Perform graph propagation updates to obtain enhanced node representations, capturing subtle and time-evolving dependencies between AUs.

[0053] Specifically, the enhanced nodes Represented as:

[0054] In step 3) above, the node features of the multi-scale AU graph are weighted and fused. Adaptive fusion of node features at different scales is achieved through learnable gating coefficients to obtain the fused AU feature representation. This includes the following steps: 3.1) For each scale of AU node features, a learnable gating function is set to control the information flow of the features at that scale during the fusion process; wherein, the gating function consists of a linear transformation and a sigmoid activation order, and outputs a gating coefficient vector of the same dimension as the input features.

[0055] 3.2) The AU node features at the corresponding scale are weighted element by element by gating coefficients to suppress irrelevant features and enhance key information.

[0056] 3.3) The gated weighted AU node features at each scale are summed across all scales to obtain the fused AU node feature representation.

[0057] In this embodiment, specifically for the AU node features at each scale... The system generates dimension-wise weights through a gating function consisting of a linear layer and a sigmoid activation function. These weights are then used to weight and merge features at each scale to obtain the final multi-scale fused features. The calculation process is as follows:

[0058] in, and For learnable parameters, For the Sigmoid function; Indicates the first l The weight vector of layer node features. Indicates the first l The constant bias term of the layer node features.

[0059] In step 4) above, the timing encoder is composed of bidirectional Mamba modules, which are used to capture long-term timing dependencies.

[0060] In this embodiment, the fused AU features are input into a temporal encoder to model the temporal relationship of AUs in the video sequence, including the following steps: 4.1.1) Input the fused frame-by-frame AU node features into a multi-layer bidirectional Mamba encoder to model the feature sequence of each AU in the time dimension.

[0061] 4.1.2) In each Mamba coding block, the input features are normalized by RMSNorm and mapped to a higher-dimensional latent space by a linear transformation. A one-dimensional convolution is applied to the mapped feature sequence to introduce local contextual information.

[0062] Specifically, for multi-scale fusion features RMSOrm normalization is performed, and the representation is projected to a high-dimensional latent space through a linear mapping to enhance the representational power. A one-dimensional convolution per channel is introduced in the time dimension to capture local contextual dependencies, denoted as node features. .

[0063] 4.1.3) To simultaneously model forward and backward time dependencies, a dual-stream coding strategy is adopted. One stream processes the feature sequence in the forward time direction, while the other stream processes the feature sequence in the reverse time direction. Then, the output of the reverse stream is reversed again in time direction and added element by element to the output of the forward stream to achieve symmetrical bidirectional temporal context modeling.

[0064] Specifically, the output of the reverse flow is inverted again and added element-wise to the result of the forward flow to obtain a bidirectional enhanced representation. The calculation formula is as follows:

[0065] In this embodiment, within the Mamba Block, an efficient State Space Model (SSM) mechanism is employed to achieve temporal modeling through recursive state updates. Its core update form is as follows:

[0066] Where A, B, C, and D are learnable state transition matrices. These are dynamic weighting coefficients based on the input (obtained via softplus activation). As input features, It is in a hidden state. This is the output feature. This update process captures long-range dependencies while maintaining linear computational complexity.

[0067] In step 4) above, the temporal features are mapped to the intensity values ​​of each target AU, including: 4.2.1) Establish a set of learnable prototype vectors that correspond one-to-one with the target AU. The prototype vectors are used to represent the feature directions of each target AU.

[0068] 4.2.2) Perform L2 normalization on the AU node features and prototype vectors respectively to ensure numerical stability and geometric alignment.

[0069] 4.2.3) Based on the normalized AU node features and the corresponding AU prototype vector, the cosine similarity is calculated, and a learnable bias term is introduced to obtain the intensity estimate of each target AU in each frame.

[0070] The prototype vector is shared across all frames to guide the model to learn a consistent semantic direction for the same AU in the temporal dimension, thereby improving the generalization ability across time frames.

[0071] In this embodiment, specifically, a prototype vector is preset for each AU, resulting in a prototype set. To ensure numerical stability and to make the AU node feature P consistent with the prototype vector To maintain geometric consistency, the cosine similarity is calculated after L2 normalization of the two samples, and a learnable bias term is introduced to obtain the final AU intensity estimate. The calculation process is as follows:

[0072]

[0073] In this embodiment, the computer executing the algorithm uses an AMD EPYC 7742 CPU, runs Ubuntu 22.04 with CUDA version 12.9, and employs PyTorch version 2.5 as the neural network architecture. During training, the AdamW optimizer is used with an initial learning rate of 1e-4, a weight decay of 5e-4, a batch size of 6, and a total of 30 training epochs. Parameters can be adjusted appropriately based on different training set sizes. After training, the network weights are obtained. In the testing phase, the input is a video of the face to be tested, and the AU intensity estimation network obtains the final output.

[0074] In summary, this invention enables dynamic continuous estimation of AU intensity values ​​for facial expression videos.

[0075] To verify the effectiveness and practicality of the method proposed in this invention, an example on the FEAFA dataset is given below. Table 1 shows the ablation experiment results of the example, with the evaluation metrics being MAE (Mean Absolute Error) and ICC (Intraclass Correlation Coefficient).

[0076] Table 1 shows the ablation experimental results of the examples on the FEAFA dataset.

[0077] As shown in Table 1, the model performance gradually improved after the introduction of multi-scale feature extraction, dynamic graph construction and temporal feature extraction modules. In the temporal feature extraction module, TCN and Bi-Mamba were compared and verified to show the advantages of the bidirectional temporal mechanism in capturing the evolution of dynamic expressions.

[0078] Table 2 shows the AU intensity estimation results of the example on the validation set compared with MDHR, ME-Graph, and RA-UWML. Among them, RA-UWML is an algorithm for AU intensity estimation, while the others are AU detection methods. To ensure fairness, this invention replaces its classification head with a regression head and changes the loss function from cross-entropy to mean squared error, while keeping the rest of the network structure and training method unchanged to adapt to the regression task of AU intensity estimation.

[0079] Table 2 shows the evaluation results of this invention on the FEAFA dataset using the ICC and MAE metrics.

[0080] As shown in Table 2, the method proposed in this invention significantly outperforms existing methods in terms of regression accuracy and label consistency. In particular, it achieves the best ICC performance on 11 out of 24 AUs, and makes significant improvements on key AUs (e.g., AU1 / 2 - left and right eyes closed, AU12 / 13 - left / right corners of the mouth turned up, AU19 / 20 - upper lip up / lower lip down, AU23 - cheek puffing), fully demonstrating the advantages and effectiveness of this invention in continuous AU intensity estimation tasks.

[0081] Depend on Figure 3 As can be seen, the proposed method model focuses on areas that are highly consistent with the facial regions where key AUs are located, such as wrinkling the nose, closing the eyes, and mouth movements, demonstrating good temporal tracking capabilities. At the same time, shallow features focus more on local fine areas (such as the corners of the eyes and mouth), while deep features capture more overall facial dynamics. The combination of the two enhances the representation ability of complex expressions.

[0082] In one embodiment of the present invention, a dynamic continuous facial motion unit intensity estimation system based on a multi-scale graph structure is provided, comprising: The feature extraction module extracts features from the input face video frame sequence to obtain multi-scale spatiotemporal difference features; The AU graph generation module generates corresponding multi-scale AU node features based on multi-scale spatiotemporal difference features, and combines frame-level correlation and semantic constraints to dynamically generate AU graph structures at each scale using a dynamic graph builder. The adaptive fusion module performs weighted fusion of node features in multi-scale AU graphs. It achieves adaptive fusion of node features at different scales through learnable gating coefficients to obtain the fused AU feature representation. The intensity acquisition module inputs the fused AU features into the temporal encoder to model the temporal relationship of AUs in the video sequence and maps the temporal features to the intensity values ​​of each target AU.

[0083] In the above embodiments, feature extraction is performed on the input face video frame sequence to obtain multi-scale spatiotemporal difference features, including: using ResNet-50 as the backbone network to extract hierarchical facial feature maps from shallow to deep layers; For each scale of facial feature sequence, the temporal difference between adjacent frames is calculated to capture fine-grained dynamic changes, and the dynamic features at different scales are processed with uniform resolution within the temporal window through two-dimensional convolution and three-dimensional average pooling operations. For each scale of dynamic features, a spatial attention map is generated using a convolution-based attention weight generation module, and scale weights are calculated to represent the importance of the dynamic features at that scale to the overall representation. Static global features are extracted from the final layer of the backbone network and mapped to a unified representation space through a fully connected layer to ensure consistency with the multi-scale dynamic features in terms of dimension and semantic space. The static global features are added element-wise to the weighted multi-scale dynamic features to obtain multi-scale features that integrate global appearance information and dynamic changes.

[0084] In the above embodiments, multi-scale AU node features are generated based on multi-scale spatiotemporal difference features, and combined with frame-level correlation and semantic constraints, a dynamic graph builder is used to dynamically generate AU graph structures at each scale, including: For multi-scale features extracted from shallow to deep, the differences between shallow local texture and high-level semantic representation are utilized to construct the corresponding AU feature distribution and dynamic association structure at each scale. From the feature map at each scale, the corresponding AU node features are extracted by N parallel AU feature extractors; By introducing a learnable AU correlation matrix, matrix multiplication is performed on the features of AU nodes to enhance the structured representation of nodes and capture co-occurrence patterns and dependencies between AUs. At each scale, a frame-level AU dependency adjacency matrix is ​​dynamically constructed based on AU node features. The adjacency matrix is ​​obtained by projecting node features into query vectors and key vectors respectively, and using a multi-head scaling dot product attention mechanism to calculate the similarity between nodes. The frame-level adjacency matrix sequence is input into the temporal smoothing module based on the ConvGRU convolutional gated recurrent unit, and the hidden state is updated through local two-dimensional convolution to obtain a temporally smoothed adjacency matrix. The smoothed adjacency matrix is ​​normalized to obtain a directed graph structure with normalized transition probabilities. The AU node features are then updated by graph propagation based on the directed graph to capture the subtle and time-evolving dependencies between AUs.

[0085] In the above embodiments, the node features of the multi-scale AU graph are weighted and fused. Adaptive fusion of node features at different scales is achieved through learnable gating coefficients to obtain the fused AU feature representation, including: For each scale of AU node features, a learnable gating function is set to control the information flow of features at that scale during the fusion process; the gating function consists of a linear transformation and a sigmoid activation order, and outputs a gating coefficient vector with the same dimension as the input features. The AU node features at the corresponding scale are weighted element-wise by gating coefficients to suppress irrelevant features and enhance key information. The gated weighted AU node features at each scale are summed across all scales to obtain the fused AU node feature representation.

[0086] In the above embodiments, the timing encoder is composed of a bidirectional Mamba module, which is used to capture long-term timing dependencies.

[0087] In this embodiment, the fused AU features are input into a temporal encoder to model the temporal relationship of AUs in the video sequence, including: The fused frame-by-frame AU node features are input into a multi-layer bidirectional Mamba encoder to model the feature sequence of each AU in the time dimension. In each Mamba encoding block, the input features are normalized by RMSNorm and mapped to a higher-dimensional latent space through a linear transformation. A one-dimensional convolution is applied to the mapped feature sequence to introduce local contextual information. A dual-stream coding strategy is adopted, in which one stream processes the feature sequence in the forward time direction, and the other stream processes the feature sequence in the reverse time direction. Then, the output of the reverse stream is reversed again in time direction and added element by element to the output of the forward stream to achieve symmetrical bidirectional temporal context modeling.

[0088] In this embodiment, the temporal features are mapped to the intensity values ​​of each target AU, including: Establish a set of learnable prototype vectors that correspond one-to-one with the target AU. The prototype vectors are used to represent the feature directions of each target AU. The AU node features and prototype vectors are subjected to L2 normalization to ensure numerical stability and geometric alignment. The cosine similarity is calculated based on the normalized AU node features and the corresponding AU prototype vector, and a learnable bias term is introduced to obtain the intensity estimate of each target AU in each frame. The prototype vector is shared across all frames to guide the model to learn a consistent semantic direction for the same AU in the temporal dimension, thereby improving the generalization ability across time frames.

[0089] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0090] In one embodiment of the present invention, a computing device is provided. This computing device can be a terminal and may include a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. When the computer programs are executed by the processor, they implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0091] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0093] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0094] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0095] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic continuous facial action unit intensity estimation based on multi-scale graph structure, characterized in that, Comprise: Feature extraction is performed on the input face video frame sequence to obtain multi-scale spatio-temporal difference features; Based on the multi-scale spatio-temporal difference features, the corresponding multi-scale AU node features are generated, and the frame-level correlation and semantic constraints are combined to dynamically generate the AU graph structure at each scale by using the dynamic graph constructor; The node features of the multi-scale AU graph are weighted and fused, and the adaptive fusion of the node features of different scales is realized through the learnable gating coefficient to obtain the fused AU feature representation; The fused AU feature is input into the time sequence encoder to model the AU time sequence relationship in the video sequence, and the time sequence feature is mapped to the intensity value of each target AU. 2.The method of claim 1, wherein, Feature extraction is performed on the input face video frame sequence to obtain multi-scale spatio-temporal difference features, including: using ResNet-50 as the backbone network to extract hierarchical face feature maps from shallow to deep; For each scale of face feature sequence, the time sequence difference between adjacent frames is calculated to capture fine-grained dynamic changes, and two-dimensional convolution and three-dimensional average pooling operations are used to process dynamic features of different scales within the time sequence window to uniform resolution; For each scale of dynamic feature, a spatial attention map is generated using a convolution-based attention weight generation module, and a scale weight is calculated to represent the importance of the scale dynamic feature to the overall representation; The static global feature is extracted from the final layer of the backbone network and mapped to a unified representation space through a fully connected layer to ensure consistency in dimension and semantic space with the multi-scale dynamic feature; The static global feature and the weighted multi-scale dynamic feature are added element by element to obtain a multi-scale feature that combines global appearance information and dynamic changes. 3.The method of claim 1, wherein, Based on the multi-scale spatio-temporal difference features, the corresponding multi-scale AU node features are generated, and the frame-level correlation and semantic constraints are combined to dynamically generate the AU graph structure at each scale by using the dynamic graph constructor, including: For multi-scale features extracted from shallow to deep, the difference in local texture at shallow level and semantic representation at high level is utilized to construct the corresponding AU feature distribution and dynamic association structure at each scale; From each scale of feature map, the corresponding AU node feature is extracted through N parallel AU feature extractors; By introducing a learnable AU correlation matrix, matrix multiplication is performed on the AU node features to enhance the structured representation ability between nodes and capture the co-occurrence patterns and dependency relationships between AUs; At each scale, a frame-level AU dependency adjacency matrix is dynamically constructed based on the AU node features, the adjacency matrix is obtained by projecting the node features into query vectors and key vectors respectively, and a multi-head scaled dot-product attention mechanism is used to calculate the similarity between nodes to obtain a dense attention matrix; The sequence of frame-level adjacency matrices is input into the time sequence smoothing module based on the ConvGRU convolution gating recurrent unit, and the local two-dimensional convolution is used to update the hidden state to obtain the time-smoothed adjacency matrix; The smoothed adjacency matrix is normalized to obtain a directed graph structure with normalized transition probabilities, and the AU node features are updated through graph propagation based on the directed graph to capture the subtle and time-evolving dependency relationships between AUs. 4.The method of claim 1, wherein, The node features of the multi-scale AU graph are weighted and fused, and adaptive fusion of the node features of different scales is realized through learnable gating coefficients to obtain the fused AU feature representation, including: For the AU node features of each scale, a learnable gating function is set to control the information flow of the features in the fusion process; the gating function is composed of linear transformation and Sigmoid activation in sequence, and outputs a gating coefficient vector with the same dimension as the input feature; The AU node features of the corresponding scale are element-wise weighted through the gating coefficient to suppress irrelevant features and enhance key information; The AU node features of each scale after the gating and weighting process are summed between scales to obtain the fused AU node feature representation. 5.The method of claim 1, wherein, The timing encoder is composed of a bidirectional Mamba module, which is used to capture long-term timing dependencies. 6.The method of claim 5, wherein, The fused AU features are input into the timing encoder to model the AU timing relationship in the video sequence, including: The fused frame-by-frame AU node features are input into the multi-layer bidirectional Mamba encoder to model the feature sequence of each AU in the time dimension; In each Mamba encoding block, the input features are normalized by RMSNorm, and are mapped to a higher-dimensional latent space through linear transformation, and one-dimensional convolution is applied to the mapped feature sequence to introduce local context information; A double-stream encoding strategy is adopted, in which one stream processes the feature sequence in the forward direction in time, and the other stream processes the feature sequence in the reverse direction in time, then the output of the reverse stream is again time-reversed, and is element-wise added to the output of the forward stream to realize symmetric bidirectional timing context modeling.

7. The method of claim 6, wherein the method further comprises: determining a plurality of facial action unit intensities based on the plurality of facial landmarks; and determining a plurality of facial action unit intensities based on the plurality of facial landmarks. The timing features are mapped to the intensity values of each target AU, including: A set of learnable prototype vectors corresponding to each target AU is established, and the prototype vectors are used to represent the feature direction of each target AU; The AU node features and the prototype vectors are normalized by L2 respectively to ensure numerical stability and geometric alignment; The cosine similarity between the normalized AU node features and the corresponding AU prototype vectors is calculated, and a learnable bias term is introduced to obtain the intensity estimate of each target AU in each frame; The prototype vectors are shared across all frames to guide the model to learn consistent semantic directions for the same AU in the time dimension, thereby improving the generalization ability across time frames.

8. A multi-scale graph structure based dynamic continuous facial action unit intensity estimation system, characterized in that, including: The feature extraction module extracts features from the input face video frame sequence to obtain multi-scale spatio-temporal difference features; The AU graph generation module generates corresponding multi-scale AU node features based on the multi-scale spatio-temporal difference features, and combines frame-level relevance and semantic constraints to dynamically generate AU graph structures at each scale using a dynamic graph constructor; The adaptive fusion module weights and fuses the node features of the multi-scale AU graph, and realizes adaptive fusion of the node features of different scales through learnable gating coefficients to obtain the fused AU feature representation; The intensity acquisition module inputs the fused AU features into the timing encoder to model the AU timing relationship in the video sequence, and maps the timing features to the intensity values of each target AU.

9. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method of any of claims 1-8. The one or more programs, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-7.

10. A computing device, comprising: comprise: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods of claims 1-7.

Images