Joint expression localization and recognition method and device based on aggregated features and dual anchoring

Abstract

This invention discloses a joint expression localization and recognition method and apparatus based on aggregated features and dual anchoring. Addressing the need for fine-grained expression analysis of single faces in first-person view videos, it constructs an end-to-end deep learning framework to achieve unified localization and recognition of macro and micro expressions. The method first processes the acquired single-face video data through a frame-by-frame AU recognition network that aggregates local and global features. Combining the advantages of convolutional neural networks and visual Transformers, multi-scale features are extracted and AU codes are generated. Then, a dual-stream 3D convolutional network assisted by AU codes is used to fuse spatiotemporal features, combined with a multi-scale sliding window strategy and a feature pyramid network, and a dual anchoring mechanism is proposed to jointly locate expression intervals. Finally, video segments are extracted based on the localization results, and classification is completed through an expression localization and recognition network. This invention effectively solves the problems of multiple coexisting expressions, large duration differences, and difficulties in micro-expression detection in videos through local-global feature aggregation, dual-stream 3D convolutional networks, and dual anchoring strategies, significantly improving the accuracy and robustness of expression localization and recognition.

Description

Technical Field

[0001] This invention relates to a method and apparatus for joint facial expression localization and recognition based on aggregated features and dual anchoring, belonging to the field of computer vision technology. Background Technology

[0002] In fields such as interpersonal interaction analysis, psychological state assessment, and intelligent security, accurate localization and recognition of facial expressions are core technologies for understanding human emotional intentions. With the widespread adoption of video capture devices and breakthroughs in deep learning technology, computer vision-based facial expression analysis technology is gradually moving towards practical applications. Especially in scenarios such as social robots, mental health assessment, and remote conferencing emotion analysis, systems need to possess the ability to accurately capture and understand expressions in natural interaction scenarios. Traditional psychological research categorizes facial expressions into macro-expressions, which are relatively long-lasting (0.5-4 seconds) and involve significant movements, and micro-expressions, which are brief (<0.5 seconds) and low-intensity, resulting from emotional suppression or concealment. While these two types of expressions differ significantly in action unit (AU) activation patterns and temporal dynamics, both can reflect an individual's true emotional state, making the joint detection and recognition of macro / micro expressions a cutting-edge topic in affective computing.

[0003] Early facial expression recognition methods primarily relied on hand-designed features and shallow classifiers (such as SVMs), classifying expressions by extracting local texture or geometric features from static images. However, these methods are sensitive to changes in lighting and head pose, and struggle to model the temporal dynamics of facial expressions. With the development of deep learning, methods based on 2D / 3D convolutional neural networks (CNNs) have achieved joint optimization of feature representation and classification through end-to-end learning. 3D CNNs directly model short-term motion patterns in video sequences using spatiotemporal convolutional kernels, while Transformer-based architectures utilize self-attention mechanisms to capture long-range dependencies. To further enhance fine-grained facial expression analysis capabilities, researchers introduced a multi-task learning framework that combines AU (Facial Action Unit) detection with facial expression classification, utilizing the local muscle movement information of the AU to assist in facial expression recognition.

[0004] In terms of facial expression localization technology, mainstream methods can be divided into two categories: candidate interval generation based on sliding windows and temporal detection based on anchors. The former traverses the video sequence through multi-scale windows, but it has high computational complexity and is difficult to handle facial expression segments with significant differences in length; the latter borrows the anchor box mechanism from object detection, predefines anchor intervals of different durations for regression optimization, but has insufficient adaptability to extremely short or extremely long expressions. In addition, existing methods often process facial expression localization and recognition as independent modules in stages, leading to problems of fragmented temporal information and error accumulation.

[0005] Despite significant progress in existing methods, practical applications still face challenges such as insufficient synergy between local and global features, limitations in multi-scale temporal modeling, and a lack of joint optimization frameworks. Feature aggregation techniques and multi-task joint learning have become crucial breakthroughs in overcoming these bottlenecks. Feature Pyramid Networks (FPNs) enhance the model's ability to detect targets of different sizes through multi-scale feature fusion, while the combination of dynamic anchoring mechanisms and anchorless methods improves the flexibility of temporal detection. Simultaneously, models based on a two-stream architecture strengthen spatiotemporal modeling capabilities through heterogeneous feature complementarity. In the field of micro-expression analysis, explorations are underway to utilize AU detection as an intermediate supervisory signal, enhancing the interpretability of expression recognition through fine-grained muscle motion encoding. Summary of the Invention

[0006] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a joint expression localization and recognition method and device based on aggregated features and dual anchoring. It can collaboratively process macro / micro expression localization in multi-expression overlapping scenarios through dual anchoring mechanism, and improve the detection sensitivity of expressions of different durations by combining AU encoding and multi-scale spatiotemporal aggregation. While achieving end-to-end joint optimization, it ensures high-precision recognition and localization stability.

[0007] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0008] A joint expression localization and recognition method based on aggregated features and dual anchoring is proposed. Addressing the need for fine-grained expression analysis of single faces in first-person view videos, an end-to-end deep learning framework is constructed to achieve unified localization and recognition of macro and micro expressions. The method first processes the acquired single-face video data through a frame-by-frame AU recognition network that aggregates local and global features. Combining the advantages of convolutional neural networks and visual Transformers, multi-scale features are extracted and AU codes are generated. Then, a dual-stream 3D convolutional network assisted by AU codes is used to fuse spatiotemporal features, combined with a multi-scale sliding window strategy and a feature pyramid network, and a dual anchoring mechanism is proposed to jointly locate expression regions. Finally, video segments are extracted based on the localization results, and classification is completed through the expression localization and recognition network. The method includes the following steps:

[0009] S01: Perform face tracking on the input first-person perspective video, extract the trajectory of each face in the first-person perspective video, and form individual face videos.

[0010] S02: Perform local-global feature aggregation on each frame of the single-face video to generate AU encoding; the local-global feature aggregation is implemented through cascaded local feature aggregators and global feature aggregators, wherein the local feature aggregator uses a multi-head convolutional module and a multi-layer perceptron module to extract local high-frequency action features, and the global feature aggregator uses a multi-head self-attention module, a multi-head convolutional module and a multi-layer perceptron module to extract global low-frequency face features, and suppresses gradient vanishing of the local feature aggregator and the global feature aggregator through residual connections;

[0011] S03: Reshape and concatenate the AU-encoded time series to convert it into a single-channel temporal feature matrix I. Extract features from a single-face video to obtain a single-channel temporal feature matrix II. Input the single-channel temporal feature matrix I and the single-channel temporal feature matrix II into a two-stream 3D convolutional network for multi-scale spatiotemporal feature fusion to generate a fused expression feature F. (e) ;

[0012] S04: Fusing facial expression features based on a sliding window strategy (e) Multi-scale sampling is performed, and facial expression localization is achieved by combining a feature pyramid network with both anchored and unanchored localization mechanisms. The final facial expression localization is then filtered by non-maximum suppression.

[0013] S05: Extract facial feature segments based on the start and end frames of the final facial expression localization, and input the facial feature segments into the facial expression localization and recognition network to predict the macro-facial expression category and micro-facial expression category;

[0014] S06: By jointly optimizing the AU recognition loss, dual anchoring localization loss and expression recognition loss, train the first-view fine-grained expression localization and recognition unified model consisting of steps S01 to S05;

[0015] S07: Input a first-person perspective video with any number of frames into the trained first-person perspective fine-grained expression localization and recognition unified model to identify macro expression categories and micro expression categories.

[0016] Specifically, in step S02, the single-face video is first input into a 2D convolutional layer for precoding to obtain a precoded feature map, and then processed through N... (lg) A cascaded local-global feature aggregation module simultaneously captures local high-frequency action features and global low-frequency face features of a single face image. This is then processed through a batch normalization layer and a global average pooling layer to obtain the AU encoding F. t(u) Finally, the AU recognition probability is obtained after passing through a fully connected layer. The precoded feature map is used as the input to the first local-global feature aggregation module, the output of the previous local-global feature aggregation module is used as the input to the next local-global feature aggregation module, and the output of the last local-global feature aggregation module is used.

[0017] Each local-global feature aggregation module uses a sub-block embedding layer, N (l) A cascaded local feature aggregator and N (g) The system is implemented using a global feature aggregator. The sub-block embedding layer is used to embed and perform preliminary feature transformation on the input of the local-global feature aggregation module, and map it to a feature space suitable for subsequent processing. The local feature aggregator consists of a multi-head convolution module and a multi-layer perceptron module, and is used to extract local high-frequency action features (reflecting subtle changes in the face). The global feature aggregator consists of a 2D pointwise convolutional layer, a batch normalization layer, a multi-head self-attention module, a multi-head convolution module, and a multi-layer perceptron module, and is used to extract global low-frequency face features (capturing the overall structure and pose changes of the face). The outputs of both the local and global feature aggregators are connected using residual connections to suppress the vanishing gradient problem.

[0018] The sub-block embedding layer embeds and performs preliminary feature transformation on the input of the local-global feature aggregation module, maps it to a feature space suitable for subsequent processing, and uses the resulting feature map as the input of the first local feature aggregator.

[0019] The implementation process of the local feature aggregator is as follows: In the multi-head convolution module, the input feature map is first divided into multiple sub-feature maps along the channel dimension, and 2D convolution operation is performed on each sub-feature map. The convolution results of each sub-feature map are concatenated along the channel dimension, and batch normalization, non-linear activation, and 2D pointwise convolution are performed in sequence to further enhance the feature representation capability and obtain the output of the multi-head convolution module. In the multi-layer perceptron module, the output of the multi-head convolution module is residually connected with the input and used as the input of the multi-layer perceptron module. 2D convolution, non-linear activation, and dropout are performed in sequence to obtain the output of the multi-layer perceptron module, which enhances the expressive capability of local features. The feature map obtained by residually connecting the input and output of the multi-layer perceptron module is used as the output of the local feature aggregator, and the output of the last local feature aggregator is used as the input of the first global feature aggregator.

[0020] The implementation process of the global feature aggregator is as follows: First, the number of input channels is adjusted through a 2D pointwise convolutional layer. Then, it passes through a batch normalization layer and a multi-head self-attention module to capture global low-frequency facial features. The output of the multi-head self-attention module is residually concatenated with the input of the batch normalization layer and used as the input of the multi-head convolutional module. The output of the multi-head convolutional module is residually concatenated with the input and used as the input of the multi-layer perceptron module. The feature map obtained by residually concatenating the input and output of the multi-layer perceptron module is used as the output of the global feature aggregator to accurately represent the global shape and dynamic features of the face. Finally, the output of the global feature aggregator passes through a batch normalization layer and a global average pooling layer to obtain the AU encoding F. t(u) .

[0021] Specifically, in step S02, in order to improve the accuracy of AU recognition, the AU recognition probability is... Using weighted binary cross-entropy loss L (u) Supervision:

[0022]

[0023] in: M represents the probability of AU recognition. (u) Let N represent the total number of AUs, and let N represent the total number of sample images in the training set. This represents the total number of sample images in the training set that contain the j-th AU. w represents the probability of the j-th AU appearing in the training set. j This represents the weight that suppresses frequency imbalances between the j-th AU and other AUs, v j This represents the weights used to suppress the j-th AU from appearing too infrequently (used in the first term of cross-entropy, i.e., the occurrence term). and Let represent the actual occurrence probability and the predicted occurrence probability of the j-th AU in the t-th frame sample image, respectively.

[0024] Weight calculation is used to address the issue of uneven frequency of AU occurrences in AU identification. The weights are in the following form:

[0025]

[0026] in: The weight w represents the total number of sample images in the training set that contain the j-th AU; j This helps to suppress the problem of frequency imbalance among different AUs, giving lower-frequency AUs higher weights, making the model more sensitive to less common AUs, and preventing high-frequency AUs from dominating model training; weight v j This involves further adjusting the frequency of each AU, which helps to suppress the problem that the frequency of each AU is often lower than the frequency of no occurrence.

[0027] Specifically, in step S03, each AU code is initially a vector. To ensure compatibility with temporal data, the AU code for each frame is reshaped from a vector into a single-channel matrix. These matrices are then concatenated to form a single-channel temporal feature matrix I. Single-channel temporal feature matrix I represents the change of AU features in each frame of a single-face video over time. Feature extraction from the single-face video yields a single-channel temporal feature matrix II, which reflects the dynamic changes in the single-face video. The dual-stream 3D convolutional network includes two inputs: the first input is the single-channel temporal feature matrix I, and the second input is the single-channel temporal feature matrix II. The first input extracts spatiotemporal feature I through an independent spatiotemporal feature extraction module, and the second input extracts spatiotemporal feature II through the same module. The spatiotemporal feature I and spatiotemporal feature II are then added to obtain the fused expression feature F. (e) .

[0028] The spatiotemporal feature extraction module extracts spatiotemporal features from different receptive fields using multiple multi-scale 3D convolutional modules, aiming to capture spatial and temporal information at different scales in the video. The module consists of cascaded 3D convolutional layers, max pooling layers, a set of multi-scale 3D convolutional modules, and a global average pooling layer. Each multi-scale 3D convolutional module contains a set of parallel branches, each using 3D convolutional kernels of different sizes (including 3×3×3 and 5×5×5 kernels) to capture information from different receptive fields (different kernel sizes can extract features from different receptive fields, effectively capturing spatiotemporal changes in local details and global structure). The outputs of each branch are concatenated at the channel level as the output of the multi-scale 3D convolutional module. The output of the last multi-scale 3D convolutional module is then dimensionality-reduced using a global average pooling layer to reduce redundant information while extracting spatiotemporal features with different receptive fields. This multi-scale feature extraction method can simultaneously focus on local motion features and global structural features in single-face videos, thereby enhancing sensitivity to facial expression changes.

[0029] Specifically, in step S04, the fused facial expression features F are first processed based on a sliding window strategy. (e) Multi-scale sampling is performed, and sliding windows of different lengths are set to cover the typical duration of macro and micro expressions. Two adjacent sliding windows in the time dimension have partial overlap. T window frames are uniformly sampled from single-face video segments in different sliding windows to obtain different window frame sequences.

[0030] Along the fusion facial feature F (e)After applying a sliding window in the time dimension, the window frame sequence is first encoded using 1D convolutional layers and max pooling layers. Then, a feature pyramid network with L levels is input. In each level of the feature pyramid network, a dual localization mechanism is executed in parallel, using 1D convolutional layers to generate anchored candidate regions and unanchored candidate regions simultaneously.

[0031] The expression localization of the anchored candidate interval is achieved through intersection-union ratio (IU) threshold filtering and dynamic regression parameter adjustment: for an anchored candidate interval to be predicted, a predefined anchor box is set as (c d w d The predicted classification score is The intersection-union ratio (CUI) between the predicted anchored candidate interval and the closest real expression interval is p. o Let the dynamic regression parameters of the anchored candidate intervals to be predicted be denoted as {Δ c Δ w If the anchored candidate interval to be predicted is represented by dynamic regression parameters c = c d +αΔ c w d w = w d exp(βΔ ω During training, only the accurately predicted anchored candidate intervals are used to learn the intersection-union ratio p. o and dynamic regression parameters {Δ c Δ ω}; For the generated N p(ab) For each anchored candidate interval, first calculate the Intersection over Union (IoU), then filter out M intervals based on the IoU threshold. p(ab) We calculate the intersection-union loss based on the positive candidate intervals (samples that match the real expression intervals). Simultaneously, at each level of the feature pyramid network, a classification loss based on both positive and negative candidate intervals is calculated. Regression loss based solely on positive candidate intervals

[0032] Where: c d and w d c and w represent the center point and width of the predefined anchor frame, respectively; c and w represent the predicted center point and predicted width of the candidate interval with anchors, respectively; Δ c Indicates c relative to c d The offset, Δ w Indicates w relative to w d The offset; M represents the score for anchored candidate intervals located as expression n. (e) The total number of emoji types, with a prediction confidence level of . α and β are scale parameters;

[0033] The expression localization of the anchorless candidate interval is achieved through foreground classification and dynamic regression parameters: In the feature pyramid network, the input of the lowest level is a sequence of window frames consisting of T window frames. The output of the next lower level is used as the input of the next higher level. Each level downsamples the input of its own level (in the feature pyramid network, the time dimension is usually halved with each upward layer). The positions of the window frames in the window frame sequence are mapped to different positions in the feature map: Let j∈{0,1,…,T} represent the index of a certain window frame in the window frame sequence, and the mapping position corresponding to the index in the i-th level is represented as j. i Based on feature map mapping, if j i If the corresponding window frame j falls within a certain expression range, then j will be... i Call it the foreground, otherwise j i This is called the background (meaning the window frame, not belonging to any expression region); for a candidate region to be predicted without anchors, the predicted classification score is calculated as follows: The dynamic regression parameter is {Δ b Δ e During training, calculate all N... p(af) The classification error of each anchored candidate interval is used to obtain the classification loss. For M predicted as the prospect p(af) The offset is optimized for each anchorless candidate interval to obtain the regression loss.

[0034] in: This represents the score for anchorless candidate intervals located as expression m, with a prediction confidence of max(s). (af) );Δ b =jb, is the starting frame offset, representing the prediction distance from window frame j to the expression's starting frame b; Δ e =ej is the end frame offset, which represents the predicted distance from the window frame to the end frame b of the expression;

[0035] In each level of the feature pyramid network, each window frame has both anchored and unanchored expression localization results. All expression localization results from each level of the feature pyramid network are aggregated and non-maximum suppression is performed to obtain the final expression localization.

[0036] Specifically, in step S05, based on the final expression localization, along the fused expression feature F (e) The time dimension of facial expression features is extracted, and these extracted features are input into an expression localization and recognition network consisting of 2D convolutional layers, temporal pooling layers, and a fully connected classifier to predict macro-expression and micro-expression categories. A multivariate cross-entropy loss L0 is then applied. ce Calculate the loss for facial expression recognition.

[0037] Specifically, in step S06, in the loss function of the unified model for fine-grained facial expression localization and recognition from the first perspective, the AU recognition loss is a weighted binary cross-entropy loss L. (u) Dual anchoring positioning loss includes classification loss. Cross-combination loss Regression loss Classification loss and distance regression loss The loss for facial expression recognition is the multivariate cross-entropy loss L. ce .

[0038] An apparatus for implementing any of the above-mentioned joint expression localization and recognition methods based on aggregated features and dual anchoring includes a single face video extraction unit, a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual anchoring localization unit, an expression recognition unit, and a joint training and optimization unit.

[0039] The single-face video extraction unit performs face tracking on the input first-person perspective video, extracts the trajectory of each face in the first-person perspective video, and forms a single-face video with a continuous time sequence.

[0040] The local-global feature aggregation unit first inputs a single-face video into a 2D convolutional layer for precoding to obtain a precoded feature map, and then passes it through N... (lg) A series of cascaded local-global feature aggregation modules capture local high-frequency action features and global low-frequency face features of a single face image. Then, an AU encoding F is obtained after passing through a batch normalization layer and a global average pooling layer. t(u) Finally, the AU recognition probability is obtained after passing through a fully connected layer.

[0041] The spatiotemporal feature fusion unit first processes the single-face video and AU-encoded F... t(u) Feature extraction is performed on the time series data to obtain two single-channel temporal feature matrices. Then, a two-stream 3D convolutional network is used to fuse the two single-channel temporal feature matrices at multiple scales to generate a fused facial expression feature F. (e) ;

[0042] The dual-anchoring localization unit is based on multi-layer temporal features extracted from a feature pyramid network with L levels. It generates predefined anchor boxes and dynamic regression parameters by combining a sliding window strategy. It selects the expression localization interval corresponding to the maximum confidence value through non-maximum suppression, and simultaneously achieves classification and regression with and without anchors.

[0043] The expression recognition unit, based on the final expression localization result of the dual-anchoring localization unit, follows the fused expression feature F... (e)The time dimension of facial expression features is extracted, the feature dimension is compressed through a temporal pooling layer, and a fully connected classifier is used to distinguish between macro-expression categories and micro-expression categories to achieve unified facial expression recognition.

[0044] The joint training and optimization unit simultaneously optimizes the accuracy of facial expression localization and the performance of facial expression recognition through a multi-task loss function, and introduces weight parameters to balance the gradients of the two types of losses. Specifically, it calculates the parameters and loss values ​​of the first-view fine-grained facial expression localization and recognition unified model, which consists of a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual anchoring localization unit, and an facial expression recognition unit, and updates the parameters based on the gradient optimization method.

[0045] This invention trains a unified first-view fine-grained facial expression localization and recognition model using an end-to-end method. The model includes a frame-by-frame AU recognition network based on local-global feature aggregation units, a spatiotemporal feature extraction network based on spatiotemporal feature fusion units, and an facial expression localization and recognition network based on dual anchoring localization units and facial expression recognition units. First, the frame-by-frame AU recognition network is trained to extract the AU encoding F. t(u) As input to the facial expression localization and recognition network, the network then learns specific patterns and spatiotemporal correlation features of the AUs, and uses these features to facilitate the localization and recognition of facial action units.

[0046] Specifically, the local-global feature aggregation module uses a sub-block embedding layer and N (l) A cascaded local feature aggregator and N (g) A global feature aggregator is implemented, and a sub-block embedding layer is used to embed the input and perform preliminary feature transformation, mapping it to a feature space suitable for subsequent processing. The local feature aggregator (based on multi-head convolution to extract local facial texture) uses a multi-head convolution module and a multi-layer perceptron module to extract local high-frequency action features. The global feature aggregator (based on multi-head self-attention to model global muscle correlation) uses a multi-head self-attention module, a multi-head convolution module, and a multi-layer perceptron module to extract global low-frequency facial features. Gradient vanishing of the local feature aggregator and the global feature aggregator is suppressed through residual connections.

[0047] Specifically, the dual-stream 3D convolutional network uses independent spatiotemporal feature extraction modules to extract spatiotemporal features from the two single-channel temporal feature matrices. The extracted spatiotemporal features are then added together to obtain the fused expression feature F. (e) ;

[0048] The spatiotemporal feature extraction module consists of cascaded 3D convolutional layers, max pooling layers, a set of multi-scale 3D convolutional modules, and a global average pooling layer. The multi-scale 3D convolutional module contains a set of parallel branches, each branch using 3D convolutional kernels of different sizes to capture information from different receptive fields. The outputs of each branch are concatenated at the channel level to serve as the output of the multi-scale 3D convolutional module. The output of the last multi-scale 3D convolutional module is then dimensionality-reduced by the global average pooling layer to generate the corresponding spatiotemporal features.

[0049] Beneficial Effects: The joint facial expression localization and recognition method and device based on aggregated features and dual anchoring provided by this invention has the following advantages over the prior art: 1. By aggregating high-frequency action and low-frequency structural information through a local-global feature perception module, it effectively captures the fine-grained features of AU (Active Expression). Combined with a weighted loss function, it alleviates the problem of data imbalance and improves the accuracy of frame-by-frame AU recognition; 2. It adopts dual-stream 3D convolution to fuse the original video and AU encoded features, and uses a dual anchoring strategy to take into account the localization needs of facial expressions of different durations. It achieves efficient screening of multi-scale candidate intervals through a feature pyramid network and non-maximum suppression; 3. It optimizes the localization and recognition tasks simultaneously through a unified framework, supports parallel processing of macro / micro expressions, adapts to the complex situation of multiple expressions appearing in real-world scenarios, and has both high robustness and real-time performance. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the implementation process of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of the frame-by-frame AU recognition network based on local-global feature aggregation;

[0052] Figure 3 Here are schematic diagrams of the local feature aggregator and the global feature aggregator.

[0053] Figure 4 This is a schematic diagram of the structure of a multi-scale 3D convolutional module;

[0054] Figure 5 This is a schematic diagram of the prediction head structure;

[0055] Figure 6 This is a schematic diagram of the structure of the unified model for fine-grained facial expression localization and recognition in the first-person perspective. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] This invention provides a joint expression localization and recognition method and apparatus based on aggregated features and dual anchoring. In a frame-by-frame AU recognition network based on local-global feature aggregation, cascaded local-global feature aggregation modules simultaneously capture local high-frequency action features focused by multi-head convolution and global low-frequency face features modeled by multi-head self-attention. Because it integrates fine-grained local and global face information, AU encoding can accurately represent the semantics and spatial relevance of each AU. In the dual-anchored expression localization and recognition network, a dual-stream 3D convolutional network is used to extract multi-scale spatiotemporal features from single-face videos and AU encodings. Combined with the dual anchoring strategy of a feature pyramid network, in the temporal domain, an anchor-based localization module adapts to medium-duration expressions, while an anchorless localization module captures extremely short or extremely long expression fragments, outputting the start and end frames and category of the final expression localization. Due to the collaborative optimization of local-global feature aggregation and dual anchoring, this invention can adapt to complex situations in uncontrolled scenes where macro and micro expressions appear mixed and have large duration differences, and is expected to improve the robustness of multi-expression localization while achieving high-precision expression recognition.

[0058] like Figure 1 The diagram shown is a flowchart of a joint facial expression localization and recognition method based on aggregated features and dual anchoring. The specific steps are explained below.

[0059] S01: Perform face tracking on the input first-person perspective video, extract the trajectory of each face needed for training from the first-person perspective video, and form individual face videos.

[0060] For single-face videos, in order to avoid the situation where too few frames are captured, making it difficult to learn the correct spatiotemporal relationships of action units, or too many frames are captured, resulting in excessively long learning times, it is necessary to selectively extract video frames.

[0061] S02: Perform local-global feature aggregation on each frame of the single-face image in the single-face video to generate AU encoding.

[0062] like Figure 2 As shown, for a given single-face video, it is first input into a 2D convolutional layer for precoding to obtain a precoded feature map, and then processed through N... (lg) A series of cascaded local-global feature aggregation modules capture local high-frequency action features and global low-frequency face features of a single face image. Then, an AU encoding F is obtained after passing through a batch normalization layer and a global average pooling layer. t(u) Finally, the AU recognition probability is obtained after passing through a fully connected layer. The precoded feature map is used as the input to the first local-global feature aggregation module, the output of the previous local-global feature aggregation module is used as the input to the next local-global feature aggregation module, and the output of the last local-global feature aggregation module is used.

[0063] like Figure 3 As shown, each local-global feature aggregation module uses a sub-block embedding layer and N... (l) A cascaded local feature aggregator and N (g) The system is implemented using a global feature aggregator. The sub-block embedding layer is used to embed and perform preliminary feature transformation on the input of the local-global feature aggregation module, and map it to a feature space suitable for subsequent processing. The local feature aggregator consists of a multi-head convolution module and a multi-layer perceptron module, and is used to extract local high-frequency action features. The global feature aggregator consists of a 2D pointwise convolutional layer, a batch normalization layer, a multi-head self-attention module, a multi-head convolution module, and a multi-layer perceptron module, and is used to extract global low-frequency face features. The outputs of both the local and global feature aggregators are connected using residual connections.

[0064] The sub-block embedding layer embeds and performs preliminary feature transformation on the input of the local-global feature aggregation module, maps it to a feature space suitable for subsequent processing, and uses the resulting feature map as the input of the first local feature aggregator.

[0065] The implementation process of the local feature aggregator is as follows: In the multi-head convolution module, each input feature map is first divided into multiple sub-feature maps along the channel dimension, and 2D convolution operation is performed on each sub-feature map. The convolution results of each sub-feature map are concatenated along the channel dimension, and batch normalization, non-linear activation, and 2D pointwise convolution are performed sequentially to obtain the output of the multi-head convolution module. In the multi-layer perceptron module, the output of the multi-head convolution module is residually connected with the input and used as the input of the multi-layer perceptron module. 2D convolution, non-linear activation, and temporary deceleration are performed sequentially to obtain the output of the multi-layer perceptron module. The multi-layer perceptron module enhances the expressive power of local features. The feature map obtained by residually connecting the input and output of the multi-layer perceptron module is used as the output of the local feature aggregator, and the output of the last local feature aggregator is used as the input of the first global feature aggregator.

[0066] The implementation process of the global feature aggregator is as follows: First, the number of input channels is adjusted through a 2D pointwise convolutional layer. Then, the input is sequentially passed through a batch normalization layer and a multi-head self-attention module to capture global low-frequency face features. The output of the multi-head self-attention module is residually concatenated with the input of the batch normalization layer and used as the input of the multi-head convolutional module. The output of the multi-head convolutional module is residually concatenated with the input and used as the input of the multi-layer perceptron module. The feature map obtained by residually concatenating the input and output of the multi-layer perceptron module is used as the output of the global feature aggregator. Finally, the output of the global feature aggregator is passed through a batch normalization layer and a global average pooling layer to obtain the AU encoding F. t(u) .

[0067] During training, a weighted binary cross-entropy loss function was used for supervision to optimize the accuracy of AU recognition. The loss function is expressed as:

[0068]

[0069] in: M represents the probability of AU recognition. (u) Let N represent the total number of AUs, and let N represent the total number of sample images in the training set. This represents the total number of sample images in the training set that contain the j-th AU. w represents the probability of the j-th AU appearing in the training set. j This represents the weight that suppresses frequency imbalances between the j-th AU and other AUs, v j This represents the weights used to suppress the j-th AU from appearing too infrequently (used in the first term of cross-entropy, i.e., the occurrence term). and Let represent the actual occurrence probability and the predicted occurrence probability of the j-th AU in the t-th frame sample image, respectively.

[0070] Weight calculation is used to address the issue of uneven frequency of AU occurrences in AU identification. The weights are in the following form:

[0071]

[0072] in: The weight w represents the total number of sample images in the training set that contain the j-th AU; j This helps to suppress the problem of frequency imbalance among different AUs, giving lower-frequency AUs higher weights, making the model more sensitive to less common AUs, and preventing high-frequency AUs from dominating model training; weight v j This involves further adjusting the frequency of each AU, which helps to suppress the problem that the frequency of each AU is often lower than the frequency of no occurrence.

[0073] S03: The AU-encoded time series is reshaped and concatenated, and then fused with single-face videos through a dual-stream 3D convolutional network to generate fused expression features.

[0074] The AU-encoded time series is reshaped and concatenated to convert it into a single-channel temporal feature matrix I. Feature extraction is then performed on a single-face video to obtain a single-channel temporal feature matrix II. A dual-stream 3D convolutional network includes two inputs: the first input is the single-channel temporal feature matrix I, and the second input is the single-channel temporal feature matrix II. The first input extracts spatiotemporal feature I through an independent spatiotemporal feature extraction module, and the second input extracts spatiotemporal feature II through the same module. The spatiotemporal features I and II are then added to obtain the fused expression feature F. (e) .

[0075] The spatiotemporal feature extraction module consists of cascaded 3D convolutional layers, max pooling layers, a set of multi-scale 3D convolutional modules, and a global average pooling layer. For example... Figure 4 As shown, the multi-scale 3D convolutional module contains a set of branches set up in parallel. Each branch uses 3D convolutional kernels of different sizes (including 3×3×3 and 5×5×5 kernels) to capture information from different receptive fields. Different sized kernels can extract features from different receptive fields, thus effectively capturing the spatiotemporal changes of local details and global structure. The outputs of each branch are concatenated at the channel level to serve as the output of the multi-scale 3D convolutional module. A global average pooling layer is then used to reduce the dimensionality of the output of the last multi-scale 3D convolutional module, generating the corresponding spatiotemporal features.

[0076] By using this multi-scale feature extraction method, the network can simultaneously focus on local motion features and global structural features in the video, thereby enhancing its sensitivity to changes in facial expressions.

[0077] S04: Fusing facial expression features based on a sliding window strategy (e) Multi-scale sampling is performed, and facial expression localization is achieved by combining a feature pyramid network with both anchored and unanchored localization mechanisms. Finally, non-maximum suppression is used to filter the final facial expression localization.

[0078] Based on the sliding window strategy, the fused facial expression features F (e) To perform multi-scale sampling, it is necessary to set sliding windows of different lengths to cover the typical duration of macro and micro expressions. Two adjacent sliding windows in the time dimension have partial overlap. T window frames are uniformly sampled from single-face video segments within different sliding windows to obtain different window frame sequences.

[0079] Along the fusion facial feature F (e)After applying a sliding window in the time dimension, the window frame sequence is first encoded using 1D convolutional layers and max pooling layers. Then, it is input into a feature pyramid network containing L levels. In each level of the feature pyramid network, a dual localization mechanism (such as...) is executed in parallel. Figure 5 As shown, 1D convolutional layers are used to generate both anchored and unanchored candidate regions simultaneously.

[0080] (41) Expression localization with anchored candidate intervals is achieved by cross-union ratio threshold screening and dynamic regression parameter adjustment.

[0081] For an anchored candidate interval to be predicted, the predefined anchor box is set as (c d w d The predicted classification score is The intersection-union ratio (CUI) between the predicted anchored candidate interval and the closest real expression interval is p. o Let the dynamic regression parameters of the anchored candidate intervals to be predicted be denoted as {Δ c Δ w If the anchored candidate interval to be predicted is represented by dynamic regression parameters c = c d +αΔ c w d w = w d exp(βΔ ω ).

[0082] During training, the intersection-union ratio p is learned only using the accurately predicted anchored candidate intervals. o and dynamic regression parameters {Δ c Δ ω}; For the generated N p(ab) For each anchored candidate interval, first calculate the Intersection over Union (IoU), then filter out M intervals based on the IoU threshold. p(ab) We calculate the intersection-union loss based on the positive candidate intervals (samples that match the real expression intervals). Simultaneously, at each level of the feature pyramid network, a classification loss based on both positive and negative candidate intervals is calculated. and regression loss based solely on positive candidate intervals

[0083]

[0084] Where: c d and w d c and w represent the center point and width of the predefined anchor frame, respectively; c and w represent the predicted center point and predicted width of the candidate interval with anchors, respectively; Δ c Indicates c relative to c d The offset, Δ w Indicates w relative to w dThe offset; M represents the score for anchored candidate intervals located as expression m. (e) The total number of emoji types, with a prediction confidence level of . α and β are scale parameters. M p(ab) It is the number of positive samples corresponding to the label assignment; It is a real label; focal The focusing loss IoU(c, c′) represents the intersection-union ratio between the predicted candidate interval and the true candidate interval; CSmoothL1(x) is the smoothing L1 loss, used to alleviate the training instability problem when there is a large bias.

[0085] (42) Expression localization without anchor candidate intervals is achieved through foreground classification and dynamic regression parameters.

[0086] In a feature pyramid network, the input to the lowest level is a sequence of T window frames. The output of the next lower level is used as the input to the level above. Each level downsamples its own input (in a feature pyramid network, the time dimension is typically halved with each level increase). The positions of the window frames in the sequence are mapped to different positions in the feature map: Let j∈{0,1,…,T} represent the index of a window frame in the sequence, and let the mapping position corresponding to index j in the i-th level be denoted as j. i Based on feature map mapping, if j i If the corresponding window frame j falls within a certain expression range, then j will be... i Call it the foreground, otherwise j i This is called the background (meaning window frame j does not belong to any expression region); for a candidate region without anchors to be predicted, the predicted classification score is calculated as follows: The dynamic regression parameter is {Δ b Δ e}

[0087] During training, calculate all N p(af) The classification error of each anchored candidate interval is used to obtain the classification loss. For M predicted as the prospect p(af) The offset is optimized for each anchorless candidate interval to obtain the regression loss.

[0088]

[0089] in: This represents the score for anchorless candidate intervals located as expression m, with a prediction confidence of max(s). (af) );Δ b =jb, is the starting frame offset, representing the prediction distance from window frame j to the expression's starting frame b; Δ e=ej, is the end frame offset, representing the predicted distance from window frame j to the end expression frame b; M p(af) It refers to the number of attractions in front.

[0090] (43) During testing, the anchorless localization module outputs the distance from the foreground point to the start frame of the expression and the distance to the end frame of the expression, and its prediction confidence is the classification score s. (af) The maximum value in max(s) (af) The anchor-based positioning module outputs the interval position (c, w) and classification score s. (ab) and intersection ratio p o Its prediction confidence level is Each point in each layer of the feature pyramid network has both anchor-free and anchor-based localization results. After aggregating all the results from each layer, non-maximum suppression is performed to obtain the final localization results for each expression.

[0091] (44) Calculate the total loss function of the dual-anchored facial expression localization and recognition network: The loss function of the unified model includes classification loss based on both positive and negative candidate intervals. Regression loss based solely on positive candidate intervals Classification loss and regression loss Each loss term is assigned an appropriate weight. These loss terms are optimized through weighted summation during training of the unified framework:

[0092]

[0093] In each level of the feature pyramid network, each window frame has both anchored and unanchored expression localization results. All expression localization results from each level of the feature pyramid network are aggregated and non-maximum suppression is performed to obtain the final expression localization.

[0094] S05: Extract facial feature segments based on the start and end frames of the final facial expression localization, and input the facial feature segments into the facial expression localization and recognition network to predict the macro-facial expression category and micro-facial expression category.

[0095] Based on the final facial expression localization, along the fused facial expression feature F (e) The time dimension of facial expression features is extracted, and these extracted features are input into an expression localization and recognition network consisting of 2D convolutional layers, temporal pooling layers, and a fully connected classifier to predict macro-expression and micro-expression categories. A multivariate cross-entropy loss L0 is then applied. ce Calculate the facial expression recognition loss to achieve unified macro-expression and micro-expression recognition.

[0096] S06: By jointly optimizing the AU recognition loss, dual anchoring localization loss, and expression recognition loss, train the first-view fine-grained expression localization and recognition unified model consisting of steps S01 to S05.

[0097] In the loss function of the unified model for first-person perspective fine-grained facial expression localization and recognition, the AU recognition loss is a weighted binary cross-entropy loss L. (u) Dual anchoring positioning loss includes classification loss. Cross-combination loss Regression loss Classification loss and distance regression loss The loss for facial expression recognition is the multivariate cross-entropy loss L. ce .

[0098] A first-person perspective fine-grained unified model for facial expression localization and recognition (e.g., AU recognition network that aggregates local and global features, spatiotemporal feature extraction network, and dual-anchored facial expression localization and recognition network) is trained using an end-to-end method. Figure 6 (As shown); First, a frame-by-frame AU recognition network is trained, and the accurate AU codes are extracted as input to the expression localization and recognition network. Then, the expression localization and recognition network learns the specific patterns and spatiotemporal correlation features of AUs, and uses the spatiotemporal correlation features between AUs to promote the localization and recognition of facial action units.

[0099] S07: Input a first-person perspective video with any number of frames into the trained first-person perspective fine-grained expression localization and recognition unified model to identify macro expression categories and micro expression categories.

[0100] The prediction results of the facial motion unit are directly output during the prediction process.

[0101] The method of this invention can be implemented entirely by computer without the need for manual assistance; this indicates that the invention can achieve batch automatic processing, which can greatly improve processing efficiency and reduce labor costs.

[0102] A joint facial expression localization and recognition device based on aggregated features and dual anchoring includes a single face video extraction unit, a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual anchoring localization unit, an facial expression recognition unit, and a joint training and optimization unit;

[0103] The single-face video extraction unit performs face tracking on the input first-person perspective video, extracts the trajectory of each face in the first-person perspective video, and forms a single-face video with a continuous time sequence.

[0104] The local-global feature aggregation unit first inputs a single-face video into a 2D convolutional layer for precoding to obtain a precoded feature map, and then passes it through N...lg) A series of cascaded local-global feature aggregation modules capture local high-frequency action features and global low-frequency face features of a single face image. Then, an AU encoding F is obtained after passing through a batch normalization layer and a global average pooling layer. t(u) Finally, the AU recognition probability is obtained after passing through a fully connected layer.

[0105] The spatiotemporal feature fusion unit first processes the single-face video and AU-encoded F... t(u) Feature extraction is performed on the time series data to obtain two single-channel temporal feature matrices. Then, a two-stream 3D convolutional network is used to fuse the two single-channel temporal feature matrices at multiple scales to generate a fused facial expression feature F. (e) ;

[0106] The dual-anchoring localization unit is based on multi-layer temporal features extracted from a feature pyramid network with L levels. It generates predefined anchor boxes and dynamic regression parameters by combining a sliding window strategy. It selects the expression localization interval corresponding to the maximum confidence value through non-maximum suppression, and simultaneously achieves classification and regression with and without anchors.

[0107] The expression recognition unit, based on the final expression localization result of the dual-anchoring localization unit, follows the fused expression feature F... (e) The time dimension of facial expression features is extracted, the feature dimension is compressed through a temporal pooling layer, and a fully connected classifier is used to distinguish between macro-expression categories and micro-expression categories to achieve unified facial expression recognition.

[0108] The joint training and optimization unit simultaneously optimizes the facial expression localization accuracy (cross-union loss) and facial expression recognition performance (cross-entropy loss) through a multi-task loss function, and introduces weight parameters to balance the gradients of the two types of losses; it calculates the parameters and loss values ​​of the first-view fine-grained facial expression localization and recognition unified model composed of a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual anchoring localization unit, and a facial expression recognition unit, and updates the parameters based on the gradient optimization method.

[0109] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A joint facial expression localization and recognition method based on aggregated features and dual anchoring, characterized in that: Includes the following steps: S01: Perform face tracking on the input first-person perspective video, extract the trajectory of each face in the first-person perspective video, and form individual face videos. S02: Perform local-global feature aggregation on each frame of the single-face video to generate AU encoding; the local-global feature aggregation is implemented through cascaded local feature aggregators and global feature aggregators, wherein the local feature aggregator uses a multi-head convolutional module and a multi-layer perceptron module to extract local high-frequency action features, and the global feature aggregator uses a multi-head self-attention module, a multi-head convolutional module and a multi-layer perceptron module to extract global low-frequency face features, and suppresses gradient vanishing of the local feature aggregator and the global feature aggregator through residual connections; S03: Reshape and concatenate the AU-encoded time series to convert it into a single-channel temporal feature matrix I. Extract features from a single-face video to obtain a single-channel temporal feature matrix II. Input the single-channel temporal feature matrix I and the single-channel temporal feature matrix II into a two-stream 3D convolutional network for multi-scale spatiotemporal feature fusion to generate a fused expression feature F. (e) ; S04: Fusing facial expression features based on a sliding window strategy (e) Multi-scale sampling is performed, and facial expression localization is achieved by combining a feature pyramid network with both anchored and unanchored localization mechanisms. The final facial expression localization is then filtered by non-maximum suppression. S05: Extract facial feature segments based on the start and end frames of the final facial expression localization, and input the facial feature segments into the facial expression localization and recognition network to predict the macro-facial expression category and micro-facial expression category; S06: By jointly optimizing the AU recognition loss, dual anchoring localization loss and expression recognition loss, train the first-view fine-grained expression localization and recognition unified model consisting of steps S01 to S05; S07: Input a first-person perspective video with any number of frames into the trained first-person perspective fine-grained expression localization and recognition unified model to identify macro expression categories and micro expression categories.

2. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 1, characterized in that: In step S02, the single-face video is first input into a 2D convolutional layer for precoding to obtain a precoded feature map, and then passed through N... (lg) A series of cascaded local-global feature aggregation modules capture local high-frequency action features and global low-frequency face features of a single face image. Then, an AU encoding F is obtained after passing through a batch normalization layer and a global average pooling layer. t (u) Finally, the AU recognition probability is obtained after passing through a fully connected layer. The precoded feature map is used as the input to the first local-global feature aggregation module, the output of the previous local-global feature aggregation module is used as the input to the next local-global feature aggregation module, and the output of the last local-global feature aggregation module is used. Each local-global feature aggregation module uses a sub-block embedding layer, N (l) A cascaded local feature aggregator and N (g) The local feature aggregator is implemented using a global feature aggregator. The sub-block embedding layer is used to embed and perform preliminary feature transformation on the input of the local-global feature aggregator module, and map it to a feature space suitable for subsequent processing. The local feature aggregator consists of a multi-head convolution module and a multi-layer perceptron module, and is used to extract local high-frequency action features. The global feature aggregator consists of a 2D pointwise convolutional layer, a batch normalization layer, a multi-head self-attention module, a multi-head convolution module, and a multi-layer perceptron module, and is used to extract global low-frequency face features. The outputs of the local feature aggregator and the global feature aggregator are both connected using residual connections. The sub-block embedding layer embeds and performs preliminary feature transformation on the input of the local-global feature aggregation module, and maps it to a feature space suitable for subsequent processing. The resulting feature map is used as the input of the first local feature aggregator. The implementation process of the local feature aggregator is as follows: In the multi-head convolution module, the input feature map is first divided into multiple sub-feature maps along the channel dimension, and 2D convolution operation is performed on each sub-feature map. The convolution results of each sub-feature map are concatenated along the channel dimension, and batch normalization, non-linear activation, and 2D pointwise convolution are performed sequentially to obtain the output of the multi-head convolution module. In the multi-layer perceptron module, the output of the multi-head convolution module is residually connected with the input and used as the input of the multi-layer perceptron module. 2D convolution, non-linear activation, and temporary deceleration are performed sequentially to obtain the output of the multi-layer perceptron module, thereby enhancing the expressive power of local features. The feature map obtained by residually connecting the input and output of the multi-layer perceptron module is used as the output of the local feature aggregator, and the output of the last local feature aggregator is used as the input of the first global feature aggregator. The implementation process of the global feature aggregator is as follows: First, the number of input channels is adjusted through a 2D pointwise convolutional layer. Then, the input is sequentially passed through a batch normalization layer and a multi-head self-attention module to capture global low-frequency face features. The output of the multi-head self-attention module is residually concatenated with the input of the batch normalization layer and used as the input of the multi-head convolutional module. The output of the multi-head convolutional module is residually concatenated with the input and used as the input of the multi-layer perceptron module. The feature map obtained by residually concatenating the input and output of the multi-layer perceptron module is used as the output of the global feature aggregator. Finally, the output of the global feature aggregator is passed through a batch normalization layer and a global average pooling layer to obtain the AU encoding F. t(u) .

3. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 2, characterized in that: In step S02, the AU recognition probability Using weighted binary cross-entropy loss L (u) Supervision: in: M represents the probability of AU recognition. (u) Let N represent the total number of AUs, and let N represent the total number of sample images in the training set. This represents the total number of sample images in the training set that contain the j-th AU. w represents the probability of the j-th AU appearing in the training set. j This represents the weight that suppresses frequency imbalances between the j-th AU and other AUs, v j This represents the weight used to suppress the j-th AU from occurring too infrequently. and Let represent the actual occurrence probability and the predicted occurrence probability of the i-th AU in the t-th sample image, respectively.

4. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 1, characterized in that: In step S03, the dual-stream 3D convolutional network includes two inputs: a first input is a single-channel temporal feature matrix I, and a second input is a single-channel temporal feature matrix II. The first input extracts spatiotemporal feature I through an independent spatiotemporal feature extraction module, and the second input extracts spatiotemporal feature II through an independent spatiotemporal feature extraction module. The spatiotemporal feature I and spatiotemporal feature II are added together to obtain the fused facial expression feature F. (e) ; The spatiotemporal feature extraction module consists of cascaded 3D convolutional layers, max pooling layers, a set of multi-scale 3D convolutional modules, and a global average pooling layer. The multi-scale 3D convolutional module contains a set of parallel branches, each branch using 3D convolutional kernels of different sizes to capture information from different receptive fields. The outputs of each branch are concatenated at the channel level to serve as the output of the multi-scale 3D convolutional module. The output of the last multi-scale 3D convolutional module is then dimensionality-reduced by the global average pooling layer to generate the corresponding spatiotemporal features.

5. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 1, characterized in that: In step S04, the fused facial expression features F are first processed based on a sliding window strategy. (e) Multi-scale sampling is performed, and sliding windows of different lengths are set to cover the typical duration of macro and micro expressions. Two adjacent sliding windows in the time dimension have partial overlap. T window frames are uniformly sampled from single face video segments in different sliding windows to obtain different window frame sequences. Along the fusion facial feature F (e) After applying a sliding window in the time dimension, the window frame sequence is first encoded using 1D convolutional layers and max pooling layers. Then, a feature pyramid network with L levels is input. In each level of the feature pyramid network, a dual localization mechanism is executed in parallel, using 1D convolutional layers to generate anchored candidate regions and unanchored candidate regions simultaneously. The expression localization of the anchored candidate interval is achieved through intersection-union ratio (IU) threshold filtering and dynamic regression parameter adjustment: for an anchored candidate interval to be predicted, a predefined anchor box is set as (c d w d The predicted classification score is The intersection-union ratio (CUI) between the predicted anchored candidate interval and the closest real expression interval is p. o Let the dynamic regression parameters of the anchored candidate intervals to be predicted be denoted as {Δ c Δ w If the anchored candidate interval to be predicted is represented by dynamic regression parameters c = c d +αΔ c w d w = w d exp(βΔ ω During training, only the accurately predicted anchored candidate intervals are used to learn the intersection-union ratio p. o and dynamic regression parameters {Δ c Δ ω }; For the generated N p(ab) For each anchored candidate interval, first calculate the intersection-union ratio (CIR), then filter out M intervals based on the CIR threshold. p(ab) Identify several positive candidate intervals, and finally calculate the intersection-union ratio (IUGR) loss based on these positive candidate intervals. Simultaneously, at each level of the feature pyramid network, a classification loss based on both positive and negative candidate intervals is calculated. Regression loss based solely on positive candidate intervals Where: c d and w d c and w represent the center point and width of the predefined anchor frame, respectively; c and w represent the predicted center point and predicted width of the candidate interval with anchors, respectively; Δ c Indicates c relative to c d The offset, Δ w Indicates w relative to w d offset M represents the score for anchored candidate intervals located as expression m. (e) Let p be the total number of emoji types. o max(s (ab) ); α and β are scale parameters; The expression localization of the anchorless candidate interval is achieved through foreground classification and dynamic regression parameters: In the feature pyramid network, the input of the lowest level is a sequence of T window frames. The output of the next lower level is used as the input of the previous level. Each level downsamples the input of its own level, mapping the window frame positions in the window frame sequence to different mapping positions in the feature map: Let j∈{0,1,…,T} represent the index of a certain window frame in the window frame sequence, and the mapping position corresponding to the index in the i-th level is represented as j. i Based on feature map mapping, if j i If the corresponding window frame j falls within a certain expression range, then j will be... i Call it the foreground, otherwise j i This is called the background; for a candidate interval without anchors to be predicted, the predicted classification score is calculated as follows: The dynamic regression parameter is {Δ b Δ e During training, calculate all N... p(af) The classification error of each anchored candidate interval is used to obtain the classification loss. For Mp predicted as the prospect (af) The offset is optimized for each anchorless candidate interval to obtain the regression loss. in: This represents the score for anchorless candidate intervals located as expression m, with a prediction confidence of max(s). (af) );Δ b =jb, is the starting frame offset, representing the prediction distance from window frame j to the expression's starting frame b; Δ e =ej is the end frame offset, which represents the predicted distance from the window frame to the end frame b of the expression; In each level of the feature pyramid network, each window frame has both anchored and unanchored expression localization results. All expression localization results from each level of the feature pyramid network are aggregated and non-maximum suppression is performed to obtain the final expression localization.

6. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 1, characterized in that: In step S05, based on the final expression localization, along the fused expression feature F (e) The time dimension of facial expression features is extracted, and these extracted features are input into an expression localization and recognition network consisting of 2D convolutional layers, temporal pooling layers, and a fully connected classifier to predict macro-expression and micro-expression categories. A multivariate cross-entropy loss L0 is then applied. ce Calculate the loss for facial expression recognition.

7. The joint facial expression localization and recognition method based on aggregated features and dual anchoring according to claim 1, characterized in that: In step S06, the loss function of the first-view fine-grained expression localization and recognition unified model includes an AU recognition loss, which is a weighted binary cross-entropy loss L. (u) Dual anchoring positioning loss includes classification loss. Cross-combination loss Regression loss Classification loss and distance regression loss The loss for facial expression recognition is the multivariate cross-entropy loss L. ce .

8. An apparatus for implementing any of the joint facial expression localization and recognition methods based on aggregated features and dual anchoring as described in claims 1 to 7, characterized in that: It includes a single-face video extraction unit, a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual-anchor localization unit, an expression recognition unit, and a joint training and optimization unit; The single-face video extraction unit performs face tracking on the input first-person perspective video, extracts the trajectory of each face in the first-person perspective video, and forms a single-face video with a continuous time sequence. The local-global feature aggregation unit first inputs a single-face video into a 2D convolutional layer for precoding to obtain a precoded feature map, and then passes it through N... (lg) A series of cascaded local-global feature aggregation modules capture local high-frequency action features and global low-frequency face features of a single face image. Then, an AU encoding F is obtained after passing through a batch normalization layer and a global average pooling layer. t(u) Finally, the AU recognition probability is obtained after passing through a fully connected layer. The spatiotemporal feature fusion unit first processes the single-face video and AU-encoded F... t(u) Feature extraction is performed on the time series data to obtain two single-channel temporal feature matrices. Then, a two-stream 3D convolutional network is used to fuse the two single-channel temporal feature matrices at multiple scales to generate a fused facial expression feature F. (e) ; The dual-anchoring localization unit is based on multi-layer temporal features extracted from a feature pyramid network with L levels. It generates predefined anchor boxes and dynamic regression parameters by combining a sliding window strategy. It selects the expression localization interval corresponding to the maximum confidence value through non-maximum suppression, and simultaneously achieves classification and regression with and without anchors. The expression recognition unit, based on the final expression localization result of the dual-anchoring localization unit, follows the fused expression feature F... (e) The time dimension of facial expression features is extracted, the feature dimension is compressed through a temporal pooling layer, and a fully connected classifier is used to distinguish between macro-expression categories and micro-expression categories to achieve unified facial expression recognition. The joint training and optimization unit simultaneously optimizes the facial expression localization accuracy and facial expression recognition performance through a multi-task loss function, and introduces weight parameters to balance the gradients of the two types of losses. It calculates the parameters and loss values ​​of the first-view fine-grained facial expression localization and recognition unified model, which consists of a local-global feature aggregation unit, a spatiotemporal feature fusion unit, a dual anchoring localization unit, and an facial expression recognition unit, and updates the parameters based on the gradient optimization method.

9. The apparatus according to claim 8, characterized in that: The local-global feature aggregation module uses a sub-block embedding layer, N (l) A cascaded local feature aggregator and N (g) A global feature aggregator is implemented, and a sub-block embedding layer is used to perform preliminary feature transformation on the input and map it to a feature space suitable for subsequent processing. The local feature aggregator uses a multi-head convolution module and a multi-layer perceptron module to extract local high-frequency action features, while the global feature aggregator uses a multi-head self-attention module, a multi-head convolution module, and a multi-layer perceptron module to extract global low-frequency face features. Gradient vanishing of the local feature aggregator and the global feature aggregator is suppressed through residual connections.

10. The apparatus according to claim 8, characterized in that: The dual-stream 3D convolutional network extracts spatiotemporal features from the two single-channel temporal feature matrices using independent spatiotemporal feature extraction modules. The extracted spatiotemporal features are then added together to obtain the fused facial expression feature F. (e) ; The spatiotemporal feature extraction module consists of cascaded 3D convolutional layers, max pooling layers, a set of multi-scale 3D convolutional modules, and a global average pooling layer. The multi-scale 3D convolutional module contains a set of parallel branches, each branch using 3D convolutional kernels of different sizes to capture information from different receptive fields. The outputs of each branch are concatenated at the channel level to serve as the output of the multi-scale 3D convolutional module. The output of the last multi-scale 3D convolutional module is then dimensionality-reduced by the global average pooling layer to generate the corresponding spatiotemporal features.

Images