AU identification method fusing channel topology convolution and attention mechanism
Through the AU recognition method that integrates channel topology convolution and attention mechanism, the problem of failure to effectively capture multi-dimensional information in facial action unit recognition is solved, and the accuracy and robustness of AU recognition is improved, especially when processing complex images performance.
Patent Information
- Application Number
- CN202411089176.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art fails to effectively capture the multi-dimensional information between AU features and the contribution information of local structures and different regional textures in facial action unit recognition, and there is room for improvement in the graph convolutional structure.
AU recognition method using the fusion channel topology convolution and attention mechanism, including backbone network, multi-scale attention feature generation module, graph neural network and multi-channel topology feature fusion module, multi-scale aggregation of multi-scale attention mechanism and graph neural network Information, build topological relationships, and enhance the expression ability of AU relationship learning.
Effectively balance local and global information extraction, improve the model's performance in facial action unit recognition, improve the accuracy and robustness of AU recognition, especially when processing complex extreme images, it can supplement information from multiple angles.
Smart Images

Figure CN119091487A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of facial expression recognition, and in particular relates to an AU recognition method integrating channel topological convolution and attention mechanism. Background Art
[0002] Among the various channels for human emotion transmission, facial expressions occupy an important position. They are one of the most direct ways to express emotions and one of the important references for the cognitive evaluation of neurological functions in the medical field. Studies have shown that when people communicate face to face, words only account for 7% of emotional information, sounds account for 38%, and more than half of emotional information is transmitted through human facial expressions. Facial images have rich semantic information and can be divided into two categories, namely low-level semantic information and high-level semantic information. Low-level semantic information mainly involves aspects such as the shape, color, and texture of the image, while high-level semantic information involves more abstract features, and relevant information needs to be extracted through deep learning methods in order to interpret the image more deeply.
[0003] Facial action unit recognition (FAU Recognition) is a field closely related to facial expression recognition. It involves predicting the probability of each facial action unit (AU) appearing or not appearing, and is an important task in the field of computer vision and emotional computing. At present, the recognition of facial action units is mainly implemented by deep neural networks. Prior art 1 pre-trains the ResNet-34 network on the datasets of two AU-related tasks, namely overall expression recognition and face recognition, and then fine-tunes the two networks on the AU dataset, and takes the average of the AU occurrence probabilities predicted by the two networks as the final prediction value. Prior art 2 adopts multi-task learning to jointly realize face AU recognition and face registration based on CNN.
[0004] Since there is a priori positional relationship between facial key points and AUs, previous studies are usually based on this relationship. For example, facial geometric features and local texture features are combined for AU recognition; shape regularization for AU recognition is constructed based on landmark points; typical target detection tasks are introduced to define the bounding box of AUs through facial key points; the relationship between AUs is modeled using graph convolutional networks (GCNs) to improve recognition results; a method for learning facial action unit relationship graphs based on multi-dimensional edge features. The method based on graph convolutional networks can better integrate the relationship information between AUs, effectively improving the recognition effect of facial action units. These studies have promoted the development of the field of facial action recognition and provided important support for deepening the understanding of facial expressions.
[0005] The above methods failed to consider the multi-dimensional information between AU features when performing AU identification, and failed to effectively capture the contribution information of local structures and textures in different regions. The graph convolutional neural network GCN proposed by Luo et al. performed well in modeling edge feature connections and has obvious advantages in expressing the adjacency relationship between AUs, but there is still room for further improvement in feature extraction and graph convolution structure. Summary of the invention
[0006] In order to solve the above technical problems, the present invention proposes an AU recognition method that integrates channel topology convolution and attention mechanism to solve the problems existing in the above-mentioned prior art.
[0007] To achieve the above object, the present invention provides an AU recognition method integrating channel topology convolution and attention mechanism, comprising:
[0008] Constructing a facial action unit recognition model; the facial action unit recognition model includes a backbone network, a multi-scale attention feature generation module, a graph neural network, a multi-channel topological feature fusion module, and a prediction module;
[0009] Acquire a training data set, wherein the training data set includes a plurality of face images having multiple AU occurrence labels;
[0010] Constructing a weighted asymmetric loss function, and training the facial action unit recognition model using an AdamW optimizer and a cosine decay learning rate scheduler based on a training data set and the weighted asymmetric loss function;
[0011] Obtain a face image to be recognized, input it into the trained facial action unit recognition model, and obtain the facial action unit recognition result.
[0012] Optionally, the backbone network adopts Resnet50, the face image is input into the backbone network, the backbone network forward propagates and extracts facial features layer by layer to obtain facial feature representation.
[0013] Optionally, the multi-scale attention feature generation module (MAFG) adopts a multi-scale channel attention mechanism, inputs the facial feature representation output by the backbone network into the multi-scale channel attention mechanism, performs a multi-scale feature extraction operation on the facial feature representation to obtain a multi-scale feature, performs a channel mixing operation on the multi-scale feature to obtain a mixed feature, and performs element-wise multiplication of the mixed feature with the facial feature representation to obtain an enhanced feature representation. The features after multi-scale attention are passed through a linear block including a fully connected layer FC and a normalization layer to generate multiple feature map representations.
[0014] Optionally, in the graph neural network stage, the multi-scale attention mechanism first generates AU feature maps from the main view at different scales, obtains the initial feature representation of the relationship modeling stage, constructs an adjacency matrix for the relationship learning of the graph neural network, and aggregates multi-scale information by the multi-scale attention mechanism, enhancing the expression ability of its activation state during AU relationship learning. The graph neural network uses a multi-layer propagation strategy to make each feature representation as the information of the node's nearest neighbor aggregation, iteratively learns to gradually integrate potential long-distance adjacency relationships, and finally obtains the AU relationship feature representation after graph association learning.
[0015] Optionally, the multi-channel topological feature fusion module (CCFG) includes a channel topological convolution structure, which learns the relationship between image channels, reorganizes and adjusts features, and constructs a topological relationship. Based on this relationship, each independently generated AU feature map aggregates joint features in different channels, and finally obtains multiple AU feature map representations through a linear block including an average pooling layer and a fully connected layer.
[0016] Optionally, the multi-channel topological feature fusion module (CCFG) includes a feature fusion structure, which uses a splicing fusion method to balance the information of the two branches, splicing the joint features of the CCFG into the AU relationship features, so that the model has two different learning angles to complement each other when processing complex and extreme images.
[0017] Optionally, the weighted asymmetric loss function is:
[0018]
[0019] Among them, p i ,y i and w i Represent the predicted value, true value and weight of the occurrence probability of the i-th AU, r i is the occurrence rate of the i-th AU calculated in the training set.
[0020] Optionally, the process of training the facial action unit recognition model includes:
[0021] Based on the weighted asymmetric loss function, the backbone network and the multi-scale attention feature generation module, the graph neural network, the multi-channel topological feature fusion module and the graph neural network are trained, and a cosine decay learning rate scheduler is used to more effectively converge to the optimal solution of the model. The backbone network Resnet50 is pre-trained on the large dataset ImageNet to improve the model generalization ability and model training speed.
[0022] The present invention also provides an electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned facial action unit recognition method.
[0023] The present invention also provides a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the above-mentioned facial action unit recognition method.
[0024] Compared with the prior art, the present invention has the following advantages and technical effects:
[0025] The present invention proposes a channel topology convolution and attention module for fusion recognition of AU, which can effectively balance the extraction of local and global information, thereby improving the performance of the model in facial action unit recognition. A dual-branch feature fusion structure is proposed to splice the information of AU relationship learning and multi-channel joint features, so that the model can complement each other from two different learning angles when processing complex extreme images; the AU feature map in the main view is generated from different scales through a multi-scale attention mechanism, and an adjacency matrix is constructed for relationship learning of the graph neural network. Aggregate multi-scale information to enhance the expression ability of its activation state during AU relationship learning; use a convolutional topology structure to jointly generate AU feature maps of branch views, dynamically learn and aggregate joint features in different channels, so that the model pays attention to the AU area from another angle to provide fused information. The channel topology convolution and attention module proposed in the present invention help capture this complex and long-distance dependency between the two facial action units. Even if the learning field of view is limited or wrong information is learned from a certain angle, the required information can still be supplemented as much as possible from another angle to improve the performance of the overall model. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0027] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0028] Figure 2 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION
[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0030] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0031] Embodiment 1
[0032] like Figure 1 As shown, this embodiment provides an AU recognition method integrating channel topology convolution and attention mechanism, including:
[0033] Construct a facial action unit recognition model; the facial action unit recognition model includes a backbone network, a multi-scale attention feature generation module, a graph neural network, a multi-channel topological feature fusion module, and a prediction module;
[0034] In some specific embodiments, the backbone network uses Resnet50, the face image is input into the backbone network, the backbone network forward propagates and extracts facial features layer by layer to obtain facial feature representation.
[0035] Specifically, supervised learning is performed based on the backbone network Resnet50. The input image passes through its convolution, pooling, fully connected layers, and its residual block that can solve the gradient problem. It is forward propagated and facial features are extracted layer by layer to provide semantic information for the subsequent facial motion unit classification task.
[0036] In some specific embodiments, the multi-scale attention feature generation module (MAFG) adopts a multi-scale channel attention mechanism, inputs the facial feature representation output by the backbone network into the multi-scale channel attention module, performs a multi-scale feature extraction operation on the facial feature representation to obtain a multi-scale feature, performs a channel mixing operation on the multi-scale feature to obtain a mixed feature, and performs element-wise multiplication of the mixed feature with the facial feature representation to obtain an enhanced feature representation. The features after multi-scale attention are passed through a linear block including a fully connected layer FC and a normalization layer to generate multiple AU feature map representations.
[0037] Specifically, feature representation is enhanced through the attention mechanism. This embodiment proposes a multi-scale channel attention module for AU recognition tasks, which integrates multi-scale convolutional features from the axial attention in the vertical and horizontal directions, aggregates features in the vertical and horizontal directions in turn, ensures the independence of features in each direction, effectively captures global context information, and finally generates a feature map for each AU independently based on the aggregated multi-scale information.
[0038] The characteristics of this multi-scale attention feature generation module are as follows:
[0039] Multi-scale feature extraction: Given a complete facial image X∈R H×W×C As input, three one-dimensional convolutions with different kernel sizes and padding are used to encode X in parallel. The kernel sizes are set to 1×7, 1×11, and 1×21, respectively, and are regarded as Conv1D. i (i=0, 1, 2), the formula can be written as follows:
[0040]
[0041] in Conv2D is the one-dimensional convolution along the y-axis of this scale. 5x5 Aims to perform a deep convolution on the input image as the initial input for the multi-scale convolution. It is the output of the convolution of the corresponding size along the y-axis. The output of the other spatial dimension x-axis at this scale as follows:
[0042]
[0043] Given the context information captured in the vertical direction The vertical axis features are aggregated on the horizontal axis through the convolution kernel of the same size as the final information of this scale.
[0044] Channel mixing and convolutional attention: After extracting multi-scale features through deep convolutions of different sizes, 1×1 convolution is used to mix these features through channels to integrate information of different scales and implement a convolutional attention mechanism. The formula is as follows:
[0045]
[0046] Feature map generation: Finally, the original input feature map X is multiplied element by element with the information after channel mixing and multiple feature maps are generated, realizing a multi-scale attention mechanism to focus on the structure of multi-feature map generation, as shown below:
[0047]
[0048] Where N is the number of target AU categories, and the features after multi-scale attention are passed through N linear blocks including a fully connected layer FC and a normalization layer. Separate learning is performed to generate a vector Vi containing the channel feature C value as the feature map representation of the i-th AU.
[0049] In general, by extracting multi-scale features from the original feature map and mixing channels to generate feature maps for each type of AU, an initial feature representation containing richer information is obtained in the relationship modeling stage.
[0050] In some specific embodiments, the graph neural network stage includes the input of feature representation and AU relationship modeling. First, the multi-scale attention mechanism generates the AU feature map in the main view from different scales, and obtains the initial feature representation of the relationship modeling stage. The adjacency matrix is constructed for the relationship learning of the graph neural network. The multi-scale attention mechanism aggregates multi-scale information, which enhances the expression ability of its activation state during AU relationship learning. The graph neural network uses a multi-layer propagation strategy to make each feature representation as the information of the node's nearest neighbor aggregation, and iterative learning gradually integrates potential long-distance adjacency relationships, and finally obtains the AU relationship feature representation after graph association learning.
[0051] Specifically, in the forward propagation process of the graph neural network, the dot product metric is first used to construct a dynamic graph, and the similarity matrix S of the input N AU feature graphs is calculated. For each node i, the largest K values are selected from each row of the similarity matrix S (the maximum number of adjacent points is set to 4 in this paper), and the nodes corresponding to these K values are the nearest neighbors of node i. The adjacency matrix A is constructed based on the selection results of the nearest neighbors. The adjacency matrix constructed in this way can reflect the connection between each AU node and its most similar K nodes. The calculation formula can be expressed as follows:
[0052] U=U(X),
[0053] S=UU T ,
[0054]
[0055] Where U is the feature matrix after the feature map X is projected, the element Sij of the similarity matrix S represents the dot product similarity between the feature vectors of node i and node j, and the adjacency matrix A is constructed based on the nearest neighbors of Sij. In order to avoid numerical instability during training, the adjacency matrix is normalized as follows:
[0056]
[0057] Where D ii is the diagonal element of the diagonal matrix D, is the degree of node i (i.e., the number of nodes connected to node i), and A′ is the normalized adjacency matrix, which keeps the feature scale stable in the graph neural network. Finally, the facial representation v of each AU category i (where i = 1, 2, ..., N) and its activation representation updated by the graph neural network The expression is as follows:
[0058]
[0059] Where σ is a nonlinear activation function, g is a node information update function, n is the total number of nodes in the graph, and a i,j is the element of the adjacency matrix A′, which represents the correlation between the i-th node and the j-th node. r is based on the adjacency matrix a i,j v j A function to re-weight the features.
[0060] In summary, by inputting the representation of AU feature maps as nodes, the graph neural network aggregates the information of the nearest neighbors of each node through a multi-layer propagation strategy, ensuring that the characteristics of each node can fully reflect the common characteristics and differences of all AUs related to it, thereby learning the dynamic and complex relationships between multiple AUs and obtaining the AU relationship feature representation.
[0061] In some specific embodiments, the multi-channel topological feature fusion module CCFG includes a multi-channel topological convolution structure, which learns the relationship between image channels, reorganizes and adjusts features, and constructs a topological relationship. Based on this relationship, joint features in different channels are aggregated for each independently generated AU feature map, and finally multiple AU feature map representations are obtained;
[0062] Specifically, the channel topology convolution structure first transforms the input feature map X through three one-dimensional convolution layers, combining the relational features and geometric features into a topological relationship to emphasize the channel features of the AU. rel1 and Conv1D rel2 Used to extract relationship features, Conv1D β Used to extract geometric features. rel1 and Conv1D rel2 The extracted relational features are subtracted to calculate the relational features between each channel, and then nonlinearly transformed through the tanh activation function to capture the topological structure between the features. The formula is as follows:
[0063]
[0064] in, Represents the element-by-element subtraction between two features. The calculated relationship features are passed through The convolutional layer is fused and multiplied by the adjustment coefficient α, and then the relational features are multiplied by the geometric features through the Einstein summation convention E to achieve the combination of topological structures, so that each channel feature can complement each other and form a complete topological feature as output. The formula is as follows:
[0065]
[0066] Among them, E out is the result of feature multiplication, is a convolutional layer used to fuse relational features, and C represents the number of channels. In the final stage of this branch, a linear block φ containing an average pooling layer and a fully connected layer is used to integrate the channel features corresponding to each type of AU feature map and learn additional weights, and all AU feature maps are combined as the final output to provide the feature representation of the i-th AU, which can be expressed as follows:
[0067]
[0068] By learning the topological relationship between features, dynamically learning and aggregating joint features in different channels, and with a small amount of additional parameters, the model focuses on the AU area from another perspective, and obtains a feature representation rich in channel information.
[0069] In some specific embodiments, the multi-channel topological feature fusion module CCFG includes a feature fusion structure for combining AU feature representations of different feature processing methods. and feature representation that combines channel information with topological structure In order to complement the information of the two branches and improve the performance and generalization ability of the model, the feature splicing and fusion strategy is adopted to combine the feature representations from the two branches.
[0070] Specifically, a fully connected layer is used to perform a linear transformation on the fused feature tensor, thereby adjusting the number of channels to the number of output channels required by the model and learning the weights between each output channel and the input channel. Then, batch normalization and ReLU activation functions are used to normalize and nonlinearly transform the information of each channel as the final AU activation representation. The formula is as follows:
[0071]
[0072] Finally, the present invention adopts a similarity calculation strategy to predict the i-th AU, which provides a similarity calculation strategy to f i Trainable vectors s of the same dimension i , by calculating f i With s i The cosine similarity is used to get the occurrence probability of the i-th AU:
[0073]
[0074] Where ReLU represents a nonlinear activation function. So far, the present invention has balanced the strong correlation information between AUs and the channel information aggregated from the perspective of topological structure for the output information of each type of AU. This enables the model to have two different learning angles to complement each other when processing complex and extreme images. Even if the learning field of view is limited or wrong information is learned from a certain angle, the required information can still be supplemented as much as possible from another angle to improve the performance of the overall model.
[0075] In some specific embodiments, the training is based on the weighted asymmetric loss function to train the backbone network and the multi-scale attention feature generation module, the graph neural network, the multi-channel topological feature fusion module and the graph neural network, and a cosine decay learning rate scheduler is used to more effectively converge to the optimal solution of the model. The backbone network Resnet50 is pre-trained on the large dataset ImageNet to improve the model generalization ability and model training speed.
[0076] The main learning part of the overall network contains the AU activation state and its related node features displayed for each face. It is worth noting that the existing AU dataset labels are unbalanced, with some AUs having low frequencies, while some AUs in most face images are inactive. To address this problem, the present invention introduces a weighted asymmetric loss formula to calculate the loss between the true value and the prediction generated by the face map generator module, setting a unique weight for the recognition of each AU and reducing the use of hyperparameters.
[0077] Specifically, the weighted asymmetric loss formula is:
[0078]
[0079] Among them, p i ,y i and w i Represent the prediction (probability of occurrence), true value and weight of the i-th AU. i for: r i is the occurrence rate of the i-th AU calculated in the training set. This makes the loss value less affected by the AU with a higher occurrence rate in the training set, and focuses on the loss value caused by the AU with a lower occurrence frequency. In addition, (1-y i )log(1-p i ) i The term reduces the weight loss caused by inactive AUs that are easy to identify, and their predicted probability of occurrence is close to zero (p i ≈0.5), which forces the training process to pay more attention to activated AUs and inactive AUs that are difficult to correctly identify.
[0080] This embodiment also provides an electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned facial action unit recognition method.
[0081] This embodiment also provides a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the above-mentioned facial action unit recognition method.
[0082] Experimental results
[0083] The software and hardware environment of the experiment in this embodiment is as follows: deep learning framework Pytorch, i9-12900KF CPU, and Nvidia GTX 3080Ti.
[0084] This example follows the methods of most researchers before and performs three-fold cross validation for each dataset, and reports the average results over three folds. During training, the AdamW optimizer is used, with β1=0:9, β2=0:999, and weight decay is 5e-4. For DISFA, the number of nearest neighbors K in the face generator is set to 4. The hyperparameter lambda in the overall training loss equation is set to 0.05 for the ResNet-based model.
[0085] The experiment trained the model for 40 epochs with an initial learning rate of 1e-4 and a batch size of 64. A cosine decay learning rate scheduler was used to more efficiently converge to the optimal solution of the model. The backbone network Resnet50 has been pre-trained on ImageNet to improve the model generalization ability and model training speed.
[0086] In terms of evaluation indicators, this embodiment follows the standards of existing AU event recognition research and uses the frame-based F1 score and AU average accuracy Acc to evaluate the performance of the model, which is expressed as F1 = 2*P*R / (P+R). P is the recognition accuracy, which measures how many samples of the model are truly positive among all samples judged as positive by the model; R is the recall rate, which measures how many samples of the model are correctly identified as positive among all actual positive samples.
[0087] This example evaluates the performance of the method on a widely used benchmark dataset: DISFA [Mavadati et al, 2013] uses two left and right cameras to record 4845 frames from 27 subjects (12 women and 15 men) watching a video. Seven expressions are induced by having the subjects watch a video: anger, disgust, surprise, fear, sadness, happiness, and neutral, where each frame is annotated with multiple AU occurrence labels; EmotioNet is a facial AU recognition dataset in a natural uncontrolled environment, collected from the Internet, and its face images cover a variety of backgrounds, complex perspectives, and human occlusion. Experts provided manual annotations of 12 AUs for 24,598 of the face images.
[0088] For the preprocessing of the data set, this embodiment uses the ERT cascade regression algorithm for face detector and face key point detection in the Dlib tool library used by many researchers before facial expression recognition. After extracting the video frame, the function dlib.get_frontal_face_detector() of the Dlib open source library is used to implement face detection, and then the model shape_predictor_68_face_landmarks.dat for facial feature point calibration in the Dlib library is used to identify 68 facial key points for each video frame image. Through affine transformations such as translation, rotation and scaling, specific facial feature points are aligned to a fixed position to complete facial alignment to reduce the differences in the position and angle of the face in different images. Finally, the image is cropped to 224*224 as the input of the backbone network.
[0089] The present invention conducts experiments on 8 AUs (au1; au2; au4; au6; au9; au12; au25; au26) of DISFA reasoning and 12 AUs (au1; au2; au4; au5; au6; au9; au12; au17; au20; au25; au26; au43) of EmotioNet dataset reasoning. The average results of the ablation experiments are shown in Table 1:
[0090] Table 1
[0091] Model F1 Acc Baseline (resnet50+graph neural network) 61.35 91.86 Baseline+ Multi-scale Attention Feature Generation Module (MAFG) 63.52 92.95 Baseline+ multi-channel topological feature fusion module (CCFG) 63.48 92.92 Baseline+MAFG+CCFG 66.62 93.87
[0092] The present invention proposes an AU recognition method that integrates channel topological convolution and attention mechanism. By adopting a parallel dual-branch structure, the channel topological convolution and the multi-scale attention mechanism are effectively combined with the graph neural network, and the feature extraction and relationship modeling of the facial action unit are optimized. By integrating an adaptive channel topological convolution module in the backbone network, the ability to capture complex facial geometric features is enhanced. At the same time, through the introduction of the multi-scale attention mechanism, the model focuses on AU features with higher discrimination, suppresses the weight of non-AU areas, locates local AU areas with more detailed global features, and uses the relationship modeling of the graph neural network to express the adjacency relationship between AUs, and enhances the model's understanding of the dynamic correlation of the activation states between AUs, thereby significantly improving the recognition accuracy and the robustness of the system. Experimental results show that the F1 score and recognition accuracy of this method in the facial action unit dataset DISFA reached 65.39 and 93.26% respectively, which has better recognition performance than other advanced methods and effectively improves the recognition accuracy and robustness of facial action units.
[0093] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An AU recognition method integrating channel topology convolution and attention mechanism, characterized in that: The following steps are involved: Constructing a facial action unit recognition model; the facial action unit recognition model includes a backbone network, a multi-scale attention feature generation module, a graph neural network, a multi-channel topological feature fusion module, and a prediction module; Acquire a training data set, wherein the training data set includes a plurality of face images having multiple AU occurrence labels; Constructing a weighted asymmetric loss function, and training the facial action unit recognition model based on a training data set and the weighted asymmetric loss function; Obtain a face image to be recognized, input it into the trained facial action unit recognition model, and obtain the facial action unit recognition result.
2. The AU identification method integrating channel topology convolution and attention mechanism according to claim 1, characterized in that: The backbone network adopts Resnet50, and the face image is input into the backbone network. The backbone network forward propagates and extracts facial features layer by layer to obtain facial feature representation.
3. The AU identification method integrating channel topology convolution and attention mechanism according to claim 2, characterized in that: The multi-scale attention feature generation module MAFG adopts a multi-scale channel attention mechanism, inputs the facial feature representation output by the backbone network into the multi-scale channel attention module, performs a multi-scale feature extraction operation on the facial feature representation to obtain a multi-scale feature, performs a channel mixing operation on the multi-scale feature to obtain a mixed feature, and performs element-by-element multiplication of the mixed feature and the facial feature representation to obtain an enhanced feature representation; The features after multi-scale attention are passed through a linear block containing a fully connected layer FC and a normalization layer to generate multiple AU feature map representations.
4. The AU identification method integrating channel topology convolution and attention mechanism according to claim 3, characterized in that: In the graph neural network stage, a multi-scale attention mechanism is first used to generate AU feature maps from the main view at different scales, and the initial feature representation of the relationship modeling stage is obtained. An adjacency matrix is constructed for the relationship learning of the graph neural network. The multi-scale attention mechanism aggregates multi-scale information, and enhances the expression ability of its activation state during AU relationship learning. The graph neural network uses a multi-layer propagation strategy to make each feature representation as the information of the node's nearest neighbor aggregation, and iterative learning gradually integrates potential long-distance adjacency relationships, and finally obtains the AU relationship feature representation after graph association learning.
5. The AU identification method integrating channel topology convolution and attention mechanism according to claim 3, characterized in that: The multi-channel topological feature fusion module CCFG includes a channel topological convolution structure, which learns the relationship between image channels, reorganizes and adjusts features, and constructs a topological relationship. Based on this relationship, the joint features in different channels are aggregated for each independently generated AU feature map, and finally multiple AU feature map representations are obtained.
6. The AU identification method integrating channel topology convolution and attention mechanism according to claim 1, characterized in that: The multi-channel topological feature fusion module CCFG includes a feature fusion structure, which uses a splicing fusion method to balance the information of the two branches, splicing the joint features of CCFG to the AU relationship features, so that the model has two different learning angles to complement each other when processing complex and extreme images.
7. The AU identification method integrating channel topology convolution and attention mechanism according to claim 1, characterized in that: The weighted asymmetric loss function is: Among them, p i ,y i and w i Represent the predicted value, true value and weight of the occurrence probability of the i-th AU, r i is the occurrence rate of the i-th AU calculated in the training set.
8. The AU identification method integrating channel topology convolution and attention mechanism according to claim 5, characterized in that: The process of training the facial action unit recognition model includes: Based on the weighted asymmetric loss function, the backbone network and the multi-scale attention feature generation module, the graph neural network, the multi-channel topological feature fusion module and the graph neural network are trained, and a cosine decay learning rate scheduler is adopted to more effectively converge to the optimal solution of the model; the backbone network Resnet50 is pre-trained on the large dataset ImageNet to improve the model generalization ability and model training speed.
9. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the facial action unit recognition method as described in any one of claims 1 to 8.
10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the facial action unit recognition method according to any one of claims 1 to 8.
Citation Information
Cited By
Neural network model-based 3D model scoring method and system, terminal and medium
CN121168536A
