Cross-domain small sample expression recognition method based on consistency of action units

Abstract

The invention discloses a cross-domain small sample expression recognition method based on motion unit consistency. The method comprises the following steps: collecting and preprocessing a multi-source basic expression data set and a composite expression data set; enhancing the original facial action unit features, and constructing a hypergraph structure; facial action unit relation graphs of the original facial action unit features and the enhanced facial action unit features are extracted respectively; respectively calculating two groups of attention distributions by adopting double paths based on the facial action unit relation graph, and taking a difference value of the two groups of attention as a contrast attention module; constructing a double-branch collaborative learning network, wherein the double-branch collaborative learning network comprises an airspace information mining branch, a context relation mining branch and a cross-branch feature fusion module; performing joint optimization on the double-branch collaborative learning network through the training set; and performing new expression classification on the query set extracted from the test set by using the trained double-branch collaborative learning network, and calculating the classification accuracy. According to the method, the recognition capability of the model on cross-domain small sample expressions is effectively enhanced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition technology, and in particular to a cross-domain few-sample facial expression recognition method based on action unit consistency. Background Technology

[0002] Facial expressions are one of the most intuitive and effective ways for humans to convey emotional signals. In recent years, facial expression recognition has become an important research direction in the fields of computer vision and artificial intelligence, and has been widely applied in various practical scenarios such as safe driving, medical monitoring, and online education.

[0003] According to research by renowned psychologists Ekman et al., facial expressions can be categorized into seven basic expression classes: anger, surprise, disgust, fear, happiness, sadness, and neutrality. Based on this, researchers have collected a series of basic expression datasets (such as CK+, OULU, MMI, RAF, SFEW, etc.) and conducted research on automated facial expression recognition. However, these basic expression classes struggle to capture the diversity of real-world facial expressions. Compared to basic expressions, complex expressions can describe more complex and nuanced emotions, and have significant application value in real-world scenarios. Due to the fine-grained nature of complex expressions, manually labeling them is not easy; the labeling cost for complex expressions is higher than that for basic expression data, and usually requires professional guidance from psychologists. When faced with unknown emotions, only a small number of observations are needed to identify a new, unknown emotion. This learning paradigm, which rapidly generalizes to new classes based on a small number of labeled samples, is called few-shot learning in computer science. Although few-shot learning has been extensively studied in areas such as image recognition and object detection, it has been rarely explored in the field of facial expression recognition. Zou et al. (Xinyi Zou, Yan Yan, Jing-Hao Xue, Si Chen, and Hanzi Wang. 2022. Learn-to-decompose: Cascaded decomposition network for cross-domain few-shot facial expression recognition, European conference on computer vision. 683-700.) first studied cross-domain few-shot facial expression recognition (CF-FER). This paradigm trains the model using only readily available basic facial expression data and identifies novel complex facial expression data in a small-sample setting, greatly reducing the burden of collecting large-scale labeled complex facial expression training data.

[0004] Existing CF-FER methods mainly face the following problems in facial expression recognition tasks: (1) The basic expression categories are limited, and the sampling repetition of the meta-task is high, which can easily lead to overfitting; (2) Under cross-domain conditions (such as from basic indoor expressions to complex outdoor expressions), the distribution differences between domains are large, and the model has poor adaptability; (3) The lack of modeling of the structured relationships between facial action units (AUs) makes it difficult to capture subtle feature changes in complex expressions. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a cross-domain few-sample expression recognition method based on action unit consistency. This invention combines a relation consistency constraint module and a contrast attention module to design a dual-branch collaborative learning network (model) for spatial information and context relation mining, which effectively improves the model's feature representation ability and cross-domain generalization performance, and enhances the model's ability to recognize cross-domain few-sample expressions.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: This invention provides a cross-domain few-sample facial expression recognition method based on action unit consistency, comprising the following steps: Step 1: Collect a multi-source basic facial expression dataset as the training set and a composite facial expression dataset as the test set, and perform data preprocessing; Step 2: Extract image features from the training set, obtain the original facial action unit features from the image features, enhance them to generate enhanced facial action unit features, and construct a hypergraph structure by using the original facial action unit features and enhanced facial action unit features as nodes and their relationships as edges. Step 3: Extract facial action unit relationship graphs based on the hypergraph structure, namely the original facial action unit features and the enhanced facial action unit features; Step 4: Based on the facial action unit relationship graph, calculate two sets of attention distributions using a dual-path approach, and use the difference between the two sets of attention as the comparison attention module; Step 5: Construct a dual-branch collaborative learning network, including: a spatial information mining branch, a context relationship mining branch, and a cross-branch feature fusion module; In the spatial information mining branch, the original facial action unit features are extracted, concatenated with image features, and input into the contrast attention module along with the image semantic description to generate spatial information. In the context relationship mining branch, a hypergraph neural network is used to update nodes, and an attention module is used to compare image features with AU semantic description inputs to generate context relationship representations; In the cross-branch feature fusion module, the spatial information and contextual relationship representation are fused. Step 6: During the training phase, the dual-branch collaborative learning network is jointly optimized using the training set; Step 7: During the testing phase, extract a query set from the test set, use the trained dual-branch collaborative learning network to classify new types of facial expressions on the query set, and calculate the classification accuracy of the query set.

[0007] Furthermore, step 1 specifically includes: Step 11: The training set used is selected from five basic facial expression datasets, including three indoor datasets: CK+, MMI, and OULU, and two outdoor datasets: RAF and SFEW. Each basic facial expression dataset contains basic facial expression labels: anger, surprise, disgust, fear, happiness, sadness, and neutrality. The CK+ dataset also includes a contemptuous expression label. Each basic facial expression dataset provides image facial expression labels as image semantic descriptions and facial action unit labels as AU semantic descriptions. Step 12: The test set used is selected from two composite expression datasets, including: the indoor composite expression dataset CFEE and the outdoor composite expression dataset EmotioNet, whose expression categories are different from those of the training set; Step 13: Both the training set and the test set are processed by a multi-task cascaded convolutional neural network for face detection and key point localization. Affine transformation is performed based on the key points to complete face alignment. All aligned face region images are uniformly cropped and transformed to a preset pixel size as input to the dual-branch collaborative learning network.

[0008] Furthermore, step 2 specifically includes: The characteristic feature is that step 2 specifically includes: Step 21: Extract image features from the training set Using AU-Net to extract image features Obtain the original facial action unit feature set For the nth original facial action unit features By applying random color jitter and brightness / contrast adjustment as appearance enhancement, and random rotation and scaling as geometric enhancement, enhanced facial motion unit features are obtained. Iterate through each original facial action unit feature to obtain the corresponding enhanced facial action unit feature set. Where n=1,2,…,N represents the index of the original facial motion unit feature; i represents the index of the image feature; N represents the number of original facial motion unit features; Step 22: Treat each original facial action unit feature as a node, and the co-occurrence relationship between original facial action unit features as edges, and construct a hypergraph structure. :

[0009] Among them, the node set , represents all nodes in the hypergraph structure, corresponding to the original facial action unit features; This represents a set of hyperedges used to connect two or more nodes simultaneously, describing higher-order relationships, and corresponding to the co-occurrence relationships among multiple original facial action unit features. Represents the node feature matrix, , among which, | | represents the number of nodes, and d represents the dimension of the feature of each node; Represents the set of real numbers; Step 23: Weight the hyperedge set according to the co-occurrence relationship between multiple original facial action unit features. Initialization is performed using the following formula:

[0010] Here, node n and node m correspond to different original facial action unit features. This represents the edge weight between node n and node m. For the nth original facial action unit feature corresponding to node n, Let m be the m-th original facial action unit feature corresponding to node m, sim() be the feature similarity function, and n and m represent different original facial action unit features and node indices, n=1,2,…,N; m=1,2,…,N; Step 24: For each enhanced facial motion unit feature, repeat steps 22-23.

[0011] Furthermore, step 3 specifically includes: Step 31: Use a hypergraph neural network to extract facial action unit (FAU) relationship graphs of the original facial action unit features and the enhanced facial action unit features, respectively. Perform message propagation on the node features in the facial action unit relationship graph. The node feature update formula is:

[0012] in, Indicates the first l Features of node n at level +1; For activation functions; Let k be the set of neighbors of node n, and k be the neighboring nodes of node n. To learn the weight matrix, This represents the edge weight between node n and node m. This represents the weight normalization factor for the nodes; For the first l Features of layer node m; Step 32: Process the original facial action unit feature set and enhanced facial action unit feature set After propagation through the hypergraph neural network, the node features of the final layer are obtained: and Calculate the graph structure consistency loss between the original facial action unit features and the enhanced facial action unit features. : Step 33: Maintain feature centers for the category of each original facial action unit feature. For the set of facial action unit labels activated in the original facial action unit features Calculate the multi-label center loss :

[0013] in, The characteristics of the neighboring node k of node n; Step 34: Combine the graph structure consistency loss and the multi-label center loss to form the total loss. :

[0014] in, This represents a hyperparameter used to balance the two losses.

[0015] Furthermore, step 4 specifically includes: Step 41: Using the image semantic description as the query vector Q, the visual features as the key vector K, and the value vector V, perform linear transformations on the input query vector Q and key vector K through two paths respectively, generating two sets of vectors: { , }and{ , }; Step 42: Calculate the two sets of attention weight matrices respectively:

[0016]

[0017] in, Represents the normalized exponential function, express The transpose of the matrix, express The transpose of the matrix, It is the dimension of the key vector; Step 43: Obtain the contrastive attention matrix by using the difference in attention weights between the two paths:

[0018] in, These are dynamic weight parameters used to adjust the suppression strength of the second path; Step 44: Use the contrastive attention matrix to perform a weighted summation on the value vector V to obtain the enhanced feature representation:

[0019] Step 45: Perform the feature representations output from steps 41-44 on each attention head. After splicing, linear transformation and grouping normalization are performed to obtain the final enhanced features.

[0020] Furthermore, the processing procedure for the spatial information mining branch in step 5 is as follows: (1) Using the AU-Net network to extract features from the input image Extract the original facial action unit feature set ; (2) Set up the original facial action unit features Image features By splicing the images together, we obtain the merged visual features. ; (3) Visual features Image semantic description The input is fed into the contrastive attention module to calculate spatial information. ;in, This represents the computation function for the contrastive attention module. The first one As a value vector, the second As a key vector As a query vector.

[0021] Furthermore, the specific processing procedure for the context relationship mining branch in step 5 is as follows: (1) Update node features through hypergraph neural network propagation:

[0022] (2) The node features of the final layer obtained after propagation through the hypergraph neural network. Image features By stitching the images together, deep visual features are obtained. ; (3) Deep visual features With AU semantic description The input is fed into the contrastive attention module to generate a contextual relationship representation. The first one As a value vector, the second As a key vector As a query vector.

[0023] Furthermore, the processing procedure of the cross-branch feature fusion module in step 5 is as follows: (1) Spatial information output by the spatial information mining branch Contextual relationship representation output from the contextual relationship mining branch Input into the cross-attention module; (2) Calculate spatial information through the cross-attention module. Representation of contextual relationships The correlation between them, and characterization from the contextual relationship Extracting spatial information Complementary information, its output is the final enhanced feature after fusion. :

[0024] Furthermore, step 6 specifically includes: Step 61: In each scenario during the training process, randomly select a source domain from all training sets for sampling; Step 62: Randomly sample N expression categories from the source domain, and randomly sample K samples from each expression category as the support set. Q samples as the query set Construct an N-way K-shot few-shot classification task; where, The sample represents the support set. The set of tags representing the support set. This represents a sample of the query set. This represents the set of actual labels for the query set; Step 63: Process the samples from the support set and query set separately using the similarity feature extractor. Forward propagation computes image features, which are then processed by the metric module. Calculate the sample distance between the support set and the query set, and assign samples from the query set to the nearest neighbor support set; its mathematical representation is as follows:

[0025] in, This represents the prediction result for a sample of the query set. M represents the classification function, and M represents the metric module used to calculate the distance or similarity between two feature vectors. Step 64: During the training process, calculate the prediction result for a single sample in the query set. With real labels Cross-entropy loss between Used to update the parameters of the two-branch collaborative learning network:

[0026] in, To query the prediction results for a single sample in the query set, This represents the true label of a single sample in the query set, and n' represents the index variable for iterating over the total number of expression categories N', where N' represents the total number of expression categories in the current task. This indicates an indicator function, when n' equals When n' is not equal to 1, the value of the indicator function is 1; when n' is not equal to 1, the value of the indicator function is 1. When the value is 0, the indicator function is 0.

[0027] Furthermore, step 7 specifically includes: Step 71: During the testing phase, randomly sample N' expression categories from the composite expression dataset. In each expression category, sample K samples as the support set and Q samples as the query set. Step 72: Use the trained dual-branch collaborative learning network to calculate the class prototypes for the samples in the support set, and perform classification predictions for all N'*Q samples in the query set based on the class prototypes. Calculate the classification accuracy of all samples in the query set as the performance evaluation index of the current task. Step 73: Repeat steps 71-72 to complete multiple sampling tasks, and take the average classification accuracy of all tasks as the final performance evaluation index of the dual-branch collaborative learning network.

[0028] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This invention is based on a dual-branch collaborative learning network with a contrastive attention module and a relation consistency constraint module. By explicitly modeling region differences through the contrastive attention module, the network's sensitivity to subtle facial expression regions is enhanced, improving its ability to identify fine-grained facial expression regions. A hypergraph neural network is combined to model the relationship between AUs and facial expression categories, characterizing the co-occurrence and mutual exclusion relationships of AUs, and a consistency constraint is introduced to ensure the stability of the relationship modeling. Furthermore, a branch for spatial information and contextual relationship mining is designed, and cross-branch fusion achieves efficient complementarity between spatial and contextual information. This invention trains the dual-branch collaborative learning network on multiple basic facial expression datasets. With only a small number of target domain samples, it can quickly generalize existing information to new types of facial expression recognition tasks in unknown domains. Competitive results have been achieved on both indoor and outdoor composite facial expression datasets, significantly improving the robustness and generalization of facial expression recognition in cross-domain, small-sample scenarios, providing a more flexible and effective solution for facial expression recognition in complex real-world environments. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is an execution flowchart of a cross-domain few-sample facial expression recognition method based on action unit consistency provided in an embodiment of the present invention.

[0031] Figure 2 This is a diagram of the entire network framework provided in an embodiment of the present invention.

[0032] Figure 3 This is a framework diagram of the relationship consistency constraint module provided in an embodiment of the present invention.

[0033] Figure 4 A framework diagram of the contrast attention module provided in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figures 1-4 The present invention provides a cross-domain few-sample facial expression recognition method based on action unit consistency, comprising the following steps: Step 1: Construct the dataset: Collect a multi-source basic facial expression dataset as the training set and a composite facial expression dataset as the test set, and perform data preprocessing; In this embodiment, step 1 specifically includes: Step 11: The training set used is selected from five basic facial expression datasets, including three indoor datasets: CK+, MMI, and OULU, and two outdoor datasets: RAF and SFEW. The indoor datasets were captured in indoor environments with controllable external factors such as lighting and pose. The outdoor datasets refer to datasets collected under natural conditions with uncontrollable external factors. The RAF dataset was downloaded from the internet, and the SFEW dataset contains video clips extracted from movies. Each basic facial expression dataset contains basic expression labels: anger, surprise, disgust, fear, happiness, sadness, and neutrality. The CK+ dataset also includes a contempt expression label. Each basic facial expression dataset provides image expression labels as image semantic descriptions and facial action unit (AU) labels as AU semantic descriptions. Step 12: The test set used is selected from two composite expression datasets, including: the indoor composite expression dataset CFEE and the outdoor composite expression dataset EmotioNet. Their expression categories are different from those of the training set, and they have more granular expression annotations. Step 13: Both the training set and the test set are processed by a multi-task cascaded convolutional network (MTCNN, an advanced deep learning framework mainly used for face detection and face alignment) for face detection and key point localization. Affine transformation is performed based on the key points to complete face alignment. All aligned face region images are uniformly cropped and transformed to a preset pixel size as input to the dual-branch collaborative learning network.

[0036] Design the relationship consistency constraint module (including steps 2 and 3): Step 2, Sample Augmentation and Hypergraph Construction: Extract image features from the training set, obtain the original facial action unit features from the image features, augment the original facial action unit features to generate augmented facial action unit features, and construct a hypergraph structure by using the original facial action unit features and the augmented facial action unit features as nodes and their relationships as edges. In this embodiment, step 2 specifically includes: The characteristic feature is that step 2 specifically includes: Step 21: Extract image features from the training set Using AU-Net to extract image features Obtain the original facial action unit feature set For the nth original facial action unit features Random color jitter and brightness / contrast adjustment are applied as appearance enhancements, while random rotation and scaling are applied as geometric enhancements. By simulating complex environments through appearance and geometric enhancements, enhanced facial motion unit features are obtained. Iterate through each original facial action unit feature to obtain the corresponding enhanced facial action unit feature set. Where n=1,2,…,N represents the index of the original facial motion unit feature; i represents the index of the image feature; N represents the number of original facial motion unit features; Step 22: Treat each original facial action unit feature as a node, and the co-occurrence relationship between original facial action unit features as edges, and construct a hypergraph structure. :

[0037] Among them, the node set , represents all nodes in the hypergraph structure, corresponding to the original facial action unit features; This represents a set of hyperedges used to connect two or more nodes simultaneously, describing higher-order relationships, and corresponding to the co-occurrence relationships among multiple original facial action unit features. Represents the node feature matrix, , among which, | | represents the number of nodes, and d represents the dimension of the feature of each node; Represents the set of real numbers; Step 23: Weight the hyperedge set according to the co-occurrence relationship between multiple original facial action unit features. Initialization is performed using the following formula:

[0038] Here, node n and node m correspond to different original facial action unit features. This represents the edge weight between node n and node m. For the nth original facial action unit feature corresponding to node n, Let m be the m-th original facial action unit feature corresponding to node m, sim() be the feature similarity function, and n and m represent different original facial action unit features and node indices, n=1,2,…,N; m=1,2,…,N; Step 24: For each enhanced facial motion unit feature, repeat steps 22-23.

[0039] Step 3: Extract facial action unit relationship graphs based on the hypergraph structure, namely the original facial action unit features and the enhanced facial action unit features; In this embodiment, step 3 specifically includes: Step 31, Hypergraph Feature Propagation: The original facial action unit feature set is extracted using a hypergraph neural network (HgnnCycle). Each original facial action unit feature and augmented facial action unit feature set For each enhanced facial action unit feature in the facial action unit relationship graph, message propagation is performed on the node features in the facial action unit relationship graph, and the node feature update formula is:

[0040] in, Indicates the first l Features of node n at level +1; For activation functions; Let k be the set of neighbors of node n, and k be the neighboring nodes of node n. To learn the weight matrix, This represents the edge weight between node n and node m. This represents the weight normalization factor for the nodes; For the first l Features of layer node m; Step 32, Graph Structure Consistency Constraint: For the original facial action unit feature set and enhanced facial action unit feature set After propagation through the hypergraph neural network, the node features of the final layer are obtained: and Calculate the graph structure consistency loss between the original facial action unit features and the enhanced facial action unit features. : Graph structure consistency loss Used to calculate structural differences and achieve structural reinforcement of AU co-occurrence relationships; Step 33, Multi-label center loss: Maintain feature centers for each original facial action unit feature category. For the set of facial action unit labels activated in the original facial action unit features Calculate the multi-label center loss :

[0041] in, Features of neighboring nodes k of node n; multi-label center loss Used for reinforcement AU features clustering properties; Step 34: Combine the graph structure consistency loss and the multi-label center loss to form the total loss. :

[0042] in, The hyperparameters are used to balance the two losses. The co-occurrence relationships of facial action unit features are structurally reinforced through graph structure consistency loss and multi-label center loss, thereby improving the stability and expression recognition capabilities of the two-branch collaborative learning network.

[0043] Step 4: Design the contrastive attention module: Based on the facial action unit relationship graph, two sets of attention distributions are calculated using a dual-path approach, and the difference between the two sets of attention is used as the contrastive attention module to model regional differences, highlight key expression areas, and suppress redundant and noisy areas. In this embodiment, step 4 specifically includes: Step 41: Using the image semantic description as the query vector Q, the visual features as the key vector K, and the value vector V, perform linear transformations on the input query vector Q and key vector K through two paths respectively, generating two sets of vectors: { , }and{ , }; Step 42: Calculate the two sets of attention weight matrices respectively:

[0044]

[0045] in, Represents the normalized exponential function, express The transpose of the matrix, express The transpose of the matrix, It is the dimension of the key vector; Step 43: Obtain the contrastive attention matrix by using the difference in attention weights between the two paths:

[0046] in, These are dynamic weight parameters used to adjust the suppression strength of the second path; Step 44: Use the contrastive attention matrix to perform a weighted summation on the value vector V to obtain the enhanced feature representation:

[0047] Step 45: Perform the feature representations output from steps 41-44 on each attention head. After splicing, linear transformation and grouping normalization are performed to obtain the final enhanced features.

[0048] Step 5: Construct a dual-branch collaborative learning network, including: a spatial information mining branch, a context relationship mining branch, and a cross-branch feature fusion module; In the spatial information mining branch, the original facial action unit features are extracted, concatenated with image features, and input along with the image semantic description into the contrast attention module to generate spatial information. The specific processing procedure of the spatial information mining branch is as follows: (1) Using the AU-Net network to extract features from the input image Extract the original facial action unit feature set ; (2) Set up the original facial action unit features Image features By splicing the images together, we obtain the merged visual features. ; (3) Visual features Image semantic description The input is fed into the contrastive attention module to calculate spatial information. ;in, This represents the computation function for the contrastive attention module. The first one As a value vector, the second As a key vector As the query vector, in the context relationship mining branch, nodes are updated using a hypergraph neural network, and a context relationship representation is generated by combining image features with the AU semantic description input and contrasting them with an attention module; the specific processing procedure of the context relationship mining branch is as follows: (1) Update node features through hypergraph neural network propagation:

[0049] (2) The node features of the final layer obtained after propagation through the hypergraph neural network. Image features By stitching the images together, deep visual features are obtained. ; (3) Deep visual features With AU semantic description The input is fed into the contrastive attention module to generate a contextual relationship representation. The first one As a value vector, the second As a key vector As a query vector.

[0050] In the cross-branch feature fusion module, the spatial information and contextual relationship representation are fused; the specific processing procedure of the cross-branch feature fusion module is as follows: (1) Spatial information output by the spatial information mining branch Contextual relationship representation output from the contextual relationship mining branch Input into the cross-attention module; (2) Calculate spatial information through the cross-attention module. Representation of contextual relationships The correlation between them, and characterization from the contextual relationship Extracting spatial information Complementary information, its output is the final enhanced feature after fusion. :

[0051] By using a cross-branch feature fusion module, the spatial information and contextual relationship representation are complemented, thereby improving the accuracy and robustness of facial expression recognition.

[0052] Step 6, Network Model Training: During the training phase, the dual-branch collaborative learning network is jointly optimized using the training set; In this embodiment, step 6 specifically includes: Step 61: In each scenario during the training process, randomly select one source domain from all training sets for sampling; this is to prevent overfitting to repeated sampling tasks due to the limited number of basic expression categories. Overall, scenario training samples from multiple domains to enhance the diversity of training data (data / meta-task) and alleviate the domain difference problem between the training source domain and the test target domain.

[0053] Step 62: Randomly sample N expression categories from the source domain, and randomly sample K samples from each expression category as the support set. Q samples as the query set We construct an N-way K-shot few-shot few-shot classification task (meta-task) (a task description format that defines the specific settings of a few-shot classification task: N-way: indicates that there are N different classes involved in this task; K-shot: indicates that for each of the N classes, we only provide K labeled samples as references); where, The sample represents the support set. The set of tags representing the support set. This represents a sample of the query set. This represents the set of true labels for the query set; the purpose of the similarity branch is to use a small number of labeled sample information from the support set to complete the classification task of the query set.

[0054] Step 63: Process the samples from the support set and query set separately using the similarity feature extractor. Forward propagation computes image features, which are then processed by the metric module. Calculate the sample distance between the support set and the query set, and assign samples from the query set to the nearest neighbor support set; its mathematical representation is as follows:

[0055] in, This represents the prediction result for a sample of the query set. Let M represent the classification function, and let M represent the metric module used to calculate the distance or similarity between two feature vectors. Existing baseline methods for small samples have studied various metric module forms such as Euclidean distance, cosine distance, and learning-based methods. Experiments demonstrate that the joint and alternating learning framework proposed in this method can effectively improve the performance of the aforementioned baseline methods on novel facial expression recognition problems. Step 64: During the training process, calculate the prediction result for a single sample in the query set. With real labels Cross-entropy loss between Used to update the parameters of the two-branch collaborative learning network:

[0056] in, To query the prediction results for a single sample in the set, This represents the true label of a single sample in the query set, and n' represents the index variable for iterating over the total number of expression categories N', where N' represents the total number of expression categories in the current task. This indicates an indicator function, when n' equals When n' is not equal to 1, the value of the indicator function is 1; when n' is not equal to 1, the value of the indicator function is 1. When the value is 0, the indicator function is 0.

[0057] Step 7, Network Model Testing: During the testing phase, a query set is extracted from the test set. The trained dual-branch collaborative learning network is used to classify new types of facial expressions on the query set, and the classification accuracy of the query set is calculated.

[0058] In this embodiment, step 7 specifically includes: Step 71: During the testing phase, randomly sample N' expression categories from the composite expression dataset. In each expression category, sample K samples as the support set and Q samples as the query set. Step 72: Use the trained dual-branch collaborative learning network to calculate the class prototypes for the samples in the support set, and perform classification predictions for all N'*Q samples in the query set based on the class prototypes. Calculate the classification accuracy of all samples in the query set as the performance evaluation index of the current task. Step 73: Repeat steps 71-72 to complete multiple sampling tasks, and take the average classification accuracy of all tasks as the final performance evaluation index of the dual-branch collaborative learning network.

[0059] In the verification experiment of this invention, the randomly sampled category N'=5 was defined, and the performance of the model was evaluated when K=1 and K=5 respectively. Each meta-task used Q=16, that is, a total of 5*16=80 query samples were classified. Finally, the average accuracy of 1000 meta-tasks was used to evaluate the performance of this invention.

[0060] Table 1 shows the accuracy comparison of the present invention with other small-sample methods on the CFEE and EmotioNet test datasets.

[0061]

[0062] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cross-domain few-sample facial expression recognition method based on action unit consistency, characterized in that, Includes the following steps: Step 1: Collect a multi-source basic facial expression dataset as the training set and a composite facial expression dataset as the test set, and perform data preprocessing; Step 2: Extract image features from the training set, obtain the original facial action unit features from the image features, enhance them to generate enhanced facial action unit features, and construct a hypergraph structure by using the original facial action unit features and enhanced facial action unit features as nodes and their relationships as edges. Step 3: Extract facial action unit relationship graphs based on the hypergraph structure, namely the original facial action unit features and the enhanced facial action unit features; Step 4: Based on the facial action unit relationship graph, calculate two sets of attention distributions using a dual-path approach, and use the difference between the two sets of attention as the comparison attention module; Step 5: Construct a dual-branch collaborative learning network, including: a spatial information mining branch, a context relationship mining branch, and a cross-branch feature fusion module; In the spatial information mining branch, the original facial action unit features are extracted, concatenated with image features, and input into the contrast attention module along with the image semantic description to generate spatial information. In the context relationship mining branch, the nodes are updated using a hypergraph neural network, and the attention module is used to compare the input of image features and AU semantic description to generate context relationship representations. In the cross-branch feature fusion module, the spatial information and contextual relationship representation are fused. Step 6: During the training phase, the dual-branch collaborative learning network is jointly optimized using the training set; Step 7: During the testing phase, extract a query set from the test set, use the trained dual-branch collaborative learning network to classify new types of facial expressions on the query set, and calculate the classification accuracy of the query set.

2. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 1 specifically includes: Step 11: The training set used is selected from five basic facial expression datasets, including three indoor datasets: CK+, MMI, and OULU, and two outdoor datasets: RAF and SFEW. Each basic facial expression dataset contains basic facial expression labels: anger, surprise, disgust, fear, happiness, sadness, and neutrality. The CK+ dataset also includes a contemptuous expression label. Each basic facial expression dataset provides image facial expression labels as image semantic descriptions and facial action unit labels as AU semantic descriptions. Step 12: The test set used is selected from two composite expression datasets, including: the indoor composite expression dataset CFEE and the outdoor composite expression dataset EmotioNet, whose expression categories are different from those of the training set; Step 13: Both the training set and the test set are processed by a multi-task cascaded convolutional neural network for face detection and key point localization. Affine transformation is performed based on the key points to complete face alignment. All aligned face region images are uniformly cropped and transformed to a preset pixel size as input to the dual-branch collaborative learning network.

3. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 2 specifically includes: Step 21: Extract image features from the training set Using AU-Net to extract image features Obtain the original facial action unit feature set For the nth original facial action unit features By applying random color jitter and brightness / contrast adjustment as appearance enhancement, and random rotation and scaling as geometric enhancement, enhanced facial motion unit features are obtained. Iterate through each original facial action unit feature to obtain the corresponding enhanced facial action unit feature set. Where n=1,2,…,N represents the index of the original facial motion unit feature; i represents the index of the image feature; N represents the number of original facial motion unit features; Step 22: Treat each original facial action unit feature as a node, and the co-occurrence relationship between original facial action unit features as edges, and construct a hypergraph structure. : Among them, the node set , represents all nodes in the hypergraph structure, corresponding to the original facial action unit features; This represents a set of hyperedges used to connect two or more nodes simultaneously, describing higher-order relationships, and corresponding to the co-occurrence relationships among multiple original facial action unit features. Represents the node feature matrix, , among which, | | represents the number of nodes, and d represents the dimension of the feature of each node; Represents the set of real numbers; Step 23: Weight the hyperedge set according to the co-occurrence relationship between multiple original facial action unit features. Initialization is performed using the following formula: Here, node n and node m correspond to different original facial action unit features. This represents the edge weight between node n and node m. For the nth original facial action unit feature corresponding to node n, Let m be the m-th original facial action unit feature corresponding to node m, sim() be the feature similarity function, and n and m represent different original facial action unit features and node indices, n=1,2,…,N; m=1,2,…,N; Step 24: For each enhanced facial motion unit feature, repeat steps 22-23.

4. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 3 specifically includes: Step 31: Use a hypergraph neural network to extract facial action unit (FAU) relationship graphs of the original facial action unit features and the enhanced facial action unit features, respectively. Perform message propagation on the node features in the facial action unit relationship graph. The node feature update formula is: in, Indicates the first l Features of node n at level +1; For activation functions; Let k be the set of neighbors of node n, and k be the neighboring nodes of node n. To learn the weight matrix, This represents the edge weight between node n and node m. This represents the weight normalization factor for the nodes; For the first l Features of layer node m; Step 32: Process the original facial action unit feature set and enhanced facial action unit feature set After propagation through the hypergraph neural network, the node features of the final layer are obtained: and Calculate the graph structure consistency loss between the original facial action unit features and the enhanced facial action unit features. : Step 33: Maintain feature centers for the category of each original facial action unit feature. For the set of facial action unit labels activated in the original facial action unit features Calculate the multi-label center loss : in, The characteristics of the neighboring node k of node n; Step 34: Combine the graph structure consistency loss and the multi-label center loss to form the total loss. : in, This represents a hyperparameter used to balance the two losses.

5. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 4 specifically includes: Step 41: Using the image semantic description as the query vector Q, the visual features as the key vector K, and the value vector V, perform linear transformations on the input query vector Q and key vector K through two paths respectively, generating two sets of vectors: { , }and{ , }; Step 42: Calculate the two sets of attention weight matrices respectively: in, Represents the normalized exponential function, express The transpose of the matrix, express The transpose of the matrix, It is the dimension of the key vector; Step 43: Obtain the contrastive attention matrix by using the difference in attention weights between the two paths: in, These are dynamic weight parameters used to adjust the suppression strength of the second path; Step 44: Use the contrastive attention matrix to perform a weighted summation on the value vector V to obtain the enhanced feature representation: Step 45: Perform the feature representations output from steps 41-44 on each attention head. After splicing, linear transformation and grouping normalization are performed to obtain the final enhanced features.

6. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, The specific processing procedure for the spatial information mining branch in step 5 is as follows: (1) Using the AU-Net network to extract features from the input image Extract the original facial action unit feature set ; (2) Set up the original facial action unit features Image features By splicing the images together, we obtain the merged visual features. ; (3) Visual features Image semantic description The input is fed into the contrastive attention module to calculate spatial information. ;in, This represents the computation function for the contrastive attention module. The first one As a value vector, the second As a key vector As a query vector.

7. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, The specific processing procedure for the context relationship mining branch in step 5 is as follows: (1) Update node features through hypergraph neural network propagation: (2) The node features of the final layer obtained after propagation through the hypergraph neural network. Image features By stitching the images together, deep visual features are obtained. ; (3) Deep visual features With AU semantic description The input is fed into the contrastive attention module to generate a contextual relationship representation. The first one As a value vector, the second As a key vector As a query vector.

8. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, The specific processing procedure of the cross-branch feature fusion module in step 5 is as follows: (1) Spatial information output by the spatial information mining branch Contextual relationship representation output from the contextual relationship mining branch Input into the cross-attention module; (2) Calculate spatial information through the cross-attention module. Representation of contextual relationships The correlation between them, and characterization from the contextual relationship Extracting spatial information Complementary information, its output is the final enhanced feature after fusion. : 。 9. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 6 specifically includes: Step 61: In each scenario during the training process, randomly select a source domain from all training sets for sampling; Step 62: Randomly sample N expression categories from the source domain, and randomly sample K samples from each expression category as the support set. Q samples as the query set Construct an N-way K-shot few-shot classification task; where, The sample represents the support set. The set of tags representing the support set. This represents a sample of the query set. This represents the set of actual labels for the query set; Step 63: Process the samples from the support set and query set separately using the similarity feature extractor. Forward propagation computes image features, which are then processed by the metric module. Calculate the sample distance between the support set and the query set, and assign samples from the query set to the nearest neighbor support set; its mathematical representation is as follows: in, This represents the prediction result for a sample of the query set. M represents the classification function, and M represents the metric module used to calculate the distance or similarity between two feature vectors. Step 64: During the training process, calculate the prediction result for a single sample in the query set. With real labels Cross-entropy loss between Used to update the parameters of the two-branch collaborative learning network: in, To query the prediction results for a single sample in the query set, This represents the true label of a single sample in the query set, and n' represents the index variable for iterating over the total number of expression categories N', where N' represents the total number of expression categories in the current task. This indicates an indicator function, when n' equals When n' is not equal to 1, the value of the indicator function is 1; when n' is not equal to 1, the value of the indicator function is 1. When the value is 0, the indicator function is 0.

10. The method for cross-domain few-sample facial expression recognition based on action unit consistency as described in claim 1, characterized in that, Step 7 specifically includes: Step 71: During the testing phase, randomly sample N' expression categories from the composite expression dataset. In each expression category, sample K samples as the support set and Q samples as the query set. Step 72: Use the trained dual-branch collaborative learning network to calculate the class prototypes for the samples in the support set, and perform classification predictions for all N'*Q samples in the query set based on the class prototypes. Calculate the classification accuracy of all samples in the query set as the performance evaluation index of the current task. Step 73: Repeat steps 71-72 to complete multiple sampling tasks, and take the average classification accuracy of all tasks as the final performance evaluation index of the dual-branch collaborative learning network.

Images