Cross-domain facial expression recognition method, system, device and storage medium

Abstract

The present invention discloses a cross-domain facial expression recognition method, system, device, and storage medium. The method comprises: constructing a model including a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistency prediction module, and an independent classification learning module. The feature extractor extracts multiple feature vectors including global features and local features from an image; the independent adversarial learning module distinguishes whether the image is from a source domain or a target domain based on the multiple extracted feature vectors; the independent pseudo-label generation module generates pseudo-labels based on the feature vectors extracted from the target domain image, and then obtains the pseudo-labels of the target domain image through the global-local consistency prediction module; the independent classification learning module trains the model based on the source domain dataset, the target domain dataset, and the corresponding pseudo-labels; the image to be predicted is input into the trained model, and the pseudo-labels are output. The present invention utilizes the constructed model to significantly improve the accuracy of facial expression recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain facial expression recognition method, system, computer device, and computer-readable storage medium. Background Art

[0002] Domain drift is a challenging problem in cross-domain facial expression recognition (CD-FER). Because the distribution changes (e.g., marginal distribution and class distribution) between the source and target domains cause the model's performance to degrade on the target domain, most current methods focus on learning domain-invariant features through global feature adaptation, but ignore the potential advantages of local features (e.g., mouth corners) that are easier to transfer between different domains.

[0003] Compared to traditional facial expression recognition tasks within the same dataset, cross-domain facial expression recognition is more challenging. While expressions can be accurately interpreted in most cases, people's experiences and cultures vary, leading to varying interpretations of facial expressions. This inherent subjectivity in understanding expressions can lead to significant domain shift between different expression data. Furthermore, facial expression data in different datasets often involves different people and captures different environments, further exacerbating the domain shift problem. Summary of the Invention

[0004] In order to address the above-mentioned deficiencies in the prior art, the present invention provides a cross-domain facial expression recognition method, system, computer device and computer-readable storage medium.

[0005] The first object of the present invention is to provide a cross-domain facial expression recognition method.

[0006] A second object of the present invention is to provide a cross-domain facial expression recognition system.

[0007] A third object of the present invention is to provide a computer device.

[0008] A fourth object of the present invention is to provide a computer-readable storage medium.

[0009] The first object of the present invention can be achieved by adopting the following technical solutions:

[0010] A cross-domain facial expression recognition method, the method comprising:

[0011] Obtain a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; the target domain dataset includes multiple target domain images;

[0012] The constructed adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module, wherein the feature extractor is used to extract multiple feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from the source domain or the target domain based on the multiple feature vectors extracted from the source domain image and the multiple feature vectors extracted from the target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on the multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label;

[0013] The image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and the corresponding pseudo label is output through the global-local consistency prediction module.

[0014] Furthermore, the number of the eigenvectors is 7;

[0015] The independent pseudo-label generation module is used to generate corresponding pseudo-labels based on multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold, specifically including:

[0016] Take any eigenvector f i t Input the independent pseudo-label generation module and output the corresponding pseudo-label, including:

[0017] The eigenvector f i t Input the corresponding classifier G in the independent pseudo-label generation module i , we get the prediction scores for 7 categories:

[0018]

[0019] in, is the classifier G i Based on the feature vector f i t Predicting target domain images is the score of category j, j = 0, 1…, 6, i = 0, 1…, 6;

[0020] By s i Get the predicted class p with the highest probability:

[0021] p=argmax(s i )

[0022] Using classifier G i The number of pseudo labels learned is used to calculate the threshold t of 7 categories i for:

[0023]

[0024] in, Represents the classifier G i The threshold that needs to be reached in order to generate a pseudo label for category j, Calculated by the following equation:

[0025]

[0026] Among them, θ is a fixed preset threshold, It's about The nonlinear mapping function, Represents the classifier G i To generate the ratio of pseudo labels of class j to all pseudo labels, it is calculated as follows:

[0027]

[0028] in, is the classifier G i The number of pseudo labels generated, c is the total number of categories, and is the number of pseudo labels generated by the category with the largest number of pseudo labels; The value range is 0-1, and by rewriting the denominator in the equation, right Sensitivity:

[0029]

[0030] Reliable labels are filtered by setting a dynamic threshold. The formula is as follows:

[0031]

[0032] Among them, -1 indicates that the pseudo label generation fails; class p is the feature vector f i t The corresponding pseudo labels;

[0033] The 7 feature vectors are input into the independent pseudo-label generation module respectively to obtain the corresponding pseudo-labels.

[0034] Furthermore, the method of jointly predicting the plurality of retained pseudo labels to obtain the pseudo label of the target domain image includes:

[0035] Each feature vector corresponds to 7 prediction scores and 7 thresholds Then the 7 eigenvectors get the prediction score matrix s and threshold matrix t as follows:

[0036]

[0037] By s 6 The class p′ with the highest prediction score:

[0038] p′=argmax(s 6 )

[0039] like Greater than Then take class p′ as the final predicted class;

[0040] otherwise:

[0041] Will s 0 The category with the largest prediction score is taken as the new category p′;

[0042] like Greater than Then class p′ is taken as the final predicted class. Otherwise, the prediction is made by combining the five local prediction scores and thresholds, including:

[0043] The mask matrix m is calculated according to the following formula:

[0044]

[0045] if Greater than but The value is 1, otherwise it is 0;

[0046] Multiply the points of s and m to get the matrix

[0047]

[0048] Will Sum the prediction scores in to get the final prediction score

[0049]

[0050] Will The category with the largest prediction score is taken as the new category p′, and category p′ is taken as the final prediction category;

[0051] The final predicted class is the pseudo label of the target domain image.

[0052] Furthermore, the first six eigenvectors of the seven eigenvectors are the global eigenvector and five local eigenvectors obtained by clipping the global eigenvector, and the first six eigenvectors are sequentially connected to obtain the seventh eigenvector;

[0053] When i=0,1…,6, f i t Represents 7 eigenvectors in turn, f i t Corresponding score s i and threshold t i .

[0054] Furthermore, the global feature vector is obtained by extracting the entire face of the image;

[0055] The global feature vector is cropped to obtain five local regions centered on the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth as five local feature vectors.

[0056] Furthermore, the number of the eigenvectors is 7;

[0057] The independent adversarial learning module includes 7 domain discriminators, corresponding to 7 facial feature vectors respectively;

[0058] The learning process of the independent adversarial learning module is expressed as follows:

[0059]

[0060]

[0061] Among them, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t};D i is a member of D, D is 7 domain discriminators, Is expected as D i Loss; log[D i (f i )] is the binary classification cross entropy function, which is used to compare the probability distribution of the source domain with the probability distribution of the target domain; i = 0, 1, 2, ..., 6; D s 、D t represent the source domain dataset and target domain dataset respectively.

[0062] Furthermore, the number of the eigenvectors is 7;

[0063] The independent classification learning module includes 7 classifiers G i , corresponding to 7 facial feature vectors respectively;

[0064] For the source domain image There is an accurate label from The 7 feature vectors f extracted from i s All corresponding labels For existing pseudo labels The target domain image The pseudo label Assign to image The 7 extracted eigenvectors f i t After the assignment is completed, classification learning is performed according to the following formula:

[0065]

[0066]

[0067] Among them, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t};G i Represents any one of the seven classifiers; represents the calculation operation of multi-class cross entropy, and defines the expected entropy as the classification loss; i = 0, 1, 2, ..., 6; D s 、D t represent the source domain dataset and target domain dataset respectively.

[0068] The second object of the present invention can be achieved by adopting the following technical solutions:

[0069] A cross-domain facial expression recognition system, comprising:

[0070] An acquisition module is used to acquire a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; and the target domain dataset includes multiple target domain images;

[0071] A construction and training module for constructing an adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module. The feature extractor is used to extract multiple feature vectors including global features and local features from an image; the independent adversarial learning module is used to distinguish whether an image is from a source domain or a target domain based on multiple feature vectors extracted from a source domain image and multiple feature vectors extracted from a target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on multiple feature vectors extracted from a target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label;

[0072] The recognition module is used to input the image to be predicted into the trained adaptive global-local representation learning and unified prediction network model, and output the corresponding pseudo label through the global-local consistency prediction module.

[0073] The third object of the present invention can be achieved by adopting the following technical solutions:

[0074] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned cross-domain facial expression recognition method is implemented.

[0075] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0076] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned cross-domain facial expression recognition method.

[0077] The present invention has the following beneficial effects compared to the prior art:

[0078] The cross-domain facial expression recognition method, system, computer device and storage medium proposed in the present invention obtain a source domain data set and a target domain data set, wherein the source domain data set includes multiple source domain images, each source domain image has a label; the target domain data set includes multiple target domain images; the constructed adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module, wherein the feature extractor is used to extract multiple feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from a source domain image based on the multiple feature vectors extracted from the source domain image and the multiple feature vectors extracted from the target domain image. From the source domain or the target domain; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistency prediction module is used to adopt a consistency prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label; the image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and the corresponding pseudo-label is output through the global-local consistency prediction module. By constructing an adaptive global-local representation learning and unified prediction model, the adverse effects of data inconsistency on performance in the target domain are addressed: First, the feature extractor is used to extract global and local features of the input image, which can effectively learn domain-invariant features while maintaining potential semantic consistency to reduce the impact of cross-domain distribution differences; Second, in the pseudo-target label generation process, the threshold learning strategy is dynamically adjusted in the independent pseudo-label generation module to generate richer and fine-grained pseudo labels, thereby alleviating the category imbalance problem of unlabeled images; Finally, by adopting a new unified prediction strategy in the global-local consistent prediction module, the prediction scores of the global feature vector and the local feature vector are combined to obtain the final label of the target domain image, alleviating the domain shift between cross-domains and achieving competitive performance. In addition, the method provided by the present invention is subjected to detailed experiments and ablation analysis using more datasets and backbone networks, which proves the effectiveness of the method and can significantly improve the accuracy of facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0080] Figure 1 This is a flowchart of the cross-domain facial expression recognition method according to embodiment 1 of the present invention.

[0081] Figure 2 This is a structural diagram of the AGLRLS network model of Example 1 of the present invention.

[0082] Figure 3 This is the Friedman test chart of Example 2 of the present invention.

[0083] Figure 4 This is a bar chart showing the distribution of the number of categories in the data set of Example 2 of the present invention.

[0084] Figure 5 A bar chart showing the performance of the three algorithms in Example 2 of the present invention using the class imbalance metric.

[0085] Figure 6 This is a line chart of the average accuracy after each module of Example 2 of the present invention.

[0086] Figure 7 1 is the accuracy curve of the 7 classifiers in Example 2 of the present invention during the training process.

[0087] Figure 8 It is the average accuracy of different prediction strategies of Example 2 of the present invention on 5 target domain data sets.

[0088] Figure 9 This is a structural block diagram of a cross-domain facial expression recognition system according to embodiment 3 of the present invention.

[0089] Figure 10 This is a structural block diagram of a computer device according to embodiment 4 of the present invention. DETAILED DESCRIPTION

[0090] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain this application and are not used to limit this application.

[0091] Example 1:

[0092] like Figure 1 As shown, this embodiment provides a cross-domain facial expression recognition method, including the following steps:

[0093] S101: Obtain a source domain dataset and a target domain dataset.

[0094] The source domain dataset is the public RAF-DB dataset or FER2013 dataset, and the target domain dataset is the public CK+ dataset, JAFFE dataset or ExpW dataset.

[0095] Source domain dataset D s Each sample in There is a label The target domain dataset D t Samples in There are no tags.

[0096] With labels and the number of samples is N s The source domain dataset Label C represents the number of categories; without labels and the number of samples is N t The target dataset

[0097] S102: Construct an adaptive global-local representation learning and unified prediction network model, and use the adaptive global-local representation learning and unified prediction network model to output pseudo labels corresponding to images in the target domain dataset.

[0098] like Figure 2 As shown in the figure, the adaptive global-local representation learning and unified prediction (AGLRLS) network model includes a feature extractor, an independent adversarial learning module (SAL), an independent pseudo-label generation module (IPLG), an independent classification learning module (SCL) and a global-local consistent prediction (GLPC) module.

[0099] Specifically, 7 feature vectors including global and local features are first extracted from the source domain image through the feature extractor, and then they are input into the SAL module with discriminator and the SCL module with classifier respectively; when the target domain dataset has no labels, the IPLG module is first used to classify the target domain samples according to the target domain samples. The corresponding pseudo labels are generated from the 7 feature vectors extracted, and the target domain data with pseudo labels are also sent to the SCL module for classification learning. After sufficient training, the GLPC module will be used to infer and predict the target domain data.

[0100] (1) Feature extractor.

[0101] The local area of the face plays an important role in facial expression recognition, and the local area of the face is extracted by the feature extractor F.

[0102] The image is input into the feature extractor to obtain 7 facial feature vectors including global and local features. Specifically, the whole face is extracted from the input image I, and five local regions centered on the left eye, right eye, nose, left corner of the mouth and right corner of the mouth are further cropped to obtain a global feature vector f g and five local eigenvectors f le , f re , f ne , f lm , f rm , then connect these six vectors in sequence to get the seventh eigenvector f gl :

[0103]

[0104] f={f g ,f le ,f re ,f ne f lm ,f rm ,f gl} (1)

[0105] Where f represents the 7 feature vectors extracted from image I.

[0106] In one embodiment, the feature extractor primarily uses a ResNet50 variant and MobileNet-v2 as the backbone networks to extract features. Both networks consist of four block layers. Given a 112×112 input image, the input image is processed sequentially through the four block layers, resulting in a feature map m1 of 28×28×128 from the second layer and a feature map m2 of 7×7×512 from the fourth layer.

[0107] For global features, m2 is first converted to a size of 7×7×64 through a convolution operation, and then a 64-dimensional feature vector is obtained through an average pooling layer. For local features, MT-CNN is first used to crop five regions of size 7×7×128 at the center of m1 based on the corresponding feature points. Similar convolution and average pooling operations are used as for global features to obtain five 64-dimensional local vectors. The global feature vector and the five local feature vectors are concatenated to obtain a 384-dimensional feature vector as the sixth feature vector; then, the six vectors are concatenated in sequence to obtain the seventh vector; finally, a set f is constructed based on the seven feature vectors. In addition, seven classifiers and seven discriminators are constructed using fully connected layers. The seven feature vectors in set f are then input into the corresponding classifiers and discriminators for learning.

[0108] (2) Independent Adversarial Learning Module (SAL).

[0109] The independent adversarial learning module SAL includes 7 domain discriminators, corresponding to 7 facial feature vectors, which are used to distinguish whether an image is from the source domain or the target domain, and promotes domain confusion through an adversarial objective function to minimize the distance between the source domain and target domain mapping distributions.

[0110] To perform adversarial learning, we use indices from 0 to 6 to access the input vector f. The learning process of the independent adversarial learning module can be expressed as follows:

[0111]

[0112]

[0113] Among them, f i s is extracted from the source domain image by the feature extractor Extract the generated 7 eigenvectors f s One, f i t is extracted from the target domain image by the feature extractor Extract the generated 7 eigenvectors f t One, f i ={f i s ,f i t};D i is a member of D, D is 7 domain discriminators, Is expected as D i Loss; log[D i (f i)] is a binary classification cross entropy function, which is used to compare the probability distribution of the source domain predicted by the model and the probability distribution of the target domain and measure the difference between them; the expectation is determined based on the cross entropy and the features. Since the input features contain global features and local features, global and local invariant features can be learned in this way, thereby reducing domain bias to a greater extent.

[0114] i is the number of the 7 eigenvectors, i = 0, 1, 2, ..., 6, corresponding to f g 、f le , f re , f ne , f lm , f rm and f gl .

[0115] (3) Independent pseudo-label generation module (IPLG).

[0116] The independent pseudo-label generation module filters out unreliable pseudo-labels by setting an adaptive threshold.

[0117] In CD-FER, for unlabeled target domain data, the IPLG module is introduced to independently generate pseudo-labels corresponding to the invariant features of the learning domain. Extensive experiments have demonstrated that labels have a significant impact on classification tasks. However, in CD-FER, the target domain data is unlabeled. Therefore, to fully utilize this unlabeled data and the global-local information of facial expressions, the specific process for independently generating pseudo-labels for the global-local features extracted from the same image in the target domain is as follows:

[0118] (3-1) The target domain image is input into the feature extractor, and a pseudo label is predicted based on one of the seven feature vectors generated by the extraction; unreliable pseudo labels are filtered out by setting an adaptive threshold.

[0119] First, the target domain image Perform weak enhancement and input into the feature extractor to obtain the feature vector f t ;

[0120] Then, f t Each vector f in i t Input the corresponding classifier G in the independent pseudo-label generation module i In the above example, we get the prediction scores s for 7 categories. i :

[0121]

[0122] in, is the classifier G i Based on the feature f i tPredicting target domain images is the score of category j, j = 0, 1…, 6;

[0123] By s i You can get the predicted class p with the highest probability:

[0124] p=argmax(s i ) (5)

[0125] Then, using the classifier G i The number of pseudo labels learned is used to calculate the threshold t of 7 categories i for:

[0126]

[0127] in, Represents the classifier G i The threshold that needs to be reached in order to generate a pseudo label for category j, It can be calculated by the following equation:

[0128]

[0129] Among them, θ is a fixed preset threshold, It is a The nonlinear mapping function, Represents the classifier G i To generate the ratio of pseudo labels of class j to all pseudo labels, it is calculated as follows:

[0130]

[0131] in, is the classifier G i The number of pseudo labels generated, c is the total number of categories, and is the number of pseudo labels generated by the category with the largest number of pseudo labels. It is foreseeable that The value range is 0-1. The closer it is to 1, the more pseudo labels are learned for this category, and the better the learning effect is. Rewrite the denominator in the equation to ensure right Sensitivity:

[0132]

[0133] This improvement is more suitable for the model in this embodiment;

[0134] Finally, we set a dynamic threshold to filter reliable labels, and compare them using the following formula: and get:

[0135]

[0136] Among them, -1 means that the pseudo label generation fails.

[0137] (3-2) Based on the generated 7 feature vectors, obtain the corresponding pseudo labels.

[0138] There are a total of 7 classifiers in the independent pseudo-label generation module. Repeat the above step (3-1) 7 times, and 7 pseudo labels can be obtained based on the 7 extracted feature vectors.

[0139]

[0140] (4) Utilize the global-local consistency prediction module (GLPC).

[0141] The global-local consistency prediction module (GLPC) is used to predict the category of the target domain image.

[0142] By selecting an appropriate consistency prediction strategy, the 7 pseudo labels generated by the 7 feature vectors extracted from the target domain image are jointly predicted to obtain the final pseudo label of the target domain image.

[0143] For each target domain image Each classifier corresponding to each feature vector can get a set of prediction scores s i , during the training process, the threshold t for each category is calculated i To filter the available pseudo labels, since there are 7 classifiers, we can get a 7x7 prediction score s and a 7x7 threshold matrix t, as follows:

[0144]

[0145] in, Represents the classifier G i According to the input feature vector f i t To predict the labeled image is the score of category j, Represents the classifier G i The threshold that needs to be reached to generate pseudo labels for category j.

[0146] Because the target domain data used to generate pseudo labels is the training set, and the target domain data of the test set is used in the testing phase, the two datasets naturally have similar data distributions. Therefore, the pseudo label generation process can be transferred to the prediction process of the classifier consistency test phase, which includes:

[0147] First, taking the global-local features as input, determine the s obtained by the classifier G66 The class p with the highest prediction score is:

[0148] p=argmax(s 6 ) (13)

[0149] if Greater than Then directly take p as the final predicted class; if it is less than, input the global feature into the classifier G0 to get s 0 The maximum category of the predicted score is used as the new p, and then compared and Value: If Greater than Then use p as the final predicted class; if Still less than The prediction is made by combining five local prediction scores and thresholds, including:

[0150] First, the mask matrix m is calculated according to the following formula:

[0151]

[0152] If m is a 7x7 matrix, Greater than but The value is 1, otherwise it is 0;

[0153] Next, we get the matrix by multiplying s and m.

[0154]

[0155] Then, The prediction scores of different classifiers for each class in are summed to obtain the final prediction scores of the 7 classes.

[0156]

[0157] Finally, the predicted label p is The class with the highest prediction score:

[0158]

[0159] Through the above process, pseudo labels corresponding to the target domain images are generated.

[0160] (5) Independent classification learning module (SCL).

[0161] The independent classification learning module includes 7 classifiers G i, corresponding to 7 facial feature vectors, which are mainly used to learn domain-invariant features of global features and local features.

[0162] For the source domain image There is an accurate label from The 7 eigenvectors f extracted from i s All corresponding labels For existing pseudo labels The target domain image The pseudo label Assign to image The 7 extracted eigenvectors f i t After the assignment is completed, classification learning is performed according to the following formula:

[0163]

[0164]

[0165] Among them, G i is a classifier for classifying input images into 7 basic expressions. Represents the calculation operation of multi-class cross entropy, defines the expectation of entropy as the classification loss, and uses the classification loss to train 7 classifiers.

[0166] S103, using the source domain dataset and the images in the target domain dataset and the corresponding pseudo labels to train an adaptive global-local representation learning and unified prediction network model.

[0167] The feature extractor, classifier, and domain discriminator are trained and optimized based on the loss functions of the independent adversarial learning module and the independent classification learning module (i.e., formulas (2) and (19) in step S102). A two-stage training process is used. The backbone network is initialized using a model trained on the MS-Celeb-1M dataset, while the parameters of the newly added layers are initialized using the Xavier algorithm. SGD is used as the optimizer during training.

[0168] In the first stage, the feature extractor and classifier are trained using the cross entropy loss function, with the learning rate, momentum, and weight decay set to 0.0001, 0.9, and 0.0005, respectively, and training is performed for approximately 15 epochs.

[0169] In the second stage, the domain discriminator is trained by the target loss in formula (2), and the feature extractor and classifier are fine-tuned by the target loss in formula (18), with the same momentum and weight decay as in the first stage. The learning rate of the feature extractor and source classifier is initialized to 0.00001 and divided by 10 after 20 epochs. Since the domain discriminator is trained from scratch, it is initialized to 0.0001 and divided by 10 when the error is saturated. In addition, in the pseudo-label generation module, two points need to be noted: first, although the features used to generate pseudo-labels and the features used to calculate cross-entropy come from the same target domain image, the data augmentation methods used for these two sets of features are different: the features used to generate pseudo-labels are weakly augmented, while strong augmentation such as diffraction is added when calculating cross-entropy; second, after the initial score of the label is predicted by the classifier, it must be processed by the softmax function and compared with the threshold.

[0170] This embodiment utilizes a consistency regularization training method to improve the stability and robustness of the model, thereby improving the classification performance of the model on the target domain dataset.

[0171] S104: Input the image to be predicted into the trained adaptive global-local representation learning and unified prediction network model, and output the corresponding pseudo label through the global-local consistency prediction module.

[0172] The image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and 7 feature vectors including global and local features are extracted. The independent pseudo-label generation module (IPLG) screens the 7 feature vectors to obtain the pseudo labels corresponding to the 7 feature vectors. Finally, the global-local consistent prediction (GLPC) module predicts the final pseudo label of the sample (image to be predicted) based on the 7 pseudo labels.

[0173] Example 2:

[0174] This example tests the method provided by the present invention, and the specific process is as follows:

[0175] (1) Compare the method provided by the present invention with the existing methods.

[0176] In the cross-domain facial expression recognition task, different literatures use different network backbones, source domain datasets, and target domain datasets, so fair comparison cannot be made. Fortunately, the benchmarks proposed in previous studies summarize the results of some leading methods, ensuring that the same source domain dataset, target domain dataset, and backbone network are used to make fair comparisons with these methods.

[0177] We selected RAF-DB and FER2013 as the source datasets, and used ResNet-50 and MobileNet-v2 as the backbone networks for four sets of experiments. Table 1 shows the accuracy of the proposed method compared to existing leading methods on CK+, JAFFE, SFEW2.0, FER2013, RAF-DB, and ExpW, using different datasets and networks.

[0178] Table 1 Comparison of the accuracy of prediction results between the method provided by the present invention and the existing method

[0179]

[0180]

[0181] Experimenting with different backbone networks effectively explores the differences in the feature extractor's performance in extracting discriminative features. To evaluate the robustness of the algorithm under different backbone networks, this example uses two widely used backbone networks, ResNet50 and MobileNet-v2. The results show that when using the same source domain and replacing the backbone network from ResNet50 to MobileNet-v2, the algorithm performance on almost all datasets degrades to varying degrees, demonstrating a certain degree of universality. Furthermore, using RAF-DB as the source domain and replacing ResNet50 with MobileNet-v2, the performance of leading algorithms such as AGRA, using CK+, JAFFE, SFEW2.0, FER2013, and ExpW as examples, degrades by 12.4%, 6.1%, 10.79%, 7.9%, and 4.56%, respectively. Previous work in this example, CGLRL, also degrades by 13.18%, 7.04%, 7.11%, 6.84%, and 5.15%, respectively. These results demonstrate that selecting an appropriate backbone network is crucial for ensuring algorithm performance, and also serve as a reminder that careful selection of the backbone network is crucial in practical applications to achieve optimal results. This may be due to the fact that MobileNet-v2 utilizes depthwise separable convolutions in its architecture, a design that results in relatively weaker feature extraction capabilities compared to ResNet50. Although the proposed AGLRLS algorithm also experiences a performance degradation, its performance decreases minimally compared to AGRA and CGLRL, at 4.65%, 5.61%, 8.03%, 6.36%, and 3.90%, respectively. Even when using the less powerful MobileNet-v2, the proposed method still outperforms the best algorithm on these five datasets, achieving improvements of 9.31%, 1.41%, 0.46%, 1.86%, and 4.23%, respectively. For a fair comparison, the accuracy of each target domain is averaged to obtain a metric called "Mean." On this metric, the proposed AGLRLS algorithm (a cross-domain facial expression recognition method) still performs best, outperforming the other algorithms.

[0182] Using Different Source Domains: Another important factor that can affect target domain accuracy is the similarity between the source and target domains. As shown in Table 1, when using the same backbone network but replacing the source domain from RAF-DB to FER2013, the performance of almost all algorithms on all datasets degrades to varying degrees. Next, we further analyze the data in subtables i and iii of Table 1, using the same backbone network, ResNet50, but changing the source domain from RAF-DB to FER2013. AGRA's accuracy on the CK+ dataset improves by 0.42%, but decreases by 8.76%, 7.12%, and 8.27% on JAFFE, SFEW2.0, and ExpW, respectively. CGLRL decreases by 3.11%, 6.1%, 4.59%, and 8.08% on CK+, JAFFE, SFEW2.0, and ExpW, respectively. A deeper analysis of the differences between FER2013 and RAF-DB reveals several reasons for this phenomenon. First, while the RAF-DB and FER2013 datasets are similar in size and both come from the web, the FER2013 data is grayscale, while the RAF-DB data is color. The target migration datasets, SFEW2.0 and ExpW, are both color images. From this perspective, the similarity between FER2013 and SE (SFEW2.0, ExpW) is smaller than the similarity between RAF-DB and SE, which explains why the performance impact on SE is significantly reduced when the source domain is replaced from RAF-DB to FER2013. At the same time, since CK+ is also grayscale, its similarity to FER2013 is higher than that between CK+ and RAF-DB, which also explains why the performance of some algorithms may slightly improve after replacing the source domain. A second difference between RAF-DB and FER2013 is that the collected data covers different populations. RAF-DB's data samples include Asians, while FER2013 does not. This results in the similarity between FER2013 and the Asian female dataset JAFFE being much lower than the similarity between RAF-DB and JAFFE. This likely explains the performance drop on the JAFFE dataset. Overall, replacing RAF-DB with FER2013 as the source domain significantly reduces the similarity between the source and target domains, impacting performance. Although the proposed algorithm also experiences some performance degradation when using FER2013 as the source domain, even with this relatively low similarity, the proposed algorithm still outperforms the state-of-the-art algorithms by 4.23%, 1.41%, 0.23%, and 0.69% on the CK+, JAFFE, SFEW2.0, and ExpW datasets, respectively. For a more fair comparison, the accuracy for each target domain is averaged, referred to as the mean. The AGLRLS proposed in this paper still performs best on this metric, outperforming other algorithms.

[0183] Use statistical tests: In order to more intuitively compare the superiority of the algorithms, two classic statistical methods, Friedman test and Nemeny test, are used to prove the superiority of the algorithms. However, in each configuration of the backbone network and source domain, each algorithm has only 5 accuracy data from 5 target domains. The small amount of data will lead to a lack of statistical significance. Therefore, the accuracy and average value of each algorithm under different backbone network and source domain configurations are collected in turn, so that each algorithm can have 24 accuracy data. 10 algorithms with reliable performance were selected from Table I for comparative testing. Then, the average rank of each algorithm was calculated, and the Nemeny test was used for post-test. In order to visualize the results, the critical difference is calculated using the following formula:

[0184]

[0185] Here, α is the confidence level, which is an artificially controlled statistical parameter with values of 0.05 and 0.10. n is the number of samples for each algorithm. k represents the number of algorithms. The Friedman test plot with the average rank and CD of the algorithms is shown in the figure below. Figure 3 As shown in Figure 2, it can be seen that the average rank value of the algorithm provided by the present invention is the highest (in descending order), and the method has no overlap with other methods except AGRA. This means that AGLRLS is significantly different from other algorithms.

[0186] The Friedman test chart shows the case with α = 0.05 on the left and α = 0.10 on the right. The horizontal coordinate corresponding to the midpoint of each algorithm represents the mean order value. Lower values indicate better performance. The range of the horizontal lines on either side of each midpoint represents the CD value. If the horizontal lines between two algorithms do not overlap, this indicates that the performance of the two algorithms is significantly different.

[0187] There is some overlap with AGRA, which can be explained by the fact that 24 accuracy data points are still insufficient to distinguish AGRA from AGLRLS. Across 24 accuracy data points, AGRA is almost second, with a ranking difference of only 1 from AGLRLS. However, the calculated CD values are 2.77 and 2.55, respectively. Therefore, while the results are better than AGRA, there is still some overlap due to insufficient data.

[0188] Use class imbalance metrics such as Figure 4As shown in the figure, there is an imbalance in the sample data of categories such as CK+, JAFFE, RAF-DB, FER2013, SFEW2.0 and ExpW. In order to conduct a more comprehensive experimental analysis to verify the effectiveness of the proposed model, recall, precision and F1 score indicators are used for measurement. However, considering the need to evaluate the effects of the five target domains under each configuration of the backbone network and the source domain, the average recall rate, average precision and average F1 score are used for a more intuitive comparison. The AGRA and CGLRL algorithms, two algorithms with excellent performance, are also reproduced for comparative analysis. Figure 5 As shown in Figure 3, under four configurations of backbone network and source domain, the model provided by the present invention outperforms AGRA and CGLRL in almost all indicators.

[0189] (2) Analyze the contribution of each module in the adaptive global-local representation learning and unified prediction network models.

[0190] This example analyzes the contribution of each module through a series of experiments, focusing on a more detailed analysis of the independent pseudo-label generation module and the global-local prediction consistency module.

[0191] AGLRLS mainly consists of three innovative modules: Independent Adversarial Learning (SAL), Independent Pseudo-Label Generation (IPLG), and Global-Local Prediction Consistency (GLPC). By analyzing the contribution of each module in performance, using RAF-DB as the source domain and ResNet50 as the backbone network, the contribution of each module to AGLRLS on five target domains: CK+, JAFFE, SFEW2.0, FER2013, and ExpW is verified.

[0192] First, we established a baseline that uses only labeled source domain data for independent classification learning. The baseline row in Table II shows the results. Next, we added SAL to the baseline to learn domain-invariant features. The SAL row in Table II shows the effect. After introducing SAL, the accuracy of CK+, JAFFE, SFEW2.0, FER2013, and ExpW increased by 1.55%, 0.47%, 0.23%, 1.04%, and 1.29%, respectively. This demonstrates that using domain-adaptive adversarial learning to align the feature spaces of the source and target domains and obtain domain-invariant features has a significant impact on CD-FER. Then, we introduced IPLG based on the addition of SAL and used the generated pseudo-labels to perform target-domain-independent classification learning on the target domain data. The IPLG row in Table 2 shows the results. As can be seen, the addition of SAL significantly improves performance, achieving gains of 11.63%, 1.41%, 4.82%, 2.59%, and 2.74% on the CK+, JAFFE, SFEW2.0, FER2013, and ExpW datasets, respectively. This fully demonstrates the importance of labels for classification tasks. The introduction of pseudo-labels fully utilizes the target domain data. It is worth noting that in the three experiments described above, classifier predictions based on global and local connectivity features were used as input for the final prediction results. However, after full training, all seven classifiers are capable of classification, each corresponding to both global and local face information. Therefore, combining these seven classifiers for prediction fully exploits both global and local face information. Based on this principle, the GLPC module is introduced in addition to IPLG and SAL. The GLPC row in Table II demonstrates the improved performance, achieving gains of 0.78%, 0.91%, 0.92%, 0.17%, and 0.04% on the five target datasets, respectively.

[0193] Table 2 Accuracy of the five target domains after introducing each module

[0194]

[0195]

[0196] In summary, after adding the three modules SAL, IPLG, and GLPC to the baseline in sequence, its performance on the five target domain datasets has gradually improved. In order to more intuitively see the improvement effect of each module on AGLRLS, we take the average of the effects of the five target domains and then draw a line chart of the performance improvement after adding each module, as shown in the figure below. Figure 6 It can be clearly seen that each module has a positive impact on the performance of AGLRLS.

[0197] Comparison of different prediction strategies: After sufficient training, all 7 classifiers have certain classification capabilities, so it is very important to find a suitable prediction strategy. In this section, several prediction methods will be compared, and the reasons why the proposed global-local prediction consistency method performs well will be analyzed. Similar to the experimental configuration in the above ablation study, the following experiments are also tested on CK+, JAFFE, SFEW2.0, FER2013 and ExpW datasets, with RAF-DB as the source domain and ResNet50 as the backbone network. It should be noted that in order to ensure the fairness of the experiment, the same trained model is used on the same target domain, and then several different strategies are used for prediction. For the convenience of explanation, the classifier with global features as input is called G g , the classifier that takes local features as input is called G l , the classifier that takes the global and local features as input is called G gl .

[0198] First, the first prediction strategy is adopted, which is to use the global feature classifier G g The results of this strategy are presented in the "Global" row of Table 3. This method only considers global features, not local features. Therefore, a second prediction method is introduced, which uses the global and local features to concatenate and input them into the classifier G. gl The prediction score is used as the final prediction result. The results of this strategy are shown in the "G-Local" row in Table 3. It can be clearly seen that the accuracy is improved by 1.55%, 0.47%, 0.23%, 1.04%, and 1.29% on the CK+, JAFFE, SFEW2.0, FER2013, and ExpW datasets, respectively. This demonstrates that local features can directly improve classification performance. However, the utilization of local features is not limited to feature fusion; classification learning based on independent local features is also performed. Figure 6 The accuracy of the seven classifiers during training on the FER2013 dataset is shown. It can be observed that the five local feature classifiers G lThe prediction accuracy of these five classifiers is gradually improving, and the prediction ability of these five classifiers is relatively high. Therefore, it is considered to fuse the prediction scores of these five classifiers to achieve higher prediction accuracy. Two strategies for fusing local classifiers are proposed: the averaging strategy and the voting strategy. The averaging strategy adds and averages the prediction scores of the seven classifiers, and then uses the average prediction score as the final prediction result. In the voting strategy, each classifier votes based on the class with the highest probability, ultimately selecting the class with the most occurrences as the final prediction result. The "Average" and "Votation" columns in Table 3 show the results of these two strategies. It can be observed that the averaging and voting strategy performs worse than G-Local on CK+, JAFFE, SFEW2.0, and FER2013, but performs better on ExpW. This indicates that classifiers that consider local features still have a certain impact on the final prediction result, but the averaging and voting strategy does not account for the differences in predictive ability of each classifier, resulting in a lower overall performance than G-Local. Assigning appropriate weights to the prediction scores of each classifier may yield better classification results.

[0199] Table 3 Prediction accuracy of different prediction strategies for five target domains

[0200]

[0201] The current goal is to find an appropriate metric to use as the weight. The intuitive idea is to use each classifier's performance on the test set as the weight, but this clearly violates the principle that the test set is not used in training. Therefore, we innovatively propose using the pseudo-label generation threshold for each classifier as a weight for the classifier's prediction score. There are two plausible explanations for this approach. First, during training, when the classifier generates pseudo-labels for the target domain training set, the threshold learned by the classifier is based on the classifier's predictive ability. Because each classifier's predictive ability varies, the threshold also varies for each classifier. Second, because the data distribution of the training and test sets is similar, this threshold label can be transferred to the test set. For these two reasons, using each threshold as the weight for each classifier's prediction score is appropriate. Specifically, the classifier's prediction score for each class must exceed the classifier's threshold in order to generate a pseudo-label for that class. The pseudo-labels are then incorporated into the final prediction score. The experimental results are shown in the "Consistency" row in Table 3. Compared with G-Local, the results are improved by 0.78%, 0.91%, 0.92%, 0.17% and 0.04% on CK+, JAFFE, SFEW2.0, FER2013 and ExpW respectively. For a more intuitive comparison, the average accuracy of these five target domain datasets is also calculated, as shown in Figure 2. Figure 7As shown, the consistency strategy is the best compared with the above four prediction strategies.

[0202] Further analysis of the consistency strategy can be divided into two stages. In the first stage, according to the classifier G g and G gl The prediction results are determined in turn by the threshold standards. If no definite result is obtained, the second stage is entered to directly fuse the prediction scores and threshold standards of the 7 classifiers to obtain the final prediction score. Several comparative experiments are conducted to prove that the consistency strategy is the best strategy. The first comparative experiment is to directly skip the first stage and fuse the prediction scores and threshold standards of the 7 classifiers in the second stage. The results are shown in the Con-i row of Table 3. The second comparative experiment is to use only G g In the first phase, the second phase remains unchanged, and the results are shown in the Con-ii row of Table 3. The third comparative experiment is to use only G gl In the first phase, the second phase remains unchanged, and the results are shown in the Con-iii row of Table 3. The fourth comparative experiment is to first use G g , and then use G in the first stage gl , the second stage remains unchanged, and the results are shown in the Con-iv row of Table 3. These five consistency strategies show different excellent performance on different datasets. In order to make an intuitive comparison, the average accuracy is selected as the comparison indicator. Figure 8 As shown in , the consistency strategy has the highest average accuracy. Therefore, it can be considered that the consistency strategy is the best consistency strategy. This good performance is attributed to the concept of using the order of classifiers to further optimize the weights, giving priority to classifiers with stronger predictive power. gl With the best prediction performance, it is first judged using its threshold standard, and then using the classifier G with its own threshold standard g In the second stage, for G gl and G g For difficult samples that cannot be determined, local classifiers are used to make consistent predictions, thereby improving the final prediction accuracy.

[0203] In summary, using the threshold criterion to learn the weights in the training phase as the prediction scores of each classifier effectively fuses the prediction scores of the seven classifiers, and ranking is used to further improve the weights, thereby making more accurate predictions for CD-FER through this strategy.

[0204] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0205] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0206] Example 3:

[0207] like Figure 9 As shown, this embodiment provides a cross-domain facial expression recognition system, which includes an acquisition module 901, a construction and training module 902 and a recognition module 903, wherein:

[0208] An acquisition module 901 is configured to acquire a source domain dataset and a target domain dataset, wherein the source domain dataset includes a plurality of source domain images, each of which has a label; and the target domain dataset includes a plurality of target domain images.

[0209] A construction and training module 902 is used to construct an adaptive global-local representation learning and unified prediction network model, which includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module. The feature extractor is used to extract multiple feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from the source domain or the target domain based on the multiple feature vectors extracted from the source domain image and the multiple feature vectors extracted from the target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on the multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label;

[0210] The recognition module 903 is used to input the image to be predicted into the trained adaptive global-local representation learning and unified prediction network model, and output the corresponding pseudo label through the global-local consistency prediction module.

[0211] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the device provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0212] Example 4:

[0213] This embodiment provides a computer device, which can be a computer, such as Figure 10 As shown, it comprises a processor 1002, a memory, an input device 1003, a display 1004, and a network interface 1005 connected via a system bus 1001. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 1006 and an internal memory 1007. The non-volatile storage medium 1006 stores an operating system, a computer program, and a database. The internal memory 1007 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 1002 executes the computer program stored in the memory, the cross-domain facial expression recognition method of the above-mentioned embodiment 1 is implemented as follows:

[0214] Obtain a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; the target domain dataset includes multiple target domain images;

[0215] The constructed adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module, wherein the feature extractor is used to extract multiple feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from the source domain or the target domain based on the multiple feature vectors extracted from the source domain image and the multiple feature vectors extracted from the target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on the multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label;

[0216] The image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and the corresponding pseudo label is output through the global-local consistency prediction module.

[0217] Example 5:

[0218] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the cross-domain facial expression recognition method of the above-mentioned embodiment 1 is implemented as follows:

[0219] Obtain a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; the target domain dataset includes multiple target domain images;

[0220] The constructed adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module, wherein the feature extractor is used to extract multiple feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from the source domain or the target domain based on the multiple feature vectors extracted from the source domain image and the multiple feature vectors extracted from the target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on the multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label;

[0221] The image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and the corresponding pseudo label is output through the global-local consistency prediction module.

[0222] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0223] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. A cross-domain facial expression recognition method, characterized in that: The method comprises: Obtain a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; the target domain dataset includes multiple target domain images; The constructed adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module. The feature extractor is used to extract 7 feature vectors including global features and local features from the image; the independent adversarial learning module is used to distinguish whether the image is from the source domain or the target domain based on multiple feature vectors extracted from the source domain image and multiple feature vectors extracted from the target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on the multiple feature vectors extracted from the target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label; The image to be predicted is input into the trained adaptive global-local representation learning and unified prediction network model, and the corresponding pseudo label is output through the global-local consistency prediction module; The independent pseudo-label generation module is specifically used to: Take any eigenvector f i t Input the independent pseudo-label generation module and output the corresponding pseudo-label, including: The eigenvector f i t Input the corresponding classifier G in the independent pseudo-label generation module i In the example, we get the prediction scores of 7 categories: Where, is the classifier G i Based on the feature vector f i t Predicting target domain images is the score of category j, j = 0, 1…, 6, i = 0, 1…, 6; By s i Get the predicted class p with the highest probability: p=argmax(s i ) Using classifier G i The number of pseudo labels learned is used to calculate the threshold t of 7 categories i for: Where, Represents the classifier G i The threshold that needs to be reached in order to generate a pseudo label for category j, Calculated by the following equation: Where θ is a fixed preset threshold, It's about The nonlinear mapping function, Represents the classifier G i To generate the ratio of pseudo labels of class j to all pseudo labels, it is calculated as follows: Where, is the classifier G i The number of pseudo labels generated, c is the total number of categories, and max c is the number of pseudo labels generated by the category with the largest number of pseudo labels; The value range is 0-1, and by rewriting the denominator in the equation, right Sensitivity: Reliable labels are filtered by setting a dynamic threshold. The formula is as follows: Where -1 indicates that pseudo label generation fails; class p is the feature vector f i t The corresponding pseudo labels; The seven feature vectors are input into the independent pseudo-label generation module to obtain the corresponding pseudo-labels respectively; The independent adversarial learning module includes 7 domain discriminators, each corresponding to 7 facial feature vectors. The learning process of the independent adversarial learning module is expressed as follows: Where, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t };D i is a member of D, D is 7 domain discriminators, Is expected as D i Loss; log[D i (f i )] is the binary classification cross entropy function, which is used to compare the probability distribution of the source domain with the probability distribution of the target domain; i = 0, 1, 2, ..., 6; D s 、D t Represent the source domain dataset and target domain dataset respectively; The independent classification learning module includes 7 classifiers G i , corresponding to 7 facial feature vectors; for the source domain image There is an accurate label from The 7 eigenvectors f extracted from i s Corresponding labels For existing pseudo labels The target domain image The pseudo label Assign to image The 7 extracted eigenvectors f i t After the assignment is completed, classification learning is performed according to the following formula: Where, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t };G i Represents any one of the seven classifiers; represents the calculation operation of multi-class cross entropy, and defines the expected entropy as the classification loss; i = 0, 1, 2, ..., 6; D s 、D t represent the source domain dataset and target domain dataset respectively.

2. The cross-domain facial expression recognition method according to claim 1, characterized in that The method of jointly predicting the plurality of retained pseudo labels to obtain the pseudo label of the target domain image includes: Each feature vector corresponds to 7 prediction scores and 7 thresholds Then the 7 eigenvectors get the prediction score matrix s and threshold matrix t as follows: By s 6 The class p′ with the highest prediction score: p′=argmax(s 6 ) like Greater than Then take class p′ as the final predicted class; otherwise: Will s 0 The category with the largest prediction score is taken as the new category p′; like Greater than Then class p′ is taken as the final predicted class. Otherwise, the prediction is made by combining the five local prediction scores and thresholds, including: The mask matrix m is calculated according to the following formula: if Greater than but The value is 1, otherwise it is 0; Multiply the points of s and m to get the matrix Will Sum the prediction scores in to get the final prediction score Will The category with the largest prediction score is taken as the new category p′, and category p′ is taken as the final prediction category; The final predicted class is the pseudo label of the target domain image.

3. The cross-domain facial expression recognition method according to claim 2, characterized in that The first six eigenvectors of the seven eigenvectors are the global eigenvector and the five local eigenvectors obtained by clipping the global eigenvector. The first six eigenvectors are sequentially connected to obtain the seventh eigenvector. When i=0,1…,6, f i t Represents 7 eigenvectors in turn, f i t Corresponding score s i and threshold t i .

4. The cross-domain facial expression recognition method according to claim 3, characterized in that The global feature vector is obtained by extracting the entire face of the image; The global feature vector is cropped to obtain five local regions centered on the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth as five local feature vectors.

5. A cross-domain facial expression recognition system, characterized in that: The system comprises: An acquisition module is used to acquire a source domain dataset and a target domain dataset, wherein the source domain dataset includes multiple source domain images, each source domain image has a label; and the target domain dataset includes multiple target domain images; A construction and training module for constructing an adaptive global-local representation learning and unified prediction network model includes a feature extractor, an independent adversarial learning module, an independent pseudo-label generation module, a global-local consistent prediction module and an independent classification learning module. The feature extractor is used to extract multiple feature vectors including global features and local features from an image; the independent adversarial learning module is used to distinguish whether an image is from a source domain or a target domain based on multiple feature vectors extracted from a source domain image and multiple feature vectors extracted from a target domain image; the independent pseudo-label generation module is used to generate corresponding multiple pseudo-labels based on multiple feature vectors extracted from a target domain image, and filter out unreliable pseudo-labels by setting an adaptive threshold; the global-local consistent prediction module is used to adopt a consistent prediction strategy to jointly predict the retained multiple pseudo-labels to obtain the pseudo-label of the target domain image; the independent classification learning module is used to train the adaptive global-local representation learning and unified prediction network model based on the source domain image and the corresponding label, as well as the target domain image and the corresponding pseudo-label; The recognition module is used to input the image to be predicted into the trained adaptive global-local representation learning and unified prediction network model, and output the corresponding pseudo label through the global-local consistency prediction module; The independent pseudo-label generation module is specifically used to: Take any eigenvector f i t Input the independent pseudo-label generation module and output the corresponding pseudo-label, including: The eigenvector f i t Input the corresponding classifier G in the independent pseudo-label generation module i In the example, we get the prediction scores of 7 categories: Where, is the classifier G i Based on the feature vector f i t Predicting target domain images is the score of category j, j = 0, 1…, 6, i = 0, 1…, 6; By s i Get the predicted class p with the highest probability: p=argmax(s i ) Using classifier G i The number of pseudo labels learned is used to calculate the threshold t of 7 categories i for: Where, Represents the classifier G i The threshold that needs to be reached in order to generate a pseudo label for category j, Calculated by the following equation: Where θ is a fixed preset threshold, It's about The nonlinear mapping function, Represents the classifier G i To generate the ratio of pseudo labels of class j to all pseudo labels, it is calculated as follows: Where, is the classifier G i The number of pseudo labels generated, c is the total number of categories, and max c is the number of pseudo labels generated by the category with the largest number of pseudo labels; The value range is 0-1, and by rewriting the denominator in the equation, right Sensitivity: Reliable labels are filtered by setting a dynamic threshold. The formula is as follows: Where -1 indicates that pseudo label generation fails; class p is the feature vector f i t The corresponding pseudo labels; The seven feature vectors are input into the independent pseudo-label generation module to obtain the corresponding pseudo-labels respectively; The independent adversarial learning module includes 7 domain discriminators, each corresponding to 7 facial feature vectors. The learning process of the independent adversarial learning module is expressed as follows: Where, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t };D i is a member of D, D is 7 domain discriminators, Is expected as D i Loss; log[D i (f i )] is the binary classification cross entropy function, which is used to compare the probability distribution of the source domain with the probability distribution of the target domain; i = 0, 1, 2, ..., 6; D s 、D t Represent the source domain dataset and target domain dataset respectively; The independent classification learning module includes 7 classifiers G i , corresponding to 7 facial feature vectors; for the source domain image There is an accurate label from The 7 eigenvectors f extracted from i s Corresponding labels For existing pseudo labels The target domain image The pseudo label Assign to image The 7 extracted eigenvectors f i t After the assignment is completed, classification learning is performed according to the following formula: Where, f i s Represents an image from the source domain Any one of the 7 extracted eigenvectors, f i t Represents the image from the target domain Any one of the 7 extracted eigenvectors, f i ={f i s ,f i t };G i Represents any one of the seven classifiers; represents the calculation operation of multi-class cross entropy, and defines the expected entropy as the classification loss; i = 0, 1, 2, ..., 6; D s 、D t represent the source domain dataset and target domain dataset respectively.

6. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the cross-domain facial expression recognition method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-domain facial expression recognition method according to any one of claims 1 to 4 is implemented.

Images