A face expression prediction method based on dynamic distribution fusion

Abstract

The application discloses a kind of based on dynamic distribution fusion facial expression prediction method, including obtaining facial expression dataset, the facial picture in the data set obtained is preprocessed, obtains preprocessed dataset;Auxiliary branch is constructed, and based on auxiliary branch design double-branch neural network model;Sample distribution processing is carried out to the preprocessed data set obtained using the auxiliary branch constructed;Class distribution is constructed, and the sample distribution obtained is mined emotional information processing;Dynamic distribution fusion processing is carried out to the class distribution constructed and the sample distribution extracted;Multi-task learning framework is constructed, and double-branch neural network model is optimized;Facial expression prediction is realized using the optimized double-branch neural network model;The superiority compared with single-label learning is shown by introducing label distribution learning in the application;Dynamic distribution fusion is proposed, and the utility of label distribution learning is fully played;And the prediction performance of the application is good, efficiency is high, error is less.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a facial expression prediction method based on dynamic distribution fusion. BACKGROUND

[0002] Facial expression recognition is an important research direction in the field of computer vision. As a subfield of emotion recognition, facial expression recognition can determine the expression state of a face through analysis of a face image, and provides important support for human-computer interaction, emotion computing, intelligent monitoring and other fields.

[0003] The process of facial expression recognition mainly includes facial expression image acquisition and preprocessing, facial expression feature extraction and facial expression classification. Facial expression preprocessing is to obtain the accurate position of a face from the acquired facial expression image through face detection and face alignment, and to exclude the interference of the picture background. The success rate of preprocessing is mainly affected by factors such as image quality, light intensity and occlusion. Common facial expression feature extraction includes geometric features, appearance features, hybrid features and deep features. The first three are traditional hand-crafted features, which were widely used in the early stage of facial expression recognition research, but these methods often have problems such as low accuracy and poor robustness. In recent years, with the rapid development of deep learning technology, deep features extracted by deep convolutional neural networks have achieved good performance in facial expression recognition tasks. Facial expression classification is the last step of facial expression recognition. Traditional hand-crafted features are often classified using K-nearest neighbor method, support vector machine, random forest, Adaboost algorithm, Bayesian network and single-layer perceptron. In the deep learning framework, facial expression recognition can be performed in an end-to-end manner, that is, the deep neural network directly classifies and optimizes the features after learning the features.

[0004] The facial expression model is mainly divided into 2D, 2.5D and 3D: the 2D face is the RGB face image photographed by a normal camera or the infrared image photographed by an infrared camera, which is the image representing color or texture under a certain perspective and does not contain depth information; the 2.5D face is the face depth image photographed by a depth camera under a certain perspective, the curved surface information is not continuous, and the depth information of the unoccluded part; the 3D face is the point cloud or mesh face image synthesized by multiple angle face depth images, which has complete curved surface information and contains depth information. The research on 2D facial expression recognition has a long history, and the software and hardware technologies are relatively complete, and has been widely applied. However, since the 2D face only reflects two-dimensional plane information and does not contain depth information, it cannot completely express the real face. Compared with the 2D face, the 3D face is not affected by factors such as light, occlusion or posture, and has better ability to more truly reflect face information and is applied in face synthesis, face migration and other tasks. The 3D face generally obtains face depth information through professional equipment, mainly including binocular cameras, RGB-D cameras based on the principle of structured light and TOF cameras based on the principle of light flight time. Due to the easy availability of 2D face, 2D facial expression recognition still occupies a dominant position.

[0005] Most of the current methods for facial expression prediction choose single-label learning method to realize facial expression prediction. Although these methods have achieved good prediction performance, due to the insufficient emotional information contained in the single label, it is difficult to describe the fuzzy or mislabeled samples, and the neural network is prone to overfitting, which makes it difficult to further improve the prediction accuracy.

[0006] There are also a few methods that choose label distribution learning method to realize facial expression prediction. Unlike single-label learning methods, these methods use label distribution weights instead of single labels for training. Compared with single labels, label distribution contains more emotional information, which can effectively avoid overfitting during training, and thus has significant advantages. However, label distribution annotation is often difficult to obtain, so facial expression datasets that only provide single-label annotation still occupy a dominant position. In recent years, label distribution learning methods focus on constructing label distribution from single label, but the constructed label distribution is generally of low quality and cannot fully play the advantages of label distribution learning. SUMMARY

[0007] The purpose of the present application is to provide a facial expression prediction method based on dynamic distribution fusion with good prediction performance, high efficiency and few errors.

[0008] The facial expression prediction method based on dynamic distribution fusion provided by the present application comprises the following steps:

[0009] S1. Obtain a facial expression dataset, preprocess the facial pictures in the obtained dataset, and obtain a preprocessed dataset;

[0010] S2. Construct an auxiliary branch, and design a double-branch neural network model based on the auxiliary branch;

[0011] S3. Use the auxiliary branch constructed in step S2 to process the preprocessed dataset obtained in step S1 to extract sample distribution;

[0012] S4. Construct a category distribution, and process the sample distribution obtained in step S3 to mine emotional information;

[0013] S5. Perform dynamic distribution fusion processing on the category distribution constructed in step S4 and the sample distribution obtained in step S3;

[0014] S6. Construct a multi-task learning framework, and optimize the double-branch neural network model designed in step S2;

[0015] S7. Use the double-branch neural network model optimized in step S6 to realize facial expression prediction.

[0016] The facial expression dataset obtained in step S1 is preprocessed for the facial pictures in the obtained dataset, and a preprocessed dataset is obtained, which specifically includes:

[0017] The facial expression dataset is set as and the dataset covers C class labels and N samples, uses the MTCNN algorithm for face alignment processing, and outputs fixed-size face pictures; the output face pictures are scaled to a given size, and the RandAugment technology is used for data augmentation; the mean and standard deviation of the ImageNet dataset are used for normalization processing of the RGB channels of the face pictures.

[0018] The auxiliary branch constructed in step S2 and the double-branch neural network model designed based on the auxiliary branch specifically include:

[0019] The ResNet18 network model is used to construct the double-branch neural network model. The ResNet18 network model is divided into two parts: the first layer of the frozen ResNet18 network model is used as a feature extractor, and the last three layers of the ResNet18 network model are used as a feature discriminator, and the feature discriminator is defined as a target branch. Based on the target branch, an auxiliary branch is constructed, and the parameters and structure of the auxiliary branch are consistent with those of the target branch. Based on the feature extractor, the target branch, and the constructed auxiliary branch, a double-branch neural network model is designed.

[0020] The auxiliary branch constructed in step S2 is used to extract the sample distribution of the preprocessed data set obtained in step S1, and the specific process includes:

[0021] The probability distribution output by the auxiliary branch constructed in step S2 is used as the sample distribution, and the sample distribution is expressed by the following formula:

[0022]

[0023]

[0024] Wherein, is the sample distribution of sample x i , y j is the jth label, is the description degree of sample x i for label y j , and is the prediction probability of sample x i belonging to label y j by the auxiliary branch.

[0025] The auxiliary branch is trained by cross-entropy loss to improve and maintain the ability of the auxiliary branch to extract the sample distribution, and the cross-entropy loss function is expressed by the following formula:

[0026]

[0027] Wherein, L ce is the cross-entropy loss function, is the cth value of the logical label y i of sample x i , and is the prediction probability of sample x i belonging to category c by the auxiliary branch.

[0028] The category distribution constructed in step S4 is used to mine the sentiment information of the sample distribution obtained in step S3, and the specific process includes:

[0029] The category distribution is used to find the sentiment information implied in the sample distribution, eliminate the influence of sample distribution error on model performance, and the category distribution is expressed by the following formula:

[0030]

[0031] Wherein, is the category distribution of category c, is the category distribution of sample x i belonging to category c, and N c is the number of samples belonging to category c.

[0032] A threshold t is set to determine whether the output class distribution meets the set robustness requirement. If the label y j If the description degree of the class c does not reach the threshold t, the threshold distribution is temporarily used to replace the class distribution to train the model, and the following formula is used for description:

[0033]

[0034]

[0035]

[0036] Wherein, is the class distribution of class c, is the threshold distribution of class c, is the label y j The description degree of the class c.

[0037] The dynamic distribution fusion processing is performed on the class distribution constructed in step S4 and the sample distribution obtained in step S3, and specifically includes:

[0038] The dynamic distribution fusion is based on the class distribution, and the class distribution and the sample distribution are adaptively fused according to the attention weight of each sample. The dynamic distribution fusion includes two steps: attention weight extraction and adaptive distribution fusion;

[0039] 1) Attention weight extraction:

[0040] For attention weight extraction, two attention modules are embedded into the last layer of the two branches to obtain the attention weight of the sample. The attention module is composed of a fully connected layer and a Sigmoid function. The feature output by each branch is input into the corresponding attention module to extract the attention weight of each sample. The attention weight value is used to determine whether a sample is clear or blurred, and the weight value is used for adaptive distribution fusion. The feature output by each branch is multiplied by the corresponding attention weight and then input into the corresponding classifier;

[0041] The process of attention weight extraction is as follows:

[0042] a. For a batch of samples, the face features output by the feature extractor are input into the auxiliary branch and the target branch at the same time;

[0043] b. The attention weights output by the two attention modules are averaged to benefit from the sample blur degree discrimination ability of the two branches at the same time, and the following formula is used to represent the averaged attention weight:

[0044]

[0045] Wherein, and The samples x output by the attention modules of the two branches respectively. i Attention weights;

[0046] c. Regularize the attention weights by ranking to prevent the discrimination ability of the attention module from deteriorating:

[0047]

[0048] L RR =max(0,δ-(w) H -w L ))

[0049] Among them, w H and w L Let δ be the average attention weights of the M high-weight samples and the NM low-weight samples, respectively, where δ is a fixed difference. δ and M are directly taken from the values ​​in the SCN method using the same attention module. RR It is the loss function for sorting regularization;

[0050] d. Normalize the attention weights, using the following formula to represent the process:

[0051]

[0052] Among them, w min This is the lower bound of the attention weight. It is sample x i Attention weights after sorting regularization It is sample x i Attention weights after normalization;

[0053] 2) Adaptive distributed fusion:

[0054] The following formula is used to represent the mixed distribution after fusion:

[0055]

[0056] in, It is sample x i The resulting mixed distribution It is sample x i Category distribution, It is sample x i Label distribution It is sample x i Attention weights after normalization.

[0057] Step S6, which involves constructing a multi-task learning framework and optimizing the dual-branch neural network model designed in step S2, specifically includes:

[0058] ①Target branch optimization:

[0059] The target branch is trained by using the KL divergence loss, and the training process is represented by the following formula:

[0060]

[0061] Wherein, L kld is the KL divergence loss, is the description degree of the class c to the sample x i , and i is the predicted probability of the target branch for the sample x j belonging to the label y kld .

[0062] ②Multi-task learning framework:

[0063] A multi-task learning framework is constructed, the joint loss L is minimized through the joint learning of distribution prediction and expression recognition, so that the prediction performance of the model is optimized, and the joint loss function is represented by the following formula:

[0064] L = α1·L ce + α2·L RR + L i

[0065]

[0066]

[0067] Wherein, α1 and α2 are the weighted slope functions related to the training round e, and β is the threshold of the training round, and α1 and α2 are introduced to optimize the training process.

[0068] The double-branch neural network model optimized by step S6 is used to realize the prediction of facial expression, and specifically comprises:

[0069] The double-branch neural network model optimized by step S6 is used to realize the prediction of facial expression, and specifically comprises:

[0070] The facial expression prediction method based on dynamic distribution fusion provided by the application introduces label distribution learning, is based on the rich emotional information contained in the label distribution, effectively avoids overfitting in the training process, and has the superiority compared with single-label learning; meanwhile, dynamic distribution fusion is proposed, high-quality mixed distribution close to the real distribution is generated by using the extracted sample distribution and the mined class distribution, and the utility of label distribution learning is fully utilized; and the prediction performance of the application is good, the efficiency is high, and the error is small. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 a schematic diagram of the method flow of the method of the present application; DETAILED DESCRIPTION

[0072] As Figure 1 shown is a schematic diagram of the method flow of the method of the present application: the face expression prediction method based on dynamic distribution fusion provided by the present application comprises the following steps:

[0073] S1. Obtain a face expression dataset, pre-process the face pictures in the obtained dataset, and obtain a pre-processed dataset; specifically comprising:

[0074] Suppose the face expression dataset is S = {(x i ,y i ), i = 1, 2, …, N}, and the dataset covers C labels and N samples, since the sizes of the face pictures in different datasets are different, use the MTCNN algorithm for face alignment processing, output face pictures of a fixed size, the present application outputs face pictures of 100x100; scale the output face pictures to a given size, the present application obtains a given size of 224x224, and uses RandAugment technology for data augmentation; use the mean and standard deviation of the ImageNet dataset for normalization processing of the RGB channels of the face pictures;

[0075] S2. Construct an auxiliary branch, and design a double-branch neural network model based on the auxiliary branch, specifically comprising:

[0076] A ResNet18 network model is used to construct a double-branch neural network model. The ResNet18 network model is divided into two parts: the first layer of the frozen ResNet18 network model is used as a feature extractor, and the last three layers of the ResNet18 network model are used as a feature discriminator, and the feature discriminator is defined as a target branch. An auxiliary branch is constructed based on the target branch, and the parameters and structure of the auxiliary branch are consistent with those of the target branch. Based on the feature extractor, the target branch and the constructed auxiliary branch, a double-branch neural network model is designed;

[0077] S3. Use the auxiliary branch constructed in step S2 to extract sample distribution from the pre-processed dataset obtained in step S1, specifically comprising:

[0078] Directly using the probability distribution output by the ResNet18 network model to train the model will cause degradation of the performance of the model, and the probability distribution output by the auxiliary branch constructed in step S2 is used as the sample distribution, and the following formula is used to represent the sample distribution:

[0079]

[0080]

[0081] wherein, is the sample distribution of sample x i , y j is the j-th label, is the label y j for sample x i , and is the prediction probability of auxiliary branch for sample x i belongs to label y j ;

[0082] The auxiliary branch is trained by cross-entropy loss to improve and maintain the ability of the auxiliary branch to extract the sample distribution, and the cross-entropy loss function is expressed by the following formula:

[0083]

[0084] wherein, L ce is the cross-entropy loss function, is the c-th value of the logical label y i for sample x i , and is the prediction probability of auxiliary branch for sample x i belongs to class c;

[0085] S4. Constructing a class distribution, mining emotional information processing for the sample distribution obtained in step S3, specifically including:

[0086] Based on the sensitivity of deep neural network to fuzzy or mislabeled samples, the class distribution mining is used to find out the emotional information implied in the sample distribution, eliminate the influence of sample distribution error on the model performance, and the class distribution is expressed by the following formula:

[0087]

[0088] wherein, is the distribution of class c, is the class distribution of sample x i belongs to class c, and N c is the number of samples belonging to class c;

[0089] The category distribution mining is to sum up and average the sample distribution of all samples belonging to a category to obtain the category distribution of the corresponding category; due to the instability of the parameters of the auxiliary branch and trunk at the beginning of training, the category distribution cannot meet the stability requirements set, and the category distribution at this time cannot accurately describe each category of expression. In order to avoid the deterioration of the prediction performance of the model by the wrong category distribution, a threshold t is set to judge whether the output category distribution meets the set stability requirements. If the label y j If the description degree of the category c does not reach the threshold t, the threshold distribution is used to temporarily replace the category distribution to train the model. The threshold value is set between 0 and 1, and the specific value needs to be determined through ablation experiments. The threshold value is set based on the following phenomenon: the stronger the feature extraction ability of the model, the higher the value of the corresponding sample label position in the label distribution. By setting the threshold value, it can be judged whether the feature extraction of the model is in place; the following formula is used for description:

[0090]

[0091]

[0092]

[0093] Wherein, is the category distribution of category c, is the threshold distribution of category c, is the label y j is the description degree of category c;

[0094] S5. Dynamic distribution fusion processing is performed on the category distribution constructed in step S4 and the sample distribution obtained in step S3, specifically including:

[0095] The dynamic distribution fusion is based on the category distribution and adaptively fuses the category distribution and the sample distribution according to the attention weight of each sample. The dynamic distribution fusion is divided into two steps: attention weight extraction and adaptive distribution fusion;

[0096] 1) Attention weight extraction:

[0097] For attention weight extraction, two attention modules are embedded into the last layer of the two branches to obtain the attention weight of the sample. The attention module is composed of a fully connected layer and a Sigmoid function. The feature output by each branch is input into the corresponding attention module to extract the attention weight of each sample. The attention weight value can judge whether a sample is clear or blurred, and the weight value is used for adaptive distribution fusion; the feature output by each branch is multiplied by the corresponding attention weight and then input into the corresponding classifier;

[0098] The process of attention weight extraction is as follows:

[0099] a. For a batch of samples, the facial features output by the feature extractor are simultaneously input to the auxiliary branch and the target branch;

[0100] b. The attention weights output by the two attention modules are averaged to benefit from the sample blur discrimination ability of the two branches simultaneously, and the following formula is used to represent the averaged attention weights:

[0101]

[0102] wherein, and are the attention weights of the samples x i output by the attention modules of the two branches, respectively;

[0103] c. The attention weights are sorted and regularized to avoid degradation of the discrimination ability of the attention module:

[0104]

[0105] L RR = max(0, δ-(w H -w L ))

[0106] wherein, w H and w L are the average values of the attention weights of the M samples with high weights and the N-M samples with low weights, respectively, and δ is a fixed difference value, in order to avoid repeated experiments, δ and M directly use the values in the method SCN using the same attention module, which are set to 0.07 and 0.7N in the present application, respectively, and L RR is the loss function of the sorting regularization;

[0107] d. The attention weights are normalized, and the following formula is used to represent the processing process:

[0108]

[0109] wherein, w min is the lower limit of the attention weight, is the attention weight of the sample x i after sorting regularization, is the attention weight of the sample x i after normalization processing, and the hyperparameter w min is set to prevent the blurring of low attention weight samples from degrading the model performance during fusion, and the lower the attention weight, the higher the sample blur degree.

[0110] 2) Adaptive distribution fusion:

[0111] For adaptive distribution fusion, the category distribution and the sample distribution are adaptively fused based on the obtained attention weight to balance the robustness of the category distribution and the diversity of the sample distribution, and the following formula is used to represent the mixed distribution after fusion:

[0112]

[0113] wherein, is the category distribution of the sample x i after fusion, is the category distribution of the sample x i , is the label distribution of the sample x i , is the attention weight of the sample x i after normalization processing.

[0114] S6. A multi-task learning framework is constructed to optimize the dual-branch neural network model designed in step S2, specifically including:

[0115] ① Target branch optimization:

[0116] The target branch is trained using the KL divergence loss, and the following formula is used to represent the training process:

[0117]

[0118] wherein, L kld is the KL divergence loss, is the description degree of the category c to the sample x i , is the predicted probability of the target branch for the sample x i belonging to the label y j .

[0119] ② Multi-task learning framework:

[0120] A multi-task learning framework is constructed to minimize the joint loss L through joint learning of distribution prediction and expression recognition, thereby optimizing the prediction performance of the model; and the following formula is used to represent the joint loss function:

[0121] L = α1·L kld + α2·L ce + L RR

[0122]

[0123]

[0124] Here, α1 and α2 are weighted slope functions related to the training epoch e, and β is the threshold for the training epoch. α1 and α2 are introduced to optimize the training process. In the early stage of training, the auxiliary branches are trained to ensure that they can output sample distributions and class distributions that meet the set robustness requirements. In the later stage of training, the target branches are trained to avoid overfitting of the auxiliary branches. In the inference stage, the auxiliary branches are removed, and only the target branches are used to predict the expressions of the samples.

[0125] S7. The dual-branch neural network model obtained in step S6 is used to predict facial expressions, specifically including:

[0126] The dual-branch neural network model optimized in step S6 is used to predict facial expressions by outputting the probability distribution of each sample through the target branch. The expression with the highest predicted probability in the output probability distribution is selected as the predicted expression of the sample.

Claims

1. A dynamic distribution fusion-based facial expression prediction method, comprising the following steps: S1. Obtaining a facial expression dataset, preprocessing the facial pictures in the obtained dataset, and obtaining a preprocessed dataset; S2. Constructing an auxiliary branch and designing a double-branch neural network model based on the auxiliary branch; S3. Using the auxiliary branch constructed in step S2 to extract sample distribution from the preprocessed dataset obtained in step S1; specifically including: The probability distribution output by the auxiliary branch constructed in step S2 is used as the sample distribution, and the following formula is used to represent the sample distribution: in, For the sample The sample distribution For the first Class tags, For tags For the sample The degree of description To assist branches in sampling Belongs to the label The predicted probability; The auxiliary branch is trained by cross-entropy loss to improve and maintain the ability of the auxiliary branch to extract sample distribution, and the following formula is used to represent the cross-entropy loss function: wherein, is a cross-entropy loss function, is a sample is a cth value of a logical label for the sample, is a prediction probability of the auxiliary branch for the sample to belong to class c; S4. Constructing a class distribution and mining emotional information from the sample distribution obtained in step S3; specifically including: Using class distribution mining to find the emotional information implied in the sample distribution, eliminating the influence of sample distribution error on model performance, and using the following formula to represent the class distribution: wherein, is the class distribution of class c, is the number of samples belonging to class c, is the class distribution of class c, is the number of samples belonging to class c, Setting a threshold to determine whether the class distribution of the output meets the set robustness requirement, if the label The description degree of the class c does not reach the threshold , temporarily replace the class distribution with the threshold distribution to train the model, which is described by the following formula: wherein, is a class distribution of class c, is a threshold distribution of class c, is a label is a degree of description for class c; S5. Dynamic distribution fusion processing is performed on the class distribution constructed in step S4 and the sample distribution obtained in step S3; S6. Constructing a multi-task learning framework to optimize the double-branch neural network model designed in step S2; S7. Using the double-branch neural network model optimized in step S6 to realize facial expression prediction. 2.The dynamic distribution fusion based facial expression prediction method according to claim 1, characterized in that The facial expression dataset obtained in step S1 is preprocessed for the facial pictures in the obtained dataset, and a preprocessed dataset is obtained, specifically including: The face expression dataset is set as , and the dataset covers class labels and samples, face alignment processing is performed using the MTCNN algorithm, and a fixed size face picture is output; the output face picture is scaled to a given size, and data augmentation is performed using the RandAugment technology; the face picture RGB channel is normalized using the mean and standard deviation of the ImageNet dataset. 3.The dynamic distribution fusion based facial expression prediction method according to claim 2, characterized in that The auxiliary branch constructed in step S2 and the double-branch neural network model designed based on the auxiliary branch, specifically including: A ResNet18 network model is used to construct a double-branch neural network model; the ResNet18 network model is divided into two parts: the first layer of the frozen ResNet18 network model is used as a feature extractor, and the last three layers of the ResNet18 network model are used as a feature discriminator, which is defined as a target branch; an auxiliary branch is constructed based on the target branch, and the parameters and structure of the auxiliary branch are consistent with those of the target branch; based on the feature extractor, the target branch and the constructed auxiliary branch, a double-branch neural network model is designed. 4.The dynamic distribution fusion based facial expression prediction method according to claim 1, characterized in that The dynamic distribution fusion processing of the class distribution constructed in step S4 and the sample distribution obtained in step S3 in step S5, specifically including: Dynamic distribution fusion is based on class distribution and adaptively fuses class distribution and sample distribution according to the attention weight of each sample; dynamic distribution fusion is divided into two steps: attention weight extraction and adaptive distribution fusion; 1) Attention weight extraction: For attention weight extraction, two attention modules are embedded into the last layer of the two branches respectively to obtain the attention weight of the sample; the attention module is composed of a fully connected layer and a Sigmoid function, the feature output by each branch is input into the corresponding attention module to extract the attention weight of each sample, the attention weight value is used to judge whether a sample is clear or blurred, and the weight value is used for adaptive distribution fusion; the feature output by each branch is multiplied by the corresponding attention weight and then input into the corresponding classifier; The process of attention weight extraction is as follows: a. For a batch of samples, the face feature output by the feature extractor is input into the auxiliary branch and the target branch at the same time; b. The attention weights output by the two attention modules are averaged to benefit from the sample blur discrimination ability of the two branches at the same time, and the following formula is used to represent the average attention weight: in, and Samples output by the attention modules of the two branches respectively Attention weights; c. The attention weight is sorted and regularized to avoid the degradation of the discrimination ability of the attention module: wherein, and are high-weighted attention weight averages of the samples with low-weighted is a fixed difference value, and directly use values in the method SCN with the same attention module, is a loss function of ranking regularization; d. The attention weight is normalized, and the following formula is used to represent the processing process: wherein, is a lower bound for the attention weight, is a sample is the attention weight after sorting regularization, is a sample is the attention weight after normalization. 2) Adaptive distribution fusion: The following formula is used to represent the mixed distribution after fusion: wherein, is a sample the fused mixed distribution, is a sample the class distribution of, is a sample the label distribution of, is a sample the normalized attention weight. 5.The dynamic distribution fusion based facial expression prediction method according to claim 4, characterized in that Step S6 constructs a multi-task learning framework to optimize the double-branch neural network model designed in step S2, specifically including: ① Target branch optimization: The target branch is trained using the KL divergence loss, and the following formula is used to represent the training process: wherein, is the KL divergence loss, is the degree of description of the sample by the class c, is the predicted probability that the target branch belongs to the label for the sample . ② Multi-task learning framework: A multi-task learning framework is constructed to minimize a joint loss by joint learning of distribution prediction and expression recognition Thus, the prediction performance of the model is optimized, and the joint loss function is expressed by the following formula: wherein, and is a weighted slope function related to the training epoch, is a threshold value for the training epoch, introducing and optimizes the training process. 6.The method of claim 5, wherein Step S7 uses the double-branch neural network model optimized in step S6 to realize face expression prediction, specifically including: The double-branch neural network model optimized in step S6 outputs the probability distribution of each sample through the target branch to predict the face expression, and the expression corresponding to the highest prediction probability in the output probability distribution is selected as the predicted expression of the sample.

Images