Facial depression recognition method based on joint mark distribution learning

Abstract

The invention discloses a facial depression recognition method based on joint mark distribution learning, and the method comprises the steps: taking facial images as training data, marking each facial image as a single-mark BDI-II score, taking the single-mark BDI-II score as a corresponding description degree, and generating depression score distribution of each facial image through Gaussian distribution; a deep network model with 68-dimensional output is designed, the first 64 dimensions are used for learning depression score distribution, and each face image comprises a real depression score and a score in a neighborhood; the last four bits are used for learning depression grade distribution, the depression grade distribution comprises description degrees of real depression grades and description degrees of other grades, a multi-task joint mark distribution learning framework is designed, and joint learning of depression score distribution and depression grade distribution is carried out. The problem of uncertainty of single-mark BDI-II score marking is solved, and the problem of insufficient training data is relieved; the depression score and the depression level are considered in the training process through joint mark distribution learning, and the depression diagnosis performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a facial depression recognition method based on joint label distribution learning. Background Art

[0002] Depression is a common mental disorder. People with depression usually experience sadness and self-denial, which can cause serious harm to people's health and life. Clinically, the diagnosis of depression usually relies on the subjective evaluation of psychologists. This process requires professional psychological knowledge and is difficult to promote the diagnosis of depression on a large scale in the early stages. Therefore, automated depression diagnosis has attracted widespread attention in clinical research in recent years. In particular, with the advancement of deep learning and computer vision-related technologies, depression recognition methods based on facial images have attracted widespread attention from researchers. With the popularization of video equipment, researchers can obtain a large amount of facial image data, use these facial images to build a facial depression recognition system, and deploy it in the terminal system, which is expected to achieve large-scale promotion of early depression diagnosis.

[0003] In general, existing methods generally treat the facial depression recognition problem as a single-label Beck Depression Inventories (BDI)-II score learning and prediction problem. The BDI-II score ranges from 0 to 63, and the larger the value, the more severe the depression. The facial depression recognition method based on deep learning first uses the facial image data annotated with the single-label BDI-II score to train a deep neural network (such as a convolutional deep network or Transformer), and then uses the trained network to predict the BDI-II score of the new facial image. However, existing methods still face the following three challenges: The first is the lack of sufficient labeled face image datasets. The BDI-II score is obtained by evaluating 21 BDI-II questionnaires (each questionnaire has a score range of 0 to 3), so the dataset annotation process is difficult and it is expensive to obtain sufficient labeled datasets. The lack of sufficient labeled datasets has led to the insufficient performance of existing deep learning-based facial depression recognition methods, and the recognition accuracy is difficult to meet the application requirements of actual depression diagnosis.

[0004] The second is that the uncertainty and inaccuracy of BDI-II score annotations are ignored. There is a certain ambiguity between the different options in the BDI-II questionnaire, and the assessees usually choose a similar option, which leads to a certain uncertainty and inaccuracy in the BDI-II score itself. For example, a face image with a BDI-II score of 20 will also be labeled as 19 or 21 with a certain probability. Existing methods treat BDI-II scores as absolutely correct marks, ignoring the uncertainty of the standard process, resulting in unstable recognition performance.

[0005] The third is that the training process does not take into account the level of depression. The training process of existing methods usually only uses the BDI-II score to train a prediction model, but in practical applications, it is often necessary to give a diagnosis of the level of depression, such as mild, moderate, severe, and other different levels. The BDI-II score and depression level measure the depression information of a face from a fine-grained and coarse-grained perspective respectively, and there is a strong correlation between the two. However, the existing methods ignore the level of depression during the training process, resulting in poor performance of the model when diagnosing depression levels. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a facial depression recognition method based on joint label distribution learning.

[0007] The objective of the present invention is achieved through the following technical solutions: The present invention discloses a facial depression recognition method based on joint label distribution learning, comprising the following steps: S1. Depression score distribution generation: facial images are used as training data, and each facial image is annotated with a single-label BDI-II value as its corresponding descriptive degree. The sum of all descriptive degrees is equal to 1, which is used to represent the severity of depression of the face. The BDI-II score ranges from 0 to 63. The depression score distribution of each facial image is generated through Gaussian distribution; S2. Depression score distribution learning: Design a 68-dimensional deep network model, where the first 64 dimensions are used to learn the depression score distribution. Each facial image includes learning the true depression score and the score in the neighborhood. S3. Depression level distribution learning: The last 4 bits of the 68-dimensional output deep network model are used to learn the depression level distribution. The depression level distribution includes the descriptive degree of the real depression level and the descriptive degrees of other levels. S4. Joint labeled distribution learning: Design a multi-task joint labeled distribution learning framework to jointly learn the depression score distribution and depression level distribution.

[0008] Furthermore, the step S1 specifically includes the following steps: S11. Assume that the training data is , ,in Indicates facial images, For the The single-label BDI-II depression score corresponding to the facial image; facial images, through Gaussian distribution Generate the corresponding depression score distribution, where is the depression score, is the standard deviation which is usually set to 1. is the mean, is the standard deviation; S12, through the formula Normalize, where is the normalization parameter; depression score The corresponding descriptive degrees constitute a 64-dimensional label distribution = ,in Depression score for The descriptive degree, arrive Indicates all The depression score distribution of each facial image is generated by taking the value of

[0009] Preferably, the step S2 specifically includes the following steps: S21. Design a deep network model with 68-dimensional output, where the first 64 dimensions arrive Used to learn the distribution of depression scores. After the Softmax function, the j-th dimension output is: ; The predicted depression score distribution is ; S22. Design two loss functions to learn the distribution of depression scores. First, use KL divergence to design the first loss function. Learn the overall depression score distribution; S23. The predicted depression score is equal to the mean of the predicted depression score distribution, which is equal to , so that it is close to the actual BDI-II depression score, and the second loss function is designed by the absolute value To learn the true depression score.

[0010] Preferably, the step S3 specifically includes the following steps: S31, transforming the predicted depression score distribution into the predicted depression grade distribution: ,in For depression level, express The corresponding depression score interval, that is, the predicted descriptive degree of the score within the interval and the predicted descriptive degree of the corresponding level, obtains the mapped predicted depression level distribution ; Use KL divergence to design the third loss function To learn the depression level distribution; S32, the last 4 dimensions of the deep network model arrive Used to learn the depression level distribution, after adding the Softmax function The prediction descriptiveness of the dimension is equal to ; Get the predicted depression level distribution ; Design KL divergence and design the fourth loss function Used to learn the distribution of depression levels; S33, for , based on depression scores Design the fifth loss function ,in is a constant, is the real depression level; if the real depression level When the prediction descriptiveness of is higher than that of other levels, the fifth loss is 0, otherwise a positive penalty is generated.

[0011] Preferably, the step S4 specifically includes: designing a sixth loss function To train the 68-dimensional output deep network model, wherein the deep network model is trained by a hybrid multi-head cross-attention network architecture, , .

[0012] Preferably, step S4 also includes: the deep network model is also trained by an existing convolutional neural network or a Transformer model.

[0013] The beneficial effects of the present invention are: 1) The present invention transforms the single-label BDI-II score into a 64-dimensional label distribution, in which each element represents the degree of association between the corresponding score and the face image. A face image is related to both the real BDI-II score and other scores in the neighborhood, thus solving the uncertainty problem of single-label BDI-II score annotation.

[0014] 2) Compared with learning a single-labeled BDI-II score, by learning a 64-dimensional label distribution, a face image can be used to learn both the real BDI-II score and other scores in the neighborhood, thus also having the effect of augmenting training data.

[0015] 3) The present invention jointly learns the depression score distribution and the level distribution, and the learning of the two has a mutually reinforcing effect, which can not only improve the accuracy of depression score prediction, but also improve the accuracy of depression level diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the steps of a facial depression recognition method based on joint label distribution learning according to an embodiment of the present invention; Figure 2 A schematic diagram of depression score distribution and single-marker BDI-II score comparison in an embodiment of the present invention; Figure 3 A schematic diagram showing the comparison of the results of a facial depression recognition method based on joint label distribution learning and a single label learning comparison algorithm according to an embodiment of the present invention; Figure 4 It is a schematic diagram for comparing the results of a facial depression recognition method based on joint label distribution learning and a label distribution learning comparison algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] The present invention provides a facial depression recognition method based on joint label distribution learning, which generates a label distribution according to the single-label BDI-II score to describe the depression information of the face image. A label distribution is a multidimensional vector covering the entire label space, in which each element is called a descriptive degree, which represents the degree of relevance of the corresponding label, and the sum of the descriptive degrees of all labels is 1. An additional label distribution is also introduced to describe the depression level information of the face image. The depression score represents the depression information of the face image from a fine granularity, while the depression level describes the depression information of the face from a coarse granularity, and the two labeled information are related. In practical applications, it is often necessary to diagnose the depression level of the patient. Therefore, the present invention jointly considers the depression score and level, captures the depression information contained in the face from two granularities, and designs a joint label distribution learning framework to learn the two label distributions. The schematic diagram of the step flow of the present invention is shown as follows. Figure 1 As shown, the specific steps include: S1. Depression score distribution generation: facial images are used as training data, and each facial image is annotated with a single-label BDI-II score as its corresponding descriptive degree. The sum of all descriptive degrees is equal to 1, which is used to indicate the severity of depression in the face. The BDI-II score ranges from 0 to 63, and the depression score distribution of each facial image is generated through Gaussian distribution. A labeled distribution is a 64-dimensional vector covering the entire interval (i.e., 0 to 63), in which each element is called a descriptive degree, which indicates the relevance of the corresponding score to the image, and the sum of all elements is equal to 1.

[0019] S2. In the depression score distribution generated based on Gaussian distribution, the descriptiveness decreases from the real BDI-II score to both sides. On the one hand, the real depression score has the highest descriptiveness, so the labeled distribution can effectively represent the depression information of the face image. On the other hand, the neighborhood interval of the real score also has a certain degree of descriptiveness, so a face image also has a certain degree of correlation with the score in the neighborhood, which can effectively solve the uncertainty problem of single-label BDI-II score annotation. In addition, the depression score distribution is learned during training. A face image can be used to learn both the real depression score and the score in the neighborhood, which can achieve the effect of augmenting training data. Depression score distribution learning, a 68-dimensional output deep network model is designed, of which the first 64 dimensions are used to learn the depression score distribution.

[0020] S3. The descriptive degree of the real depression score and its neighborhood is high, which can ensure that the descriptive degree of the corresponding depression level is also high, so the depression level distribution can effectively reflect the depression level information of a face. Secondly, the depression level distribution includes the descriptive degree of the real depression level as well as the descriptive degree of other levels. Learning this distribution during training can achieve the effect of augmenting training data. Depression level distribution learning, the last 4 bits of the 68-dimensional output deep network model are used to learn the depression level distribution.

[0021] S4. Joint labeled distribution learning: Design a multi-task joint labeled distribution learning framework to jointly learn the depression score distribution and depression level distribution.

[0022] Specifically, the step S1 comprises the following steps: S11. Assume that the training data is , ,in Indicates facial images, For the The single-label BDI-II depression score corresponding to the facial image; facial images, through Gaussian distribution Generate the corresponding depression score distribution, where is the depression score, is the standard deviation which is usually set to 1. is the mean, is the standard deviation; S12, through the formula Normalize, where is the normalization parameter; depression score The corresponding descriptive degrees constitute a 64-dimensional label distribution = ,in Depression score for The descriptive degree, arrive Indicates all The depression score distribution of each facial image is generated by taking the value of

[0023] Specifically, the step S2 comprises the following steps: S21. Design a deep network model with 68-dimensional output, where the first 64 dimensions arrive Used to learn the distribution of depression scores. After the Softmax function, the j-th dimension output is: ; The predicted depression score distribution is ; S22. Design two loss functions to learn the distribution of depression scores. First, use KL divergence to design the first loss function. Learn the overall depression score distribution; S23. The predicted depression score is equal to the mean of the predicted depression score distribution, which is equal to , so that it is close to the actual BDI-II depression score, and the second loss function is designed by the absolute value To learn the true depression score.

[0024] Specifically, step S3 includes the following steps: S31, transforming the predicted depression score distribution into the predicted depression grade distribution: ,in For depression level, express The corresponding depression score interval, that is, the predicted descriptive degree of the score within the interval and the predicted descriptive degree of the corresponding level, obtains the mapped predicted depression level distribution ; Use KL divergence to design the third loss function To learn the depression level distribution; S32, the last 4 dimensions of the deep network model arrive Used to learn the depression level distribution, after adding the Softmax function The prediction descriptiveness of the dimension is equal to ; Get the predicted depression level distribution ; Design KL divergence and design the fourth loss function Used to learn the distribution of depression levels; S33. In addition to learning the overall depression level distribution from the above two levels, we should also focus on the actual depression level to improve the accuracy of model prediction. , based on depression scores Design the fifth loss function ,in is a constant, is the real depression level; if the real depression level When the prediction descriptiveness of is higher than other levels, the fifth loss is 0, otherwise a positive penalty is generated. Therefore, optimizing the fifth loss function helps to focus on the true depression level in predicting the depression level distribution.

[0025] Specifically, the step S4 specifically includes: Design the sixth loss function To train the 68-dimensional output deep network model, the deep network model adopts the hybrid multi-head cross attention network (HMHN) architecture and sets and The method proposed in the present invention is not limited to a specific deep network, and can be implemented based on existing convolutional neural networks or Transformer models for training.

[0026] In addition to using BDI-II scores, existing work also uses Patient Health Questionnaire (PHQ-8) scores to mark the degree of depression. The value range of PHQ-8 is 0 to 24, and the larger the value, the deeper the degree of depression. Similarly, according to the size of the PHQ-8 value, the degree of depression can be divided into 5 levels, as shown in Table 1. The method of the present invention is also applicable to the face image dataset annotated with PHQ-8 scores. First, a 25-dimensional depression score distribution is generated, and then a 5-dimensional depression level distribution is generated according to the mapping relationship, and finally the two labeled distributions are jointly learned.

[0027] Table 1 PHQ-8 scores and corresponding depression levels PHQ-8 score Depression level 0-4 none 5-9 Mild 10-14 medium 15-19 More serious 20-24 serious We trained and tested the network on three public datasets: AVEC 2014, AVEC 2014 (based on BDI-II score annotations), and AVEC2019 (based on PHQ-8 score annotations). Given a test image, we predict the depression score distribution through the first 64-dimensional output of the network, calculate the mean of the distribution as the predicted depression score, and give the predicted depression level.

[0028] The present invention generates a multi-dimensional depression score distribution based on the single-label BDI-II score to describe the depression information of the face image, which can solve the problem of uncertainty in the annotation information of the single-label BDI-II score. Figure 2 As shown; in the single label, only the true BDI-II score of 43 is absolutely correct, and the other labels are irrelevant, and the uncertainty of the labeling is completely ignored. In contrast, in the depression score distribution, the true BDI-II score of 43 has the highest descriptive degree, and the scores in the neighborhood (such as 42 and 44) ​​also have a certain degree of descriptiveness, taking into account the uncertainty of the labeling. Secondly, compared with learning a single label, by learning the depression score distribution, a face image can be used for learning the true BDI-II score as well as for learning other related scores, which has the effect of augmenting the training data and can improve the performance of facial depression recognition. The schematic diagram of the results of the comparison algorithm between the present invention and the single-label learning is shown in Figure 3 As shown in Figure 2, the same network architecture is used for training on three public datasets AVEC 2013, AVEC 2014, and AVEC 2019. The proposed method is more efficient than the most advanced single-label comparison algorithm. [1] Better recognition results were achieved, with the Mean Absolute Error (MAE) index reduced by 1.15%, 1.99%, and 3.74%, and the Root Mean Square Error (RMSE) index reduced by 2.17%, 2.24%, and 3.48%, respectively.

[0029] The present invention also considers depression grade information, and transforms depression score distribution into grade distribution by means of the relationship between depression score and grade. In the training process, the present invention jointly learns depression score distribution and grade distribution, and the joint learning of the two has a mutually reinforcing effect, further improving the performance of facial depression recognition. Figure 4The figure is a schematic diagram of the results of the present invention and the label distribution learning comparison algorithm. The same network architecture was used for training on three public data sets AVEC 2013, AVEC 2014, and AVEC 2019. The method of the present invention achieved better recognition results than the label distribution learning comparison algorithm that only learned the depression score distribution. The mean absolute error (MAE) index was reduced by 0.33%, 1.66%, and 1.99%, respectively, and the root mean square error (RMSE) index was reduced by 1.23%, 2.11%, and 2.51%, respectively.

[0030] Most of the existing technologies regard facial depression recognition as a single-label BDI-II score prediction problem, while the present invention transforms this problem into a multi-dimensional label distribution learning problem, which can bring two advantages. First, there is uncertainty in the BDI-II score labeling itself. For example, a face image labeled as 20 may also be labeled as 19 and 21 with a high probability. The present invention transforms the single-label BDI-II score into a 64-dimensional label distribution, in which each element represents the degree of association between the corresponding score and the face image. A face image is related to both the real BDI-II score and other scores in the neighborhood, thereby solving the single-label BDI-II score labeling uncertainty problem. Secondly, compared with learning a single-label BDI-II score, by learning a 64-dimensional label distribution, a face image can be used for learning both the real BDI-II score and other scores in the neighborhood, so it can also play the role of augmenting training data. Secondly, existing methods usually only use BDI-II scores to train a prediction model, and then use the model to give a predicted BDI-II score and further give a depression level diagnosis. These methods only consider the BDI-II score during the training process, but do not consider the depression level information, resulting in poor performance of the model when diagnosing the degree of depression. The BDI-II score and the depression level describe the depression information of a face from a fine-grained and coarse-grained perspective, respectively, and there is a strong correlation between the two. Finally, the present invention also designs a joint label distribution learning method to jointly learn the depression score distribution and the level distribution, which can achieve a mutually reinforcing effect through the learning of the two, which can not only improve the accuracy of depression score prediction, but also improve the accuracy of depression level diagnosis.

[0031] The present invention discloses a facial depression recognition method based on joint label distribution learning, which introduces two label distributions: depression score distribution and level distribution, to describe the depression information of a face image, and jointly learns the depression score distribution and depression level distribution during the training process. Compared with the existing technical solutions, the introduction of label distribution in the present invention can solve the uncertainty problem of single-label BDI-II score annotation and alleviate the problem of insufficient training data; in addition, through joint label distribution learning, both depression scores and depression levels are considered during the training process, thereby improving the performance of depression diagnosis.

[0032] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A facial depression recognition method based on joint label distribution learning, characterized in that: The following steps are involved: S1. Depression score distribution generation: facial images are used as training data, and each facial image is annotated with a single-label BDI-II value as its corresponding descriptive degree. The sum of all descriptive degrees is equal to 1, which is used to represent the severity of depression of the face. The BDI-II score ranges from 0 to 63. The depression score distribution of each facial image is generated through Gaussian distribution; S2. Depression score distribution learning: Design a 68-dimensional deep network model, where the first 64 dimensions are used to learn the depression score distribution. Each facial image includes learning the true depression score and the score in the neighborhood. S3. Depression level distribution learning: The last 4 bits of the 68-dimensional output deep network model are used to learn the depression level distribution. The depression level distribution includes the descriptive degree of the real depression level and the descriptive degrees of other levels. S4. Joint labeled distribution learning: Design a multi-task joint labeled distribution learning framework to jointly learn the depression score distribution and depression level distribution.

2. A facial depression recognition method based on joint label distribution learning according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11. Assume that the training data is , ,in Indicates facial images, For the The single-label BDI-II depression score corresponding to the facial image; facial images, through Gaussian distribution Generate the corresponding depression score distribution, where is the depression score, is the standard deviation which is usually set to 1. is the mean, is the standard deviation; S12, through the formula Normalize, where is the normalized parameter; depression score The corresponding descriptive degrees constitute a 64-dimensional label distribution = ,in Depression score for The descriptive degree, arrive Indicates all The depression score distribution of each facial image is generated by taking the value of 3. The facial depression recognition method based on joint label distribution learning according to claim 2, characterized in that: The step S2 specifically includes the following steps: S21. Design a deep network model with 68-dimensional output, where the first 64 dimensions arrive Used to learn the distribution of depression scores. After the Softmax function, the j-th dimension output is: ; The predicted depression score distribution is ; S22. Design two loss functions to learn the distribution of depression scores. First, use KL divergence to design the first loss function. Learn the overall depression score distribution; S23. The predicted depression score is equal to the mean of the predicted depression score distribution, which is equal to , so that it is close to the actual BDI-II depression score, and the second loss function is designed by the absolute value To learn the true depression score.

4. The facial depression recognition method based on joint label distribution learning according to claim 3 is characterized in that: The step S3 specifically comprises the following steps: S31, transforming the predicted depression score distribution into the predicted depression grade distribution: ,in For depression level, express The corresponding depression score interval, that is, the predicted descriptive degree of the score within the interval and the predicted descriptive degree of the corresponding level, obtains the mapped predicted depression level distribution ; Use KL divergence to design the third loss function To learn the depression level distribution; S32, the last 4 dimensions of the deep network model arrive Used to learn the depression level distribution, after adding the Softmax function The prediction descriptiveness of the dimension is equal to ; Get the predicted depression level distribution ; Design KL divergence and design the fourth loss function Used to learn the distribution of depression levels; S33, for , based on depression scores Design the fifth loss function ,in is a constant, is the real depression level; if the real depression level When the prediction descriptiveness of is higher than that of other levels, the fifth loss is 0, otherwise a positive penalty is generated.

5. The facial depression recognition method based on joint label distribution learning according to claim 4, characterized in that: The step S4 specifically includes: Design the sixth loss function To train the 68-dimensional output deep network model, wherein the deep network model is trained by a hybrid multi-head cross-attention network architecture, , .

6. The facial depression recognition method based on joint label distribution learning according to claim 5, characterized in that: The step S4 also includes: the deep network model is also trained by an existing convolutional neural network or a Transformer model.

Images