A facial video-based automatic depression recognition method

Abstract

The application relates to a facial video-based automatic depression recognition method, which comprises the following steps: obtaining facial video data to be recognized and performing pretreatment to obtain facial video sub-samples V c ; a deep learning model LDSML for combining depression label distribution and label measurement is constructed; the sub-samples V c are input into the LDSML deep learning model for training; a Softmax-T activation function with a temperature coefficient is used at the output end of the model to obtain a predicted depression label distribution p, and the mathematical expectation E of the label distribution is calculated; in the prediction stage, the mathematical expectation E of the predicted depression label distribution P is used to represent the final depression prediction result. Without increasing the training data set, the feature capturing capability of the deep learning model for facial depression video data is improved. The more detailed label distribution can effectively enable the model to learn the hidden label semantic information, and effectively improve the current situation of the lack of facial depression video data.

Description

Technical Field

[0001] This invention relates to the field of computer recognition technology, specifically to an automatic method for identifying depression based on facial video. Background Technology

[0002] Depression is a very serious mental illness characterized by high incidence, frequent relapses, and a trend towards affecting younger people. However, there are currently no unique and effective clinical features for depression. In most cases, the diagnosis of depression requires methods such as structured clinical interviews and depression scales, and is highly dependent on the experience of clinicians. The entire process lacks objective and effective evidence and is time-consuming. Therefore, there is an urgent need for an objective, effective, and rapid method for identifying depression.

[0003] Because facial expressions of people with depression often exhibit clear features related to sadness, researchers in the field of affective computing have attempted to leverage knowledge from psychology and computer vision to automatically identify depression through facial visual features. Early methods for facial depression recognition required manual extraction of facial features, such as using Local Phase Quantization (LPQ) to obtain visual features and Support Vector Regression (SVR) for regression prediction. This manual feature extraction method required careful design and relied on prior knowledge from experts.

[0004] Compared to manual feature extraction methods, deep learning possesses powerful feature extraction capabilities. Some deep learning models have been used to capture features related to depression from facial videos, thereby enabling depression identification. The effectiveness of deep learning methods relies on reliable datasets. However, depression involves patient privacy, making it difficult to obtain reliable, large-scale datasets for model training in this field.

[0005] Currently, some deep learning methods use deep label distribution learning to predict depression label distribution, obtaining the predicted depression score by calculating the mathematical expectation of the label distribution. However, they neglect to explore the form of facial depression label distribution and suppress noise label information. Given these two problems with existing research, capturing more facial features from limited facial depression datasets and exploring facial differences among patients with depression has become a key focus for future research. Summary of the Invention

[0006] To address the problems existing in the prior art, the purpose of this invention is to provide an automatic depression recognition method based on facial video. This method can improve the feature extraction capability of the model and enhance the accuracy and robustness of facial depression recognition without increasing the dataset.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An automatic depression identification method based on facial video includes the following steps:

[0009] Acquire and preprocess the facial video data to be identified to obtain facial video subsamples V. c ;

[0010] Construct a deep learning model LDSML that jointly learns the distribution and metric of depression labels;

[0011] Subsample V c Input into the LDSML deep learning model for training;

[0012] The predicted depression label distribution p is obtained by using the Softmax-T activation function with temperature coefficient at the model output, and the expected value E of the label distribution is calculated.

[0013] In the prediction phase, the expected value E of the predicted depression label distribution P is used to represent the final depression prediction result.

[0014] Furthermore, the deep learning model LDSML is built on the I3D-Resnet50 neural network model. This network uses four residual modules to extract spatiotemporal features and finally outputs 128-dimensional label distribution features through a fully connected layer.

[0015] Furthermore, label distribution learning is used to solve facial depression recognition. The deep learning model outputs a depression label distribution through Softmax-T and adopts higher label distribution segmentation accuracy so that the model can capture the implicit label information.

[0016] Furthermore, by adjusting the temperature coefficient t of the Softmax-T activation function, redundant noise label information in the depression label distribution p is reduced, encouraging the model to focus more on the true depression labels during training. The formula is as follows: Where t is the temperature coefficient used to adjust the distribution of output tags.

[0017] Furthermore, during the model training phase, the subsample V is calculated. c The loss function between the expected depression score E and the true label l is calculated as follows:

[0018] Where J represents the total loss function of the LDSML framework, J er J represents the expected regression loss function. bsm Let J represent the expected supervised loss function for the batch of samples, where N represents the number of batches of samples input simultaneously, and α represents the J value. bsm The weight.

[0019] Furthermore, during the training phase, the expected regression loss function J is used. er The formula for minimizing the distance between the expected value E of the depression label distribution P and the true label l is as follows: Where M represents the number of samples.

[0020] Furthermore, during the training phase, the expected supervised loss J is used with batch samples. bsm The formula for minimizing the difference between the expected distance and the true distance between samples in each batch is as follows:

[0021] D ij,E =E i -E j ;

[0022] D ij,l =l i -l j ;

[0023] J sm (x i ,x j )=||D ij,E -D ij,l ||2;

[0024] Where D represents the distance between the labels or expected depression scores of the two samples.

[0025] Furthermore, the expected supervision metric J is used to measure the loss. bsm The distance label information of all samples in each training batch is fully captured, and its calculation formula is as follows:

[0026]

[0027] Where P(m) represents the number of sample pairs.

[0028] Furthermore, the loss function J bsm By fully utilizing the distance label information of all subsamples in each batch of training samples, we can explore the correlation information between sequential labels, thereby capturing more facial difference features among different patients with depression.

[0029] In summary, the present invention has the following advantages:

[0030] By employing deep label distribution learning and supervised metric learning, the distribution information of depression labels and the differences between samples were effectively learned. This improved the feature capture ability of the deep learning model for facial depression video data without increasing the training dataset. Softmax-T with a temperature coefficient effectively suppressed noisy label distribution information, prompting the model to focus more on realistic labels. A more refined label distribution allows the model to effectively learn implicit label semantic information, significantly improving the current situation of scarce facial depression video data. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the data preprocessing of the present invention.

[0032] Figure 2 This is a flowchart of the overall identification method of the present invention and a schematic diagram of the deep learning model LDSML.

[0033] Figure 3 This is a schematic diagram illustrating the batch sample expected supervision metric loss calculation strategy and its effect according to the present invention.

[0034] Figure 4 and Figure 5 This is a comparison diagram of the effects of the present invention and existing methods in identifying depression.

[0035] Figure 6 and Figure 7 This is a schematic diagram illustrating the depression recognition effect of each module in this invention. Detailed Implementation

[0036] The present invention will now be described in further detail.

[0037] An automatic depression identification method based on facial video includes the following steps:

[0038] Acquire and preprocess the facial video data to be identified to obtain facial video subsamples V. c ;

[0039] Construct a deep learning model LDSML that jointly learns the distribution and metric of depression labels;

[0040] Subsample V c Input into the LDSML deep learning model for training;

[0041] The predicted depression label distribution p is obtained by using the Softmax-T activation function with temperature coefficient at the model output, and the expected value E of the label distribution is calculated.

[0042] In the prediction phase, the expected value E of the predicted depression label distribution P is used to represent the final depression prediction result.

[0043] By employing deep label distribution learning and supervised metric learning, the distribution information of depression labels and the differences between samples were effectively learned. This improved the feature capture ability of the deep learning model for facial depression video data without increasing the training dataset. Softmax-T with a temperature coefficient effectively suppressed noisy label distribution information, prompting the model to focus more on realistic labels. A more refined label distribution allows the model to effectively learn implicit label semantic information, significantly improving the current situation of scarce facial depression video data.

[0044] The specific steps are as follows:

[0045] Step 1: Collect facial video data in a real-world environment;

[0046] Step 2: Preprocess the video data to obtain facial video subsamples V for input to the model. c ;

[0047] Step 3: Construct a deep learning framework, LDSML, capable of jointly learning the distribution and metric of depression labels;

[0048] Step 4: Subsample V c Input into the LDSML deep learning model framework;

[0049] Step 5: Use the Softmax-T activation function with temperature coefficient at the output of the deep learning model to obtain the predicted depression label distribution P, and calculate the expected value E of the label distribution.

[0050] Step 6: During the model training phase, calculate the loss between the expected depression score E of the subsample and the true label l, using the following loss function: J er It is the expected regression loss function, J bsm It is the expected supervised loss function for batch samples;

[0051] Step 7: During the model training phase, calculate the expected regression loss function J. er The formula for minimizing the difference between the expected value of the label distribution and the true label is as follows: Where M represents the number of samples;

[0052] Step 8: During the training phase, use the batch sample expected supervised measurement loss J. bsm The goal is to minimize the difference between the expected distance and the true distance between samples in each batch. The formula for this loss function is:

[0053] D ij,E =E i -E j ;

[0054] D ij,l =l i -l j ;

[0055] J sm (x i ,x j )=||D ij,E -D ij,l ||2;

[0056]

[0057] Where P(m) represents the number of sample pairs.

[0058] Step 9: Prediction phase, using the expected value E of the depression label distribution to represent the final depression prediction result.

[0059] Figure 1 The overall flowchart of the automatic depression identification method and system based on facial video proposed in this invention is shown.

[0060] Specifically, steps 1 and 2 include using a stimulating material to induce facial movements in the subject, during which facial video data V of the subject is recorded. Data preprocessing is as follows: Figure 2 As shown, the face region is extracted from V and preprocessed using methods such as mirror flipping, data augmentation, and downsampling.

[0061] Steps 3, 4, 5, and 6 include constructing a deep learning model framework, LDSML, based on the I3D-Resnet50 deep neural network to jointly learn the distribution and metric of depression labels. This network uses four residual modules to extract spatiotemporal features and finally outputs 128-dimensional label distribution features through a fully connected layer. Figure 3 A schematic diagram of the LDSML model framework is shown. This framework uses a temperature-coefficient Softmax-T activation function to suppress label distribution noise and outputs a depression label distribution p, the formula of which is:

[0062]

[0063] Where t is a temperature coefficient used to adjust the distribution of output labels. The expected value E of p is calculated, and the total loss function J of the LDSML model is constructed. The loss between the expected depression score E and the true label l is calculated using the following formula:

[0064]

[0065] J er J represents the expected regression loss function. bsm Let J represent the expected supervised loss function for the batch of samples, where N represents the number of batches of samples input simultaneously, and α represents the J value. bsmThe weight.

[0066] Steps 7 and 8 include calculating J. er and J bsm During the training phase, the expected regression loss function J is used. er The loss function minimizes the difference between the expected value of the label distribution and the true label. The formula for this loss function is:

[0067]

[0068] Where M represents the number of input samples. The expected supervised loss J is used as the measure of subsamples. sm (x i ,x j The formula for minimizing the difference between the expected depression distance and the actual distance label of two samples is:

[0069] D ij,E =E i -E j ;

[0070] D ij,l =l i -l j ;

[0071] J sm (x i ,x j )=||D ij,E -D ij,l ||2;

[0072] Using batch samples, the expected supervised measure of loss J is used. bsm The general formula for fully capturing the distance label information of all samples in each training batch is:

[0073]

[0074] Where P(m) represents the number of sample pairs.

[0075] This invention presents three batch sample expected supervision metric loss J bsm The computational strategy, Figure 3 This diagram illustrates and demonstrates the effects of three computational strategies. For strategy 1, the input subsample V... b V b All subsamples are paired together to form sample pairs. Calculate J. ij,sm Its formula is:

[0076]

[0077] For strategy 2, the input subsample V b1 and V b2 Vb1 and V b2 Subsamples at the same position in the sample are paired up to form sample pairs. Calculate J. ij,sm Its formula is:

[0078]

[0079] For strategy 3, the input subsample V b1 and V b2 V b1 All subsamples with V b2 All subsamples form sample pairs, J sm (x i ,x j The formula remains unchanged.

[0080] There are two publicly available datasets for facial depression recognition: AVEC2013 and AVEC2014. Extensive experiments were conducted on these two datasets, and the method and system of this invention were evaluated using two average metrics: root mean square error (RMSE) and mean absolute error (MAE).

[0081] Table 1 lists the correspondence between data labels and depression levels, while Tables 2 and 3 show the experimental results of the LDSML framework proposed in this invention and existing methods on the AVEC2013 and AVEC2014 datasets. Figure 4 and Figure 5 This table compares the performance of the LDSML model proposed in this invention with existing methods for depression identification. Tables 3 and 4 clearly show that the LDSML model proposed in this invention outperforms most existing deep learning-based depression identification methods, including RNN-C3D, C3D (Global+Local), DepressNet, LDLR, Bi-LSTM, and DJ-LDML. On the AVEC2013 dataset, the RMSE result is better than all existing methods in the table, and the MAE result is second only to DepressNet. On the AVEC2014 dataset, both the RMSE and MAE values ​​are better than all existing methods in the table.

[0082] Tables 4 and 5 present the ablation experiment data, listing the performance improvement effects of each module in the LDSML framework proposed in this invention. For example... Figure 6 and Figure 7As shown, the comparison results between DR and other label distribution learning methods in the table demonstrate that, under the premise of using the same training data and neural network, the label distribution learning method outperforms the classic deep regression. The comparison results of LDL-64(Softmax) and LDL-64(Softmax-T), and LDSML-128(Softmax+S3) and LDSML-128(Softmax-T+S3) show that using the Softmax-T activation function at the output of the facial depression recognition model is superior to the traditional Softmax activation function. The comparison results of LDL-128(Softmax-T) and LDSML-128(Softmax-T+S3) show that the model performance is significantly improved after introducing the batch sample expectation supervised loss function proposed in this invention.

[0083] This invention presents an automatic facial depression recognition system based on the LDRSM deep learning model framework, which jointly learns the distribution and metric of depression labels. While learning the facial depression label distribution, a batch sample expectation supervised loss function is introduced to capture the differential features between different samples. This improves the facial feature capture capability of the deep learning model without increasing the training dataset, thereby enhancing the accuracy and robustness of facial depression recognition.

[0084] Table 1

[0085]

[0086]

[0087] Table 2

[0088]

[0089] Table 3

[0090]

[0091] Table 4

[0092]

[0093] Table 5

[0094]

[0095] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for automatic identification of depression based on facial video, characterized in that: Includes the following steps, Acquire and preprocess the facial video data to be identified to obtain facial video subsamples V. c ; Construct a deep learning model LDSML that jointly learns the distribution and metric of depression labels; Subsample V c Input into the LDSML deep learning model for training; The predicted depression label distribution p is obtained by using the Softmax-T activation function with temperature coefficient at the model output, and the expected value E of the label distribution is calculated. In the prediction phase, the expected value E of the predicted depression label distribution p is used to represent the final depression prediction result; During the model training phase, the subsample V is calculated. c The loss function between the expected depression score E and the true label l is calculated as follows: Where J represents the total loss function of the LDSML framework, J er J represents the expected regression loss function. bsm Let J represent the expected supervised loss function for the batch of samples, where N represents the number of batches of samples input simultaneously, and α represents the J value. bsm The weights; During the training phase, the expected regression loss function J is used. er The formula for minimizing the distance between the expected value E of the depression label distribution p and the true label l is as follows: Where M represents the number of samples; Using batch samples, the expected supervised measure of loss J is used. bsm To minimize the difference between the expected distance and the true distance between samples in each batch, the loss J is calculated using the batch sample expected supervised metric. bsm The distance label information of all samples in each training batch is fully captured, and its calculation formula is as follows: D ij,E =And i -AND j ; D ij,l ＝l i -l j ； J sm (x i ,x j )=||D ij,E -D ij,l ||2; Where D represents the distance between the labels or expected depression scores of the two samples, and x i ,x j The subscripts i and j represent the indices of sample x; P(m) represents the number of sample pairs.

2. The automatic depression identification method based on facial video according to claim 1, characterized in that: The deep learning model LDSML is built on the I3D-Resnet50 neural network model. This network uses four residual modules to extract spatiotemporal features and finally outputs 128-dimensional label distribution features through a fully connected layer.

3. The automatic depression identification method based on facial video according to claim 1, characterized in that: Label distribution learning is used to solve facial depression recognition. The deep learning model outputs a depression label distribution through Softmax-T and adopts higher label distribution segmentation accuracy so that the model can capture the implicit label information.

4. The automatic depression identification method based on facial video according to claim 3, characterized in that: By adjusting the temperature coefficient t of the Softmax-T activation function, redundant noise label information in the depression label distribution p is reduced, encouraging the model to focus more on the true depression labels during training. The formula is as follows: Where t is the temperature coefficient used to adjust the distribution of output labels, and i and j are subscript indices, representing the original output value x of the model for the i-th and j-th categories. i and x j Q represents the dimension of vector p.

5. The automatic depression identification method based on facial video according to claim 1, characterized in that: Loss function J bsm By fully utilizing the distance label information of all subsamples in each batch of training samples, we can explore the correlation information between sequential labels, thereby capturing more facial difference features among different patients with depression.

Images