Depression recognition system based on facial features

Abstract

The application discloses a depression recognition system based on facial features; wherein, the system comprises: an acquisition module configured to acquire facial expression images of a subject; a preprocessing module configured to perform face detection on the collected facial expression images to obtain face region images; perform face alignment processing on the detected face region images to obtain aligned face region images; and perform feature extraction on the aligned face region images to extract several kinds of face geometric features; a feature fusion module configured to fuse all the face geometric features to obtain fused face geometric features; and a depression recognition module configured to input the face geometric features into a trained depression recognition neural network model to output a final depression recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of depression recognition, in particular to a depression recognition system based on facial features. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Facial expression is the most abundant part in the process of human communication, and it reflects the inner emotion. Studies have shown that there is a significant difference between the facial mimicry ability of patients with depression and normal people. Currently, doctors mainly judge whether patients have depression and the severity of depression through different scales and clinical interviews. However, this depends on the subjective experience of doctors and the cognitive ability of patients, and has a certain degree of limitation.

[0004] With the development and improvement of computer vision technology, it has become possible to assist doctors in recognizing or early screening of depression with the help of computer vision technology. Therefore, how to use facial features to efficiently and accurately recognize depression has become a problem to be solved in the industry. SUMMARY

[0005] In order to solve the problems of the prior art, the present application provides a depression recognition system based on facial features; the method extracts facial features from the facial expression voluntary mimicry image of the patient with depression, designs a depression recognition neural network model with dynamic attention, and trains and evaluates it.

[0006] The depression recognition system based on facial features comprises:

[0007] An acquisition module configured to acquire facial expression images of a subject;

[0008] A preprocessing module configured to perform face detection on the collected facial expression images to obtain face region images, perform face alignment processing on the detected face region images to obtain aligned face region images, and perform feature extraction on the aligned face region images to extract several kinds of face geometric features;

[0009] A feature fusion module configured to fuse all face geometric features to obtain fused face geometric features;

[0010] A depression recognition module configured to input the face geometric features into the trained depression recognition neural network model to output the final depression recognition result.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] (1) The application extracts facial geometric features such as facial action units, eye gaze estimation and two-dimensional and three-dimensional key point positions according to the decline of facial mimicry ability of patients with depression, and constructs an efficient depression recognition method, thereby providing a convenient and reliable screening means for early recognition and auxiliary clinical diagnosis and treatment of depression.

[0013] (2) The application is based on a residual neural network, and a double-layer dynamic attention mechanism is added to construct a depression recognition neural network model based on facial features. The positional encoding generator (PEG) can generate implicit position encoding to provide position information for the weight change module, the weight change module pays attention to important features, can effectively overcome the training bottleneck, dynamically adjusts the input of the model, effectively avoids the interference of irrelevant features on depression recognition, and improves the efficiency of depression recognition. The recognition accuracy of the depression test set of the application is as high as 85% or more. BRIEF DESCRIPTION OF DRAWINGS

[0014] The drawings accompanying the specification of the application form a part of the application and serve to further understand the application. The schematic embodiments of the application and the description thereof are used to explain the application and do not constitute an improper limitation on the application.

[0015] Figure 1 The method flowchart of example one;

[0016] Figure 2 The confusion matrix of example one;

[0017] Figure 3 The internal structure diagram of the depression recognition neural network model of example one. DETAILED DESCRIPTION

[0018] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0019] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0020] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0021] All data acquisition of the embodiments is based on the compliance with laws and regulations and user consent, and the legal application of data.

[0022] Embodiment one

[0023] The embodiment provides a depression recognition system based on facial features;

[0024] As shown in Figure 1 The depression recognition system based on facial features comprises:

[0025] An acquisition module configured to acquire facial expression images of a subject;

[0026] A preprocessing module configured to perform face detection on the collected facial expression images to obtain face region images, perform face alignment processing on the detected face region images to obtain aligned face region images, and perform feature extraction on the aligned face region images to extract a plurality of face geometric features;

[0027] A feature fusion module configured to perform feature fusion on all the face geometric features to obtain fused face geometric features;

[0028] A depression recognition module configured to input the face geometric features into a trained depression recognition neural network model to output a final depression recognition result.

[0029] Further, the subject must sit in front of the camera, face the lens, and lean against a green curtain under the guidance of a doctor to mimic the static expression image generated by the computer. When the subject's facial expression is stable, the fixed camera is used to take a front view of the subject, and the image is saved.

[0030] Further, the face detection on the collected facial expression images to obtain face region images specifically comprises:

[0031] The collected facial expression images are converted into grayscale images, an OpenCV library is used to load a Haar cascade classifier to perform face detection, and the part containing a complete face in the image is cropped into a square image of 224x224 (pixels).

[0032] Further, the face alignment processing on the detected face region images to obtain aligned face region images specifically comprises:

[0033] The face alignment operation is implemented using a model provided by the 68-point key point information detection of the Dlib library.

[0034] Further, the aligned face region image is subjected to feature extraction, and several face geometric features are extracted, specifically including:

[0035] The open source tool OpenFace 2.0 is used to perform facial feature extraction on the facial expression image, obtain several facial geometric features, and store them in a csv file; wherein the several facial geometric features include: facial action units, eye gaze estimation, two-dimensional key point positions and three-dimensional key point positions.

[0036] Further, all the face geometric features are subjected to feature fusion to obtain fused face geometric features, and the feature fusion mode adopts serial fusion.

[0037] Further, as shown in the figure, Figure 3 the face geometric features are input into the trained depression recognition neural network model, and the final depression recognition result is output, wherein the specific network structure of the trained depression recognition neural network model includes:

[0038] an input layer, a first convolutional block, a first compression excitation module SE1 (Squeeze-and-Excitation, SE), a second convolutional block, a second compression excitation module SE2, a third convolutional block, a third compression excitation module SE2, a fourth convolutional block, a fourth compression excitation module SE2, a fifth convolutional block, a fifth compression excitation module SE2, a positional encoding generator PEG (Positional Encoding Generator), and an average pooling layer connected in sequence, the output end of the average pooling layer is connected with the input end of the weight change module and the input end of the full connection layer D1 respectively, and the output end of the weight change module is also connected with the input end of the input layer.

[0039] Further, the trained depression recognition neural network model selects a residual neural network ResNet-50 as the backbone network.

[0040] wherein the first convolutional block includes a convolutional layer conv1;

[0041] wherein the second convolutional block includes a convolutional layer conv2_1, a convolutional layer conv2_2 and a convolutional layer conv2_3 connected in sequence;

[0042] wherein the third convolutional block includes a convolutional layer conv3_1, a convolutional layer conv3_2, a convolutional layer conv3_3 and a convolutional layer conv3_4 connected in sequence;

[0043] The fourth convolutional block comprises, in sequence, a convolutional layer conv4_1, a convolutional layer conv4_2, a convolutional layer conv4_3, a convolutional layer conv4_4, a convolutional layer conv4_5 and a convolutional layer conv4_6.

[0044] The fifth convolutional block comprises, in sequence, a convolutional layer conv5_1, a convolutional layer conv5_2 and a convolutional layer conv5_3.

[0045] It should be understood that the ResNet-50 is stacked by 5 group convolutional blocks, and the output channel is continuously increased with the stacking of the convolutional blocks. Assuming that the input of the original data is X, X is composed of a batch size, a step length and a feature length. In order to enable the original data to be subjected to convolution calculation, the dimension is first expanded. At this time, the input X of the model input may be defined as X input =[BS, 1, 1, FL], wherein BS is the batch size, and FL is the feature length.

[0046] Further, the first compression excitation module SE1, the second compression excitation module SE2, the third compression excitation module SE3, the fourth compression excitation module SE4 and the fifth compression excitation module SE5 have consistent internal structures; the first compression excitation module SE1 comprises, in sequence, a global average pooling layer, a fully connected layer c1 and a fully connected layer c2; the global average pooling layer is used as a compression part to compress the features; the fully connected layer c1 and the fully connected layer c2 are used as excitation parts, the fully connected layer c1 is used to compress the features, and the fully connected layer c2 is used to restore the features.

[0047] It should be understood that the addition of the SE module after each convolutional block can obtain important channels and inner-layer attention. The first compression excitation module SE1, the second compression excitation module SE2, the third compression excitation module SE3, the fourth compression excitation module SE4 and the fifth compression excitation module SE5 each comprise two parts of compression and excitation. The compression part is a global average pooling layer, and the features are compressed into a 1x1xC vector, wherein C is the number of channels. The excitation part is composed of two fully connected layers, and the features are compressed by setting a scaling parameter R. The first fully connected layer compresses the channels, and the second fully connected layer restores the channels. The first fully connected layer is activated by a ReLU function, and the second fully connected layer is activated by a Sigmoid function.

[0048] Further, the first compression excitation module SE1, the second compression excitation module SE2, the third compression excitation module SE3, the fourth compression excitation module SE4 and the fifth compression excitation module SE5 have consistent internal working principles; the formula of the first compression excitation module SE1 is as follows:

[0049] Squeeze = Avgpool(X i );

[0050] Excitaion = Sigmoid(FC1(ReLU(FC2(Squeeze))));

[0051] SE = X i ⊙Excitaion;

[0052] wherein, X i represents the output feature of the i-th convolutional block, ⊙ represents matrix point multiplication, FC1 and FC2 represent the first and second fully connected layers respectively.

[0053] Further, the position encoding generator PEG (Positional Encoding Generator) generates implicit position encoding to provide position information for the weight change module. The position encoding generator PEG is placed after the fifth compression excitation module SE5 and before the average pooling layer.

[0054] In order to adapt the input of the position encoding generator PEG and keep the number of channels unchanged, the last two dimensions of the data are combined into one dimension. At this time, the feature size of the third dimension is the same as the size of the input X input , both of which are 1.

[0055] After completing the operation of the position encoding generator PEG, the dimension is expanded again and the average pooling operation is performed.

[0056] Further, the weight change module comprises a fully connected layer F1 and a Sigmoid activation function layer connected in sequence; the fully connected layer F1 is used to reduce the dimension of the output feature after the average pooling to the same length FL as the input feature; the Sigmoid activation function layer is used to compress the output of the fully connected layer F1 to between 0 and 1.

[0057] Further, the process of the weight change module is as follows:

[0058] First, determine the current iteration period epoch;

[0059] If the current iteration period epoch is the first iteration period, the input feature X input is pointwise multiplied by a matrix w1 of the same size, which is all 1, and the size remains unchanged.

[0060] If the current iteration period epoch is a multiple of the replacement parameter n, the weight is updated w t , and the input feature is pointwise multiplied by the weight.

[0061] If the current iteration cycle epoch is not a multiple of the replacement parameter n, the weight is the weight w of the last epoch t-1 The input features are multiplied point by point with them.

[0062] Further, the weight change module has the following formula:

[0063]

[0064] It should be understood that the weight change module provides outer attention for the depression recognition neural network model, dynamically adjusts the data input according to the importance of the features, and improves the performance of the model. In the model training process, the values of unimportant features are continuously reduced under the action of the Sigmoid function, and important features are preserved as much as possible.

[0065] Further, the face geometry features are input into the trained depression recognition neural network model to output the final depression recognition result, wherein the specific training process of the trained depression recognition neural network model includes:

[0066] Constructing a training set and a test set;

[0067] Inputting the training set into the trained depression recognition neural network model to train the model, stopping the training when the loss function value of the model no longer decreases, and obtaining a preliminarily trained depression recognition neural network model;

[0068] Inputting the test set into the preliminarily trained depression recognition neural network model for testing, stopping the testing when the accuracy is higher than a set threshold, and obtaining a finally trained depression recognition neural network model; otherwise, replacing the training set and retraining and testing.

[0069] Further, the training of the model is to input the training set into the depression recognition neural network model, set the batchsize to 64, the learning rate to 1e-4, the epoch to 60, and the optimizer to Adam. The loss function uses the negative log likelihood loss function (Negative Log Likelihood, NLL), and the formula is:

[0070]

[0071] Wherein, N represents the number of categories; k represents the category number; y k represents the real label; p k represents the output of the softmax classifier. In the process of model training, the model continuously iterates to optimize the parameters of the model.

[0072] Furthermore, the test set is input into the pre-trained depression recognition neural network model for testing, and the accuracy evaluation metric is used to make predictions on the test set.

[0073] Accuracy refers to the proportion of samples that are correctly predicted out of all samples.

[0074] The higher the accuracy, the better the model performance. The formula for the accuracy evaluation metric is as follows:

[0075]

[0076] Wherein, TP represents the correctly predicted case group, TN represents the correctly predicted control group, FP represents the incorrectly predicted control group, and FN represents the incorrectly predicted case group.

[0077] Further, the construction of the training and testing sets specifically includes: constructing a voluntary facial expression imitation dataset for patients with depression, specifically by conducting a voluntary facial expression imitation experiment and collecting facial expression imitation images from 82 cases in the case group and 82 cases in the normal group. The subjects in the voluntary facial expression imitation experiment were between 18 and 65 years old. Before the experiment, all subjects were required to complete an assessment plan, which included personal information such as health history, age, gender, education status, and employment status. Subjects in the case group met the screening criteria of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) and had no other mental disorders. Subjects in the normal group had HAMD-24 scores below 8 and no history of mental illness. In the voluntary facial expression imitation experiment, subjects sat in front of a camera, facing the lens, with their backs against a green screen, and imitated computer-generated static facial expressions under the guidance of a doctor. When the subjects' facial expressions stabilized, a fixed camera was used to take frontal photos of the subjects, and these images were saved. The images of the case group and the normal group were labeled, with the case group labeled as 0 and the normal group labeled as 1.

[0078] The dataset was randomly partitioned, with 80% used as the training set and 20% as the test set.

[0079] like Figure 2 As shown in the confusion matrix, the test set contains a total of 164 samples, with 82 cases and 82 healthy individuals. Among them, 61 cases were predicted as cases and 21 as healthy individuals; 79 healthy individuals were predicted as healthy individuals and 3 as cases. The accuracy shown by this confusion matrix is ​​(61+79) / 164 = 85%.

[0080] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A depression recognition system based on facial features, characterized in that, Comprise: An acquisition module configured to acquire a facial expression image of a subject; A preprocessing module configured to perform face detection on the collected facial expression image to obtain a face region image, perform face alignment processing on the detected face region image to obtain an aligned face region image, and perform feature extraction on the aligned face region image to extract a plurality of face geometric features; A feature fusion module configured to fuse all the face geometric features to obtain fused face geometric features; A depression recognition module configured to input the face geometric features into a trained depression recognition neural network model to output a final depression recognition result; wherein the specific network structure of the trained depression recognition neural network model comprises: An input layer, a first convolutional block, a first compression excitation module SE1, a second convolutional block, a second compression excitation module SE2, a third convolutional block, a third compression excitation module SE2, a fourth convolutional block, a fourth compression excitation module SE2, a fifth convolutional block, a fifth compression excitation module SE2, a position encoding generator PEG, and an average pooling layer connected in sequence, the output end of the average pooling layer is connected with the input end of a weight change module and the input end of a full connection layer D1, and the output end of the weight change module is also connected with the input end of the input layer.

2. The facial feature based depression recognition system as claimed in claim 1, wherein The face detection on the collected facial expression image to obtain a face region image specifically comprises: Convert the collected facial expression image into a grayscale image, use the OpenCV library to load the Haar cascade classifier for face detection, and crop the part containing the complete face in the image into a 224x224 square image.

3. The facial feature based depression recognition system as claimed in claim 1, wherein The face alignment processing on the detected face region image to obtain an aligned face region image specifically comprises: Use the model provided by the Dlib library to realize the face alignment operation.

4. The facial feature based depression recognition system as claimed in claim 1, wherein The feature extraction on the aligned face region image to extract a plurality of face geometric features specifically comprises: Use the open source tool OpenFace 2.0 to perform face feature extraction on the facial expression image to obtain a plurality of face geometric features and store them in a csv file; wherein the plurality of face geometric features include: facial action units, eye gaze estimation, two-dimensional key point positions, and three-dimensional key point positions.

5. The facial feature based depression recognition system as claimed in claim 1, wherein The feature fusion of all the face geometric features to obtain fused face geometric features adopts a series fusion method.

6. The facial feature based depression recognition system as claimed in claim 1, wherein Wherein, The first convolutional block comprises a convolutional layer conv1; Wherein, the second convolutional block comprises a convolutional layer conv2_1, a convolutional layer conv2_2, and a convolutional layer conv2_3 connected in sequence; Wherein, the third convolutional block comprises a convolutional layer conv3_1, a convolutional layer conv3_2, a convolutional layer conv3_3, and a convolutional layer conv3_4 connected in sequence; The fourth convolutional block comprises, in sequence, a convolutional layer conv4_1, a convolutional layer conv4_2, a convolutional layer conv4_3, a convolutional layer conv4_4, a convolutional layer conv4_5 and a convolutional layer conv4_6. The fifth convolutional block comprises, in sequence, a convolutional layer conv5_1, a convolutional layer conv5_2 and a convolutional layer conv5_3.

7. The facial feature based depression recognition system as claimed in claim 1, wherein, The first compression excitation module SE1, the second compression excitation module SE2, the third compression excitation module SE3, the fourth compression excitation module SE4 and the fifth compression excitation module SE5 have the same internal structure. The first compression excitation module SE1 comprises, in sequence, a global average pooling layer, a fully connected layer c1 and a fully connected layer c2. The global average pooling layer is used to compress features. The fully connected layer c1 is used to compress features, and the fully connected layer c2 is used to restore features.

8. The facial feature based depression recognition system as claimed in claim 1, wherein, The weight change module comprises, in sequence, a fully connected layer F1 and a Sigmoid activation function layer. The fully connected layer F1 is used to reduce the dimension of the output features after average pooling to the same length as the input features. The Sigmoid activation function layer is used to compress the output of the fully connected layer F1 to between 0 and 1. The process of the weight change module is as follows: First, determine the current iteration period epoch. If the current iteration is the first epoch, then the input features a matrix of the same size with all ones point-wise multiplication, same size If the current iteration period, epoch, is a multiple of the replacement parameter n, then the weight update , the input features are multiplied point-wise with the weights; If the current iteration period, epoch, is not a multiple of the replacement parameter, n, then the weights are the weights of the previous epoch The input features are multiplied point-wise with them.

9. The facial feature based depression recognition system as claimed in claim 1, wherein, The face geometry features are input into the trained depression recognition neural network model to output the final depression recognition result. The specific training process of the trained depression recognition neural network model comprises: Construct a training set and a test set; The training set is input into the trained depression recognition neural network model to train the model. When the loss function value of the model no longer decreases, the training is stopped, and a preliminarily trained depression recognition neural network model is obtained. The test set is input into the preliminarily trained depression recognition neural network model for testing. When the accuracy is higher than a set threshold, the testing is stopped, and a finally trained depression recognition neural network model is obtained. Otherwise, the training set is replaced, and the training and testing are performed again.

Images