Face video-based depression detection method

Abstract

The invention discloses a depression detection method based on a face video, belongs to the technical field of depression detection, and can solve the problems of invasive equipment, long detection time, unreliable detection effect, high cost and possible physiological and psychological discomfort to a patient in the existing detection method. The method comprises the following steps: S1, determining a facial feature set of each subject according to a facial action video of each subject in a plurality of subjects; s2, training a classifier according to the facial feature set of all subjects and the depression degree label of each subject to obtain a classification model and an optimal feature subset; s3, the facial feature set of the to-be-tested person corresponds to the features of the optimal feature subset and then is input into the classification model, and a depression prediction result of the to-be-tested person is obtained. The method is used for detecting depression.

Description

Technical Field

[0001] The invention relates to a depression detection method based on facial video, belonging to the technical field of depression detection. Background Art

[0002] In the field of depression prediction, the more common research methods currently include prediction methods based on brain activity and prediction methods that judge emotions and expressions by analyzing visual and auditory signals.

[0003] The former usually involves the use of brain computer interfaces (BCI) and electroencephalograms (EEGs) to identify potential symptoms of depression by recording and analyzing changes in brain activity. However, these technologies are often invasive and may require subjects to wear bulky equipment and remain still during data collection, which limits their application in large-scale screening and daily monitoring.

[0004] The latter uses visual and auditory signals, such as facial expressions and voice features, to infer the emotional state of the subject. Facial expression analysis infers the emotional state of the subject by identifying changes in facial expressions such as smiling and frowning. Voice feature analysis evaluates emotional changes by analyzing parameters such as the pitch, rhythm and intensity of the voice. The advantages of these methods are that they are usually easier to implement and have less interference with the subject's daily activities. However, existing technologies have limitations in automation and accuracy, especially during sample collection and analysis, which may affect the accuracy of diagnosis due to improper operation or equipment limitations. Summary of the invention

[0005] The present invention provides a method for detecting depression based on facial video, which can solve the problems of existing detection methods, such as invasive equipment, long detection time, unreliable detection effect, high cost, and possible physical and psychological discomfort to patients.

[0006] The present invention provides a method for detecting depression based on facial video, the method comprising:

[0007] S1, determining a facial feature set for each subject according to a facial action video of each subject among a plurality of subjects;

[0008] S2, training a classifier based on the facial feature set of all subjects and the depression level label of each subject to obtain a classification model and an optimal feature subset;

[0009] S3, matching the facial feature set of the subject with the features of the optimal feature subset and inputting the matched features into the classification model to obtain the depression prediction result of the subject.

[0010] Optionally, the facial action video includes mouth action information, eye action information and expression action information completed according to preset requirements.

[0011] Optionally, the facial feature set includes eye movement angle features, eye movement response features, mouth movement time features, mouth movement response features, expression movement response features, facial key point features and global movement response features.

[0012] Optionally, S2 is specifically:

[0013] An evolutionary algorithm was used to select features from the facial feature sets of all subjects, and the classifier was trained in combination with the depression degree labels of all subjects to obtain the classification model and the optimal feature subset.

[0014] Optionally, the evolutionary algorithm performs search iterations based on Pareto sorting.

[0015] Optionally, the classifier is a random forest.

[0016] Optionally, the S1 specifically includes:

[0017] Obtain facial action videos and depression level labels of each subject among multiple subjects;

[0018] Feature extraction is performed on each facial action video to obtain a facial feature set for each subject.

[0019] Optionally, the feature extraction is performed on each facial action video to obtain a facial feature set of each subject, specifically:

[0020] A neural network algorithm is used to extract features from each facial action video to obtain a facial feature set for each subject.

[0021] Optionally, before S3, the method further includes:

[0022] A facial action video of the subject to be tested is collected, and features are extracted from the facial action video to obtain a facial feature set of the subject to be tested.

[0023] Optionally, S3 is specifically:

[0024] Features corresponding to the features of the optimal feature subset are selected from the facial feature set of the subject and input into the classification model to obtain the depression prediction result of the subject.

[0025] The beneficial effects that the present invention can produce include:

[0026] The facial video-based depression detection method provided by the present invention can improve the quality of samples by standardizing facial action video collection and annotation. Then, by performing feature extraction and feature selection on the facial action video, an optimal feature subset is obtained, and classifier training is performed during the feature selection process to obtain a classification model, which can improve the diagnostic bias caused by sample imbalance, shorten the detection time, and improve the detection effect.

[0027] The present invention adopts a non-invasive facial video analysis method to avoid physical contact with the subject's head and possible damage. This significantly improves the patient's comfort and the acceptability of the diagnosis. At the same time, the present invention uses an evolutionary algorithm based on Pareto sorting for feature and sample selection, effectively solves the problem of high-dimensional feature selection, balances the sample distribution, and improves the robustness of the classification model to unbalanced data. This helps to improve the diagnostic bias caused by sample imbalance in practical applications and enhances the generalization ability of the classification model. In addition, the present invention adopts a dynamic video analysis method that sets a standardized action group, which not only reduces the difficulty of analyzing facial expressions, movements and demeanor, but also improves the utilization rate of samples and the accuracy of diagnosis. This innovative method improves the depth and breadth of facial data analysis and solves the problems of imperfect facial visual perception standards and limited analysis methods in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a method for detecting depression based on facial video provided by an embodiment of the present invention;

[0029] Figure 2 A program framework diagram of the PEL algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below in conjunction with embodiments, but the present invention is not limited to these embodiments.

[0031] The embodiment of the present invention provides a method for detecting depression based on facial video. Figure 1 As shown, the method includes:

[0032] S1. Determine a facial feature set for each subject according to a facial action video of each subject among a plurality of subjects.

[0033] Among them, the facial action video includes mouth action information, eye action information and expression action information completed according to preset requirements.

[0034] S1 specifically includes:

[0035] (1) Obtain facial action videos and depression severity labels for each subject among multiple subjects.

[0036] The present invention designs a set of standardized facial action sets for subjects, which are used to study and evaluate facial muscle activities and their related physiological and psychological states. The action set contains three aspects: mouth action, eye action and expression action. Mouth action: including four parts: "open mouth", "stick out tongue", "move tongue left and right" and "close mouth". These actions are mainly used to evaluate the flexibility of the subject's mouth muscles and the ability to execute instructions. By analyzing these actions, data on the control and coordination ability of the subject's mouth muscles can be obtained, which is of great significance for the diagnosis of neurological diseases. Eye action: consisting of three actions: "open eyes", "look left" and "look right", which are mainly used to evaluate the subject's gaze direction, pupil reaction and flexibility of eye-related muscles. These actions can reveal the subject's eye movement ability and eye nerve function status, and have reference value for the study of vision health and nervous system health. Expression action: including "make a smiling expression". Smiling not only involves the coordinated activities of a large number of facial muscles, but also reflects the emotional state of the subject. By analyzing the quality and naturalness of smiling expressions, the flexibility of facial muscles can be evaluated, and at the same time, a basis can be provided for the analysis of emotional and mental health conditions (especially the detection of depression tendency). The design of this standardized facial action set not only provides a reliable tool for the quantitative evaluation of facial muscle activity, but also provides important reference data for research in the fields of psychology and neuroscience.

[0037] The above facial action set is only an example. In actual applications, different facial action group designs can also be adopted, such as adding or reducing certain actions, or changing the execution order and duration of actions to meet different diagnostic needs.

[0038] In terms of facial action video collection and annotation, it is crucial to ensure the high quality of data. Using appropriate shooting equipment such as mobile phones and cameras, by collecting standardized facial action videos of subjects of different ages and genders, the diversity and representativeness of the data can be effectively improved. In order to cover the facial features of different age groups, subjects should be selected from children, teenagers, adults and the elderly, and there should be a sufficient number of subjects in each age group to ensure the balance of the data. At the same time, the gender ratio of the subjects should be close to half male and half female. In addition, by collecting under uniform lighting conditions, the impact of lighting changes on facial features can be reduced. The present invention uses soft natural light and standardized artificial light sources for video collection, and keeps the collection resolution of each sample the same and the collection time similar. Depression experts from tertiary hospitals conduct a comprehensive diagnosis of patients (rather than just facial action videos) to obtain accurate depression degree labels.

[0039] (2) Feature extraction is performed on each facial action video to obtain a facial feature set for each subject, which includes eye action angle features, eye action response features, mouth action time features, mouth action response features, expression action response features, facial key point features, and global action response features.

[0040] Specifically: a neural network algorithm is used to extract features from each facial action video to obtain a facial feature set for each subject.

[0041] The facial action set used in the present invention covers three main aspects: mouth action, eye action and expression action. Based on these action information in the facial action video, neural network is first used to extract features, extracting time-related facial action units, gaze direction and expression information, specifically including 18 action units, 68 facial key points and eye-related position and line of sight information. The present invention designs and uses the following seven features:

[0042] a) Eye movement angle features: The present invention describes the eye movement angles in the subject's facial movement video through 4-dimensional features, including the mean and variance of the eye gaze angles when the eyes look left and right (because the action may last for several frames rather than a single frame).

[0043] b) Eye movement response characteristics: The present invention describes the response time and response degree of the eye movements in the subject's facial movement video through 12-dimensional features, which are expressed as the response degree and time of the eye-related action units.

[0044] c) Mouth movement time characteristics: The present invention describes the time required for the subject's tongue to complete the "tongue sticking out" and "tongue moving left and right" actions and the ratio of the time to the total video duration through 2D features.

[0045] d) Mouth movement response characteristics: The present invention describes the response time and response degree of mouth-related movements in the subject's facial movement video through 18-dimensional features, which are expressed as the response degree and time of the mouth-related movement unit.

[0046] e) Expression and action response characteristics: The present invention describes the response time and response degree of related actions when smiling in the subject's facial action video through 34-dimensional features, which is expressed as the response degree and time of the action unit when smiling.

[0047] f) Facial key point features: The present invention describes the facial key point features of the subject under facial movements through 272-dimensional features, and extracts features considering the relative positions of the facial key points during characteristic movements.

[0048] g) Global action response features: The present invention uses 54-dimensional features to describe the global action unit response degree in the subject's facial action video and the action unit during the characteristic action to perform feature extraction.

[0049] It should be noted that the embodiments of the present invention do not limit the specific number of dimensions of each of the above features, and those skilled in the art can set it according to actual conditions. In addition, the present invention does not limit the method of feature extraction. For example, automatic feature learning technology based on deep learning can also be used to extract more abstract or more advanced feature representations.

[0050] S2. Train a classifier based on the facial feature set of all subjects and the depression degree label of each subject to obtain a classification model and an optimal feature subset.

[0051] Specifically, an evolutionary algorithm is used to select features from the facial feature sets of all subjects, and a classifier is trained based on the depression degree labels of all subjects to obtain a classification model and the optimal feature subset.

[0052] The evolutionary algorithm performs search iteration based on Pareto sorting, and the classifier can be a random forest (RF).

[0053] In practical applications, other types of optimization algorithms, such as genetic algorithms, simulated annealing algorithms, etc., can also be proposed for feature and sample selection to improve the performance and robustness of the classifier.

[0054] At the same time, in addition to the random forest, other types of classifiers may also be used, such as support vector machine (SVM), neural network or gradient boosting machine (GBM), to adapt to different data characteristics and classification requirements. The embodiment of the present invention does not limit this.

[0055] Although the present invention extracts 360-dimensional features such as facial expressions, movements, and eyes, there are the following problems in directly using all these features to establish a depression representation system:

[0056] 1) Different features have different feature dimensions. During the classifier learning process, it may focus too much on high-dimensional features and ignore low-dimensional features, which will lead to a decrease in the disease recognition accuracy of the classifier.

[0057] 2) The total feature dimension extracted is too high, which not only causes the classifier to be unable to fully learn important features, but also increases the training and testing costs of the classifier.

[0058] 3) Different features contribute differently to the identification of depression. Problems such as feature redundancy and feature duplication often occur in high-dimensional features, which not only affects the accuracy of machine learning identification of depression, but also causes the features most relevant to the disease to be unable to be fully learned.

[0059] In addition, since some samples are representative positive or negative samples of depression and some data are outliers, the samples participating in model training should also be selected.

[0060] In view of the above problems, the usual solution is to select feature sets and sample sets to retain low-dimensional, representative features and representative samples. However, the feature dimensions used in the present invention are high and the number of samples is large. Directly searching for the optimal feature and sample combination will result in NP problems. Therefore, the present invention proposes an evolutionary algorithm based on Pareto sorting (Pareto Evolutionary Learning, PEL) to select the collected samples and the extracted multiple features, retain the optimal samples and feature combinations, so as to improve the accuracy of depression detection.

[0061] The present invention uses random forest (RF) as the base classifier for the following reasons:

[0062] 1) RF is a tree ensemble model based on random sampling without replacement, which has the advantages of strong model interpretability, good machine learning performance, robustness to imbalanced data, and not easy to overfit to noise and outliers.

[0063] 2) RF training and testing are more efficient because they can be trained or tested in parallel to improve efficiency and reduce overhead.

[0064] 3) RF embeds an out-of-bag sample unbiased error estimation algorithm, which can provide an unbiased estimate of the classification error without providing an additional validation set or test set, which is helpful to verify the performance of the feature subset obtained by feature selection.

[0065] 4) RF embeds an algorithm for ranking feature importance, which can perform a preliminary feature importance ranking of the "purity" of all training features while training the classification model.

[0066] Therefore, the present invention proposes to perform feature and sample selection based on PEL and RF.

[0067] Specifically, first let ,in Represents all features, yes The category label of the corresponding sample, is the total number of samples, is the total number of features. In addition, represent The corresponding samples are negative for depression. represent The corresponding sample was diagnosed with depression. Its characteristics The feature dimension is , so the total dimension of the feature space of each sample is , and then there is , For simplicity, use Representative Dataset Contains only categories A subset of labeled samples, so we have ,in Represents counting the number of elements in a collection.

[0068] The steps of the PEL evolutionary algorithm include: initialization, objective function design, chromosome crossover and chromosome mutation, which are summarized as follows.

[0069] 1) Initialization of the evolutionary algorithm. First, PEL initializes The chromosome vectors are used as the initial population. Each chromosome encodes the sample selection result and the feature selection result respectively.

[0070] 2) Evolutionary algorithm objective function. PEL considers multiple objective functions to calculate chromosome fitness scores during feature selection. RF is used as the base classifier, denoted as When evaluating the classification performance, cross entropy is used to calculate the classification results for depression and health categories respectively, and the diversity of chromosomes is taken into account. Pareto fitness is calculated based on the above classification results and diversity.

[0071] 3) Evolutionary algorithm crossover. PEL uses random crossover to generate new chromosomes and further generate populations in new iterations. The population after rounds of iterations , firstly, the fitness score calculated by the objective function The chromosomes with the lowest scores are sorted, and the chromosomes with the lowest scores are directly deleted. The chromosomes with the highest scores are directly retained in the next iteration population. The chromosomes with scores between the top 50% are randomly cross-selected to obtain new samples. Specifically, two chromosomes are randomly selected during the crossover process, and their dimensions are randomly exchanged at most 40% and at least 20% with each other, and then two new chromosomes are obtained and added to the population of the next iteration until the new PEL population has a total of A chromosome vector.

[0072] 4) Evolutionary algorithm mutation. In order to improve To increase the diversity of the new population, PEL will also perform mutation operations, that is, randomly change the random value of each dimension of each chromosome with a certain probability. Specifically, for each , each dimension of which will have The probability of changing from 0 to 1 or from 1 to 0. After completing the fitness score calculation, crossover and mutation of the evolutionary algorithm, a PEL completes an iteration. Iterates, and takes the features selected by the chromosome with the highest final fitness score as the final retained feature subset, i.e., the optimal feature subset. The program framework of the PEL algorithm is as follows: Figure 2 shown.

[0073] The Pareto sorting-based evolutionary algorithm PEL proposed in the present invention has the following advantages:

[0074] 1) PEL can search for multiple objective functions in a non-convex high-dimensional feature space, and then obtain the optimal combination of features and sample subsets as training data.

[0075] 2) Compared with the existing filter-based feature selection algorithm, PEL can make full use of the information related to the prediction results of the classification base model, thereby obtaining a model and feature subset that is more conducive to the classification of depression.

[0076] 3) Compared with existing wrapper-based feature selection algorithms, such as the forward sequence selection algorithm, PEL is less likely to fall into local optimality and has higher learning efficiency in high-dimensional space.

[0077] 4) Through sample selection, PEL can balance the number of positive and negative depression models in the training data to alleviate the model bias caused by the uneven distribution of training data.

[0078] S3. Match the facial feature set of the subject with the features of the optimal feature subset and input them into the classification model to obtain the depression prediction result of the subject.

[0079] Specifically, features corresponding to the features of the optimal feature subset are selected from the facial feature set of the subject and input into the classification model to obtain the depression prediction result of the subject.

[0080] Before S3, the method further includes:

[0081] The facial action video of the subject is collected, and features are extracted from the facial action video to obtain a facial feature set of the subject.

[0082] The process of processing the facial action video of the subject to be tested to obtain the facial feature set is consistent with the processing method of the facial action video of the above-mentioned subject, and will not be repeated here.

[0083] Compared with the existing depression detection methods, the depression detection method based on facial video provided by the present invention has the following beneficial effects:

[0084] 1) Non-invasive methods reduce patient discomfort.

[0085] Compared with the invasive methods based on brain-computer interface (BCI) and electroencephalogram (EEG) in the prior art, the present invention adopts a non-invasive facial video analysis method, avoiding physical contact with the subject's head and possible damage. This approach significantly improves the patient's comfort and the acceptability of the diagnosis.

[0086] 2) Improve the accuracy and efficiency of detection.

[0087] The existing technology relies on general acquisition equipment and has a low level of automation. However, the present invention improves the quality of samples and the accuracy of diagnosis through standardized facial action group design and high-quality facial action video acquisition and annotation. In addition, the application of real-time feature extraction and feature selection improves the efficiency of data processing, shortens the diagnosis time, and solves the problem of long measurement process in the existing technology.

[0088] 3) Enhance the generalization ability and robustness of classification models.

[0089] Compared with the prior art, the present invention uses an evolutionary algorithm based on Pareto sorting (PEL) for feature and sample selection, effectively solving the problem of high-dimensional feature selection, balancing sample distribution, and improving the robustness of the classification model to unbalanced data. This method is particularly helpful in improving diagnostic bias caused by sample imbalance in practical applications and enhancing the generalization ability of the model.

[0090] 4) Innovative dynamic video analysis method.

[0091] The existing technology mainly relies on static image analysis. The present invention adopts a dynamic video analysis method that sets a standardized facial action group, which not only reduces the difficulty of analyzing facial expressions, actions and demeanor, but also improves the utilization rate of samples and the accuracy of diagnosis. This innovative method improves the depth and breadth of facial data analysis and solves the problems of imperfect facial visual perception standards and limited analysis methods in the existing technology.

[0092] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for detecting depression based on facial video, characterized in that: The method comprises: S1, determining a facial feature set for each subject according to a facial action video of each subject among a plurality of subjects; S2, training a classifier based on the facial feature set of all subjects and the depression level label of each subject to obtain a classification model and an optimal feature subset; S3, matching the facial feature set of the subject with the features of the optimal feature subset and inputting the matched features into the classification model to obtain the depression prediction result of the subject.

2. The method according to claim 1, characterized in that The facial action video includes mouth action information, eye action information and expression action information completed according to preset requirements.

3. The method according to claim 1, characterized in that The facial feature set includes eye movement angle features, eye movement response features, mouth movement time features, mouth movement response features, expression movement response features, facial key point features and global movement response features.

4. The method according to claim 1, characterized in that The S2 is specifically: An evolutionary algorithm was used to select features from the facial feature sets of all subjects, and the classifier was trained in combination with the depression degree labels of all subjects to obtain the classification model and the optimal feature subset.

5. The method according to claim 4, characterized in that The evolutionary algorithm performs search iterations based on Pareto sorting.

6. The method according to claim 4, characterized in that The classifier is a random forest.

7. The method according to claim 1, characterized in that The S1 specifically includes: Obtain facial action videos and depression level labels of each subject among multiple subjects; Feature extraction is performed on each facial action video to obtain a facial feature set for each subject.

8. The method according to claim 7, characterized in that The feature extraction is performed on each facial action video to obtain a facial feature set for each subject, specifically: A neural network algorithm is used to extract features from each facial action video to obtain a facial feature set for each subject.

9. The method according to claim 1, characterized in that: Before S3, the method further includes: A facial action video of the subject to be tested is collected, and features are extracted from the facial action video to obtain a facial feature set of the subject to be tested.

10. The method according to claim 1, characterized in that The S3 is specifically: Features corresponding to the features of the optimal feature subset are selected from the facial feature set of the subject and input into the classification model to obtain the depression prediction result of the subject.

Images