A depression recognition system based on deep learning behavioral entropy

Abstract

The present invention discloses a depression recognition system based on deep learning behavioral entropy, which includes: at least one image data acquisition unit, which is used to identify the facial expressions and body movements of the subject to be tested; an expression module, which extracts the facial expression features of the subject to be tested according to the facial expressions acquired by the image data acquisition unit, and obtains the probability of the corresponding seven expressions; an action module, which extracts the position information of the site of the subject to be tested in the image according to the body movements acquired by the image data acquisition unit; and a behavioral entropy model, which outputs a behavioral entropy index representing the degree of depression according to the information acquired by the expression module and the action module. By acquiring the facial expressions and body movements of the subject to be tested, the expression entropy and the action entropy are obtained, and the behavioral entropy representing the degree of depression is obtained. The acquisition accuracy of the behavioral entropy is improved by using a deep learning method, and a higher effect of automatic recognition of depression can be achieved, providing a reference for doctors to diagnose.

Description

Technical Field

[0001] The present invention utilizes behavioral entropy defined by deep learning to characterize depression degree, and specifically relates to a depression recognition system based on deep learning behavioral entropy. Background Art

[0002] Clinically, the diagnosis of depression is mainly based on the Patient Health Questionnaire Depression Scores (QDS), Hamilton Depression Scale (HAMD), Self-rating Depression Scale (SDS), Self-rating Anxiety Scale (SAS) and the doctor's questions and answers.

[0003] 1. Most items in the Hamilton Depression Rating Scale use a 5-point scoring system ranging from 0 to 4 points. The standards for each level are: i(0) none; (1) mild; (2) moderate; (3) severe; (4) extremely severe. A few items use a 3-point scoring system ranging from 0 to 2 points. The standards for their classification are: i(0) none; (1) mild to moderate; (2) severe. A total score of more than 24 points indicates severe depression, more than 17 points indicates mild or moderate depression, and less than 7 points indicates no depressive symptoms.

[0004] 2. The Depression Self-rating Scale consists of 20 items, using a four-level scoring method of 1-4 points (never or occasionally, sometimes, often, always), of which 10 items, including 2, 5, 6, 11, 12, 14, 16, 17, 18 and 20, are reverse scoring questions, and 10 items, including 1, 3, 4, 7, 8, 9, 10, 13, 15, 19, are positive scoring questions. Those with a score index below 50 have no depression; 50-59 are mild to mild depression; 60-69 are moderate to severe depression; and 70 or above are severe depression.

[0005] 3. The anxiety self-rating scale is similar to the depression self-rating scale, consisting of 20 items, using a four-level scoring method of 1-4 points (never or occasionally, sometimes, often, always), of which the 5th, 9th, 13th, 17th and 19th items are reverse scoring questions, and the 15th items 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 18 and 20 are positive scoring questions. Those with a score index below 50 have no anxiety; 50-59 are mild to mild anxiety; 60-69 are moderate to severe anxiety; and 70 or above are severe anxiety.

[0006] The Hamilton Depression Rating Scale is administered by two trained assessors who conduct a Hamilton Joint Examination on the patient. It is generally conducted through conversation and observation. After the examination, the two assessors independently score the patient, and the scores are performed before and after treatment to evaluate the severity of the disease and the effectiveness of the treatment. This method has high requirements for assessors, who need to undergo rigorous training, and it is easy for them to fail to observe the patient's detailed emotions and make unbalanced judgments on the degree of depression. Summary of the invention

[0007] In view of the shortcomings of the prior art, the object of the present invention is to provide a depression identification system based on deep learning behavioral entropy.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A depression recognition system based on deep learning behavioral entropy, comprising:

[0010] At least one image data acquisition unit, used to identify the facial expressions and body movements of the subject;

[0011] The expression module extracts the facial expression features of the subject according to the facial expressions acquired by the image data acquisition unit, and obtains the corresponding probability of occurrence of seven expressions;

[0012] The action module extracts the position information of the subject in the image according to the body movements acquired by the image data acquisition unit;

[0013] The behavioral entropy model outputs a behavioral entropy index representing the degree of depression based on the information obtained from the expression module and the action module.

[0014] It is constructed as follows:

[0015] First, the expression entropy is calculated using the probability of the seven expressions obtained by the expression module.

[0016] Where H(X) represents the expression entropy, p(x i ) represents the probability of the i-th expression appearing;

[0017] Secondly, the position information of the subject obtained by the action module is used, and the action entropy is obtained by moving the distance D between the positions of two adjacent pictures.

[0018] Where G(Y) represents action entropy, D(y j ) represents the position movement distance of the jth site,

[0019] Finally, we use λ to linearly fit the expression entropy and action entropy to get the behavior entropy

[0020] F(X,Y)=λH(X)+(1-λ)G(Y), where F(X,Y) represents behavioral entropy and λ represents the matching parameter.

[0021] The image data acquisition unit includes a picture extraction module for performing frame operations on the acquired video.

[0022] The image data acquisition unit is also provided with an image preprocessing module.

[0023] The working steps of the image preprocessing module are as follows:

[0024] B1. Crop the target area of ​​the image to obtain face and body area data;

[0025] B2, perform data enhancement on the input data obtained in B1;

[0026] B3. Convert the enhanced face image into a grayscale image.

[0027] The expression module includes multiple convolution sub-modules.

[0028] The expression module includes:

[0029] A database, which is used to input a data set and obtain feature information of the data set, including a training set, a validation set, and a test set;

[0030] The learning model calls a set standard model framework according to the feature information to perform unsupervised learning on the input data set;

[0031] Verify the model and use the verification set to verify the progress of the learning model;

[0032] The output model is used to output the learning model when the accuracy of the learning model reaches a set threshold, and associate the learning model with the feature information of the data set.

[0033] The action model includes:

[0034] An input module, used for inputting graphic data for recognition;

[0035] Recognition model, through the established training model, human body recognition is performed on the input graphic data, where human body recognition includes landmark location and limb recognition;

[0036] The motion analysis module is used to compare the amplitude and adjacent image data based on the information identified by the recognition model.

[0037] The beneficial effects of the present invention are as follows: by obtaining the facial expressions and body movements of the subject to be tested, the expression entropy and movement entropy are obtained, and the behavioral entropy representing the degree of depression is obtained. The accuracy of obtaining the behavioral entropy is improved by using deep learning, so as to achieve a higher effect of automatic recognition of depression and provide a reference for doctors to make a diagnosis. It can greatly reduce the time spent on tedious diagnosis of depression in patients and achieve a better effect of identifying depression. It eliminates the need for too many complicated medical scales and doctor-patient interviews in the early stage, saves diagnosis and treatment time, and can be used in local hospitals where there is a shortage of psychiatrists to effectively improve the accuracy of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flow chart is designed for the overall scheme of the present invention.

[0039] Figure 2 Flow chart of the expression module of the present invention.

[0040] Figure 3 It is a flow chart of the action module of the present invention.

[0041] Figure 4 This is a map of 18 human body sites of the present invention.

[0042] Figure 5 This is a schematic diagram of the changes in the patient's seven-dimensional facial expression before and after treatment.

[0043] Figure 6 This is a schematic diagram of the changes in the patient's limb movement amplitude before and after treatment.

[0044] Figure 7 This is a schematic diagram of the changes in patients' self-assessment of depression during treatment.

[0045] Figure 8 This is a schematic diagram of the changes in the behavioral entropy depression level of this system during the patient treatment process. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0048] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or a connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The present invention provides a depression recognition system based on deep learning behavioral entropy, which comprises:

[0050] At least one image data acquisition unit, used to identify the facial expressions and body movements of the subject, which uses a camera or a camera to directly acquire video image data of the subject over a period of time;

[0051] The image data acquisition unit includes a picture extraction module for performing frame operations on the acquired video. First, it is necessary to perform frame operations on the acquired video image data and obtain pictures, because the video image data contains more than 60 frames of pictures per second. If each frame is extracted, our processing efficiency will be greatly reduced. At the same time, the change range between two adjacent frames is not large, and the facial expressions and movements are basically consistent. If the time interval between the two extracted frames is too large, local information will be lost. Therefore, in this application, we set the time interval between the two extracted frames to 1 second. We uniformly divide the multiple videos in a visit into one treatment data.

[0052] At the same time, the image data acquisition unit is also provided with an image preprocessing module, which includes face area selection, body area selection, data enhancement, grayscale conversion, etc.

[0053] The working steps of the image preprocessing module are as follows:

[0054] B1. Crop the target area of ​​the image to obtain face and body area data, ensuring that only face and body areas are included in the expression and action model, avoiding the model from processing redundant information.

[0055] B2. Perform data enhancement on the input data obtained in B1. The data enhancement method may adopt data enhancement means such as random cropping, random noise, inversion, distortion, and mirroring;

[0056] B3. Convert the enhanced face image into a grayscale image.

[0057] The expression module extracts the facial expression features of the subject according to the facial expressions obtained by the image data acquisition unit, and obtains the corresponding probability of the seven expressions. The expression module is also called the expression feature extraction stage. Figure 2 The specific process of the expression module is shown. This module can recognize the probability of seven facial expressions of the patient. It contains multiple convolution submodules with the same structure: two convolution operations plus one pooling operation. Each time a convolution submodule is performed, certain information contained in the image, such as boundaries and colors, is extracted. The more convolutions are performed, the more abstract features of the image can be captured. It can increase the robustness to some small disturbances of the input image, such as image translation and rotation, reduce the risk of overfitting, reduce the amount of calculation, and increase the size of the receptive field.

[0058] The expression module includes:

[0059] A database, which is used to input a data set and obtain feature information of the data set, including a training set, a validation set, and a test set;

[0060] The learning model calls a set standard model framework according to the feature information to perform unsupervised learning on the input data set;

[0061] Verify the model and use the verification set to verify the progress of the learning model;

[0062] The output model is used to output the learning model when the accuracy of the learning model reaches a set threshold, and associate the learning model with the feature information of the data set.

[0063] The action module extracts the position information of the subject in the image according to the limb movements obtained by the image data acquisition unit; it is also called the limb recognition module. Figure 3 The specific process of our action module is shown in Figure 1. Its function is to identify the position of each part of the human body in the image and calculate the moving distance of the part based on the change information of the part position between the previous and next photos. Figure 4 As shown in Figure 1, this module outputs the position information of 18 sites of the human body in each picture, and we calculate the action features based on the site change information.

[0064] The action model includes:

[0065] An input module, used for inputting graphic data for recognition;

[0066] Recognition model, through the established training model, human body recognition is performed on the input graphic data, where human body recognition includes landmark location and limb recognition;

[0067] The motion analysis module is used to compare the amplitude and adjacent image data based on the information identified by the recognition model.

[0068] The behavioral entropy model outputs a behavioral entropy index representing the degree of depression based on the information obtained from the expression module and the action module.

[0069] It is constructed as follows:

[0070] First, the expression entropy is calculated using the probability of the seven expressions obtained by the expression module.

[0071] Where H(X) represents the expression entropy, which indicates the complexity of the subject's expression. The larger the value, the richer the subject's facial expression. i ) represents the probability of the i-th expression appearing;

[0072] Secondly, the position information of the subject obtained by the action module is used, and the action entropy is obtained by moving the distance D between the positions of two adjacent pictures.

[0073] Where G(Y) represents the action entropy, which indicates the complexity of the subject's limb movement, and D(y j ) represents the position movement distance of the jth site,

[0074] Finally, we use λ to linearly fit the expression entropy and action entropy to get the behavior entropy

[0075] F(X,Y)=λH(X)+(1-λ)G(Y), where F(X,Y) represents behavioral entropy and λ represents the matching parameter. In the recognition process, we use Pearson similarity to determine the value of the matching parameter λ. Specific embodiment:

[0077] Step 1: Data preprocessing:

[0078] A. Patient video preprocessing:

[0079] Before deep learning, the image is preprocessed by simple target area cropping, brightness and contrast enhancement. The specific steps are as follows:

[0080] A1. Divide the patient's video into frames and capture a picture in the video every second.

[0081] A2. Use the face recognition model to select the face area in the image and process it into a 48*48 grayscale image input.

[0082] A3. Use the limb recognition model to select the human body area and perform data enhancement processing.

[0083] In the above step A1, it should be noted that there is a patient's body image in the target video, and if noise is present, it needs to be filtered.

[0084] In step A2 above, you need to select faces under different lighting and postures, and then perform grayscale conversion. We use a face recognition method based on the TensorFlow deep learning framework. Then we perform geometric normalization on the image and reshape the image into 48*48 pixels using a bilinear interpolation algorithm.

[0085] In the above step A3, the augmented data set can be quickly generated using the image augmentation function in the TensorFlow deep learning framework, including rotation, reflection transformation, flip transformation, scaling transformation, translation transformation, scale transformation, etc.

[0086] B. Acquisition of behavioral characteristics:

[0087] Behavioral features include: expression features and action features. The specific steps to obtain behavioral features are as follows:

[0088] B1. Input the 48*48 pixel image obtained in the above A2 process into our expression module, and obtain the probabilities of the seven expressions corresponding to the image based on the output of the model.

[0089] B2. Input the human body image obtained in the above A3 process into our limb recognition model, and obtain the position information of the 18 sites corresponding to the image based on the output of the model.

[0090] In step B1 above, when performing facial expression recognition on an image, the parameters of the convolution operation are: the activation function is "relu", the filling method is "same", and the weight initialization method is "he_normal". The pooling operation uses the maximum pooling scheme. At the same time, in order to prevent the model from overfitting, we added two dropouts. abandoned layer, with a dropout coefficient of 0.4. Our model then outputs to the fully connected layer, and finally outputs the probabilities of seven facial expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral. We output the facial expression features of the patient before and after treatment, such as Figure 5 As shown, the upper red shaded area represents the patient's happy mood, and the lower shaded area represents the patient's depressed mood.

[0091] The feature extraction result of step B2 above is as follows Figure 6 As shown in the figure, we use the limb recognition model to identify the 18 joints of the target person, which is based on TensorFlo w The human posture recognition implemented by the deep learning framework is relatively lightweight and can use the CPU for real-time detection.

[0092] The ordinate represents the positional movement distance of the corresponding site, and a null value (such as the left ankle) indicates that the site does not exist in the image.

[0093] Step 2: Construction of behavioral entropy model:

[0094] The behavioral entropy model is mainly divided into two parts: expression module (expression recognition) and action module (body recognition). Its advantage is that it can capture expression and action features at the same time, and the network can learn the importance of different deep features by itself, and has good robustness.

[0095] C. Expression module (expression recognition):

[0096] The expression module is also called the expression recognition stage, which contains multiple convolution submodules with the same structure: two convolution operations plus one pooling operation. Each time a convolution submodule is performed, certain information contained in the image, such as boundaries and colors, will be extracted. The more convolutions are performed, the more abstract features of the image can be captured. It can increase the robustness to some small disturbances of the input image, such as image translation and rotation, reduce the risk of overfitting, reduce the amount of calculation, and increase the size of the receptive field. Finally, the output expression features are fitted into the expression entropy.

[0097] In the above step C, Adam (Adaptive Moment Estimation) is selected as the optimizer, binary classifier binary_crossentropy is selected as the loss function, and accuracy is used as the measurement indicator. The datasets used are fer2013 and CK+ databases.

[0098] After 70 epochs of training over 3 hours, the expression module achieved an accuracy of 97.2% on the validation set.

[0099] D. Action module (body recognition):

[0100] The action module is also called the limb recognition module. Its function is to recognize the patient's limb movement. After the recognized image is sent to the module, the position information of 18 sites is recognized. We calculate the displacement based on the change in the position information of the two images before and after. Finally, it is fitted into the action entropy output.

[0101] The hardware platform in the above step D is a server with a GeForce GTX 1080Ti high-performance graphics card. After the TensorFlow deep learning framework is used for training, the patient video data is tested and output.

[0102] Step 3: Behavioral entropy parameter optimization

[0103] In order to optimize the accuracy of our behavioral entropy system, we used the patient's depression self-rating data and our behavioral entropy depression score to do a Pearson correlation test. The prediction results of this system solution are similar to those of the traditional solution. Figure 7 and Figure 8As shown in Figure 2, when the matching parameter λ is 0.98, behavioral entropy can effectively fit the patient medical data. The Pearson correlation reaches 0.89.

[0104] At the same time, we analyze the patient's multiple treatment data to determine the patient's treatment status and provide scientific information for the patient's prognosis. We judge the patient's condition improvement based on the changes in behavioral entropy indicators.

[0105] The embodiments should not be regarded as limiting the present invention, but any improvements made based on the spirit of the present invention should be within the protection scope of the present invention.

Claims

1. A depression recognition system based on deep learning behavioral entropy, characterized by: It includes: At least one image data acquisition unit, used to identify the facial expressions and body movements of the subject; The expression module extracts the facial expression features of the subject according to the facial expressions acquired by the image data acquisition unit, and obtains the corresponding probability of occurrence of seven expressions; The action module extracts the position information of the subject in the image according to the body movements acquired by the image data acquisition unit; The behavioral entropy model outputs a behavioral entropy index representing the degree of depression based on the information obtained from the expression module and the action module. It is constructed as follows: First, the expression entropy is calculated using the probability of the seven expressions obtained by the expression module. ,in represents the expression entropy, Indicates i The probability of the expression appearing; Secondly, the position information of the subject obtained by the action module is used, and the action entropy is obtained by moving the distance D between the positions of two adjacent pictures. ,in represents action entropy, Indicates j The distance of position movement of each site, ; Final use Linearly fit the expression entropy and action entropy to obtain the behavior entropy ,in represents behavioral entropy, Represents the ratio parameter, The expression module includes: A database, which is used to input a data set and obtain feature information of the data set, including a training set, a validation set, and a test set; The learning model calls a set standard model framework according to the feature information to perform unsupervised learning on the input data set; Verify the model and use the verification set to verify the progress of the learning model; Output model, used for outputting the learning model when the accuracy of the learning model reaches a set threshold, and associating the learning model with the feature information of the data set, The action module includes: An input module, used for inputting graphic data for recognition; Recognition model, through the established training model, human body recognition is performed on the input graphic data, where human body recognition includes landmark location and limb recognition; The motion analysis module is used to compare the amplitude and adjacent image data based on the information identified by the recognition model.

2. A depression identification system based on deep learning behavioral entropy according to claim 1, characterized in that: The image data acquisition unit includes a picture extraction module for performing frame operations on the acquired video.

3. A depression identification system based on deep learning behavioral entropy according to claim 1 or 2, characterized in that: The image data acquisition unit is also provided with an image preprocessing module.

4. The depression identification system based on deep learning behavioral entropy according to claim 3 is characterized by: The working steps of the image preprocessing module are as follows: B1. Crop the target area of ​​the image to obtain face and body area data; B2, perform data enhancement on the input data obtained in B1; B3. Convert the enhanced face image into a grayscale image.

5. The depression identification system based on deep learning behavioral entropy according to claim 1, characterized in that: The expression module includes multiple convolution sub-modules.

Images