Method and system for evaluating Hamilton depression scale based on expression video

Abstract

The invention discloses an expression video-based Hamilton depression scale evaluation method and system, and relates to the technical field of emotional psychology, artificial intelligence and machine learning, and the system comprises an emotional video playing module, an expression video acquisition module, a data processing module and a scale evaluation module. The emotion video playing module randomly selects a positive emotion video and a negative emotion video from a preset emotion database to play and provide the videos for an evaluated person to watch; the expression video acquisition module synchronously acquires an expression video of an evaluated person; the data processing module preprocesses the acquired expression video; a scale evaluation module evaluates a HAMD scale value. Therefore, by adopting the Hamton depression scale evaluation method and system based on the expression video, the HAMD scale can be measured and evaluated more objectively, and scientific information is provided for clinical depression diagnosis and curative effect evaluation.

Description

Technical Field

[0001] The present invention relates to the fields of emotional psychology, artificial intelligence and machine learning technology, and in particular to a Hamilton Depression Rating Scale evaluation method and system based on expression video. Background Art

[0002] Currently, the clinical diagnosis of depression severity is mainly based on scale information and the experience of psychiatrists, especially the Hamilton Depression Rating Scale (HAMD) information.

[0003] The HAMD scale is usually assessed by doctors or trained professionals, and is highly accurate, so it is known as the "gold scale" in the industry. Therefore, the HAMD scale is of great value in assessing the severity of depression.

[0004] However, manual assessment of HAMD takes 20 to 30 minutes, which is time-consuming and laborious. Facial expressions are important signals for people to communicate their emotions. A large number of studies have shown that patients with depression have dull expressions, and the more severe the depression, the more difficult it is to express facial expressions. Therefore, by collecting facial expression signals from watching videos with different emotions and using machine learning, we can achieve automatic evaluation of the HAMD scale with high accuracy, providing a scientific basis for the diagnosis of clinical depression. Summary of the invention

[0005] The purpose of the present invention is to provide a Hamilton Depression Rating Scale assessment method and system based on expression video, which can improve the assessment efficiency and assessment accuracy, and provide a scientific basis and objective standard for clinical depression diagnosis and efficacy evaluation.

[0006] To achieve the above object, the present invention provides a Hamilton Depression Rating Scale evaluation method based on facial expression video, comprising the following steps: First, through the emotional video playback module, a sad video and a happy video are selected from the preset emotional database and played for the person being evaluated; Secondly, through the data collection module, collect the expression videos of the evaluated persons when they watch sad videos and happy videos; Next, the collected expression videos are preprocessed through the data processing model; Then, the static and dynamic features of the preprocessed expression video are extracted through the scale evaluation module to evaluate the HAMD scale value.

[0007] Preferably, when the person being evaluated watches the emotion video, a camera is used to synchronously collect expression video information.

[0008] Preferably, preprocessing includes downsampling, face detection and registration, as follows: Downsampling means sampling the expression video at the same frame rate to reduce the length to half of the original expression video; Facial detection and registration refers to detecting the position of facial key points in each frame of the downsampled video, and performing image cropping and alignment operations based on the position of facial key points.

[0009] Preferably, the static features and dynamic features are extracted, including: First, extract Static texture features of a plane ,as follows: ; Secondly, extract the time intervals of each row and column along the time axis. flat, The dynamic texture features of the plane are as follows: ; ; in, ; , , They are horizontal axis, vertical axis and time axis respectively; Indicates the pixel in the 3×3×3 window area except the center pixel Outside pixels, Represents the gray value of the center pixel, Indicates the first The value of the pixel, subscript , , Indicates the corresponding plane.

[0010] Preferably, evaluating the HAMD scale value comprises calculating the evaluation value of the HAMD scale according to the extracted features through a state equation of the neural network, as follows: ; ; ; ; ; ; In the formula, is the evaluation value of the HAMD scale, Indicates input samples, represents the number of input signals of the input layer, ; represents the input signal; , Respectively represent the number of neurons in the first hidden layer and the second hidden layer, , ; , Respectively represent the connection weights between the input layer and the first hidden layer, and between the first hidden layer and the second hidden layer; , Respectively represent the input and output of the first hidden layer neurons; , They represent the input and output of the neurons in the second hidden layer, Represents the output of the output layer neurons; represents the transfer function.

[0011] A Hamilton Depression Rating Scale assessment system based on facial expression video, comprising: The emotional video playback module is connected to the data acquisition module and is used to play sad videos or happy videos; A data collection module, connected to the data processing module, is used to collect the expression videos of the evaluated person when watching the sad video and the happy video; A data processing module, connected to the scale evaluation module, is used to pre-process the collected expression videos; The scale evaluation module is used to extract the features of expression videos and evaluate the HAMD scale.

[0012] Preferably, the emotional video playback module includes randomly selecting a sad video or a happy video, editing the selected sad video or happy video into a fixed time, and providing it to the evaluated person for viewing through a player.

[0013] Therefore, the present invention adopts the above-mentioned Hamilton Depression Rating Scale assessment method and system based on expression video, which has the following technical effects: (1) By extracting the static and dynamic features of expression videos and then evaluating the HAMD scale value based on the corresponding state equation, the assessment efficiency of the HAMD score can be improved, while the assessment accuracy can be improved, providing a scientific basis and objective standard for the diagnosis and efficacy evaluation of clinical depression.

[0014] (2) Based on machine learning technology, a system for automatically evaluating HAMD scale scores is provided, which makes up for the lack of automatic evaluation methods and devices for HAMD scale information in the past.

[0015] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of the Hamilton Depression Rating Scale assessment system based on expression videos; Figure 2 It is a schematic diagram of facial expression video collection in Embodiment 2 of the Hamilton Depression Rating Scale evaluation method and system based on facial expression video; Figure 3 It is a schematic diagram of evaluating the HAMD scale value in Embodiment 2 of the Hamilton Depression Rating Scale evaluation method and system based on expression video; Figure 4 It is a schematic diagram of facial expression video collection in Embodiment 3 of the Hamilton Depression Rating Scale evaluation method and system based on facial expression video; Figure 5 The present invention is a schematic diagram of evaluating the HAMD scale value in Embodiment 3 of the Hamilton Depression Rating Scale evaluation method and system based on expression video. DETAILED DESCRIPTION

[0017] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0018] Embodiment 1 like Figure 1 As shown, the present invention provides a Hamilton Depression Rating Scale assessment system based on expression video, including: an emotion video playback module, a data acquisition module, a data processing module, and a scale assessment module.

[0019] The emotional video playback module is used to randomly select a sad video and a happy video from a preset emotional database for playback and provide it to the person being evaluated for viewing.

[0020] The data collection module is used to collect the facial expression videos of the evaluated persons when they watch sad videos and happy videos.

[0021] The data processing module is used to pre-process the collected expression videos.

[0022] The scale evaluation module is used to evaluate the HAMD scale value.

[0023] Through the above system, the HAMD scale can be measured and evaluated more objectively and efficiently, providing scientific information for the diagnosis and efficacy evaluation of clinical depression.

[0024] Embodiment 2 The present invention provides a Hamilton Depression Rating Scale evaluation method based on expression video, which is as follows: First, a happy video is randomly selected from the positive emotion video library, and a sad video is randomly selected from the negative emotion video library, and they are all cut into 90 seconds in length.

[0025] Then, the player was used to present the happy and sad videos to the person being evaluated, and the camera was used to synchronously capture the expression videos, e.g. Figure 2 In this embodiment, the camera capture frequency is 80 frames per second, the image resolution is 1280×960, the capture time for each emotion is 90 seconds, a total of 180 seconds of expression video is captured, and a total of 14,400 frames of expression images are captured.

[0026] Next, the collected expression videos were processed: starting from the first frame, sampling was performed every 10 frames to reduce the original video length by 50%; the positions of the key points of the face, such as the eyes, nose, and mouth, were detected for each frame of the downsampled video, and the images were cropped to 224×224 size according to the key point coordinates to reduce background noise, and then all the frames were aligned with the first frame. After preprocessing, each expression video had a total of 1440 expression images.

[0027] Finally, the expression images (including happy expression videos and sad expression videos) are extracted in the same feature extraction channel, sharing the same set of parameters, and the HAMD score of the pre-processed expression videos is evaluated using the scale evaluation module. Figure 3 As shown, for the preprocessed expression video, static expression features are extracted , extract dynamic expression features along the axis , extract dynamic expression features along the axis ; Then a fully connected neural network with two hidden layers is used to evaluate the HAMD scale value. The network structure includes an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0028] The subjects used in this embodiment include subjects of different ages, occupations and educational backgrounds, and the data are fully balanced. The evaluation results are shown in Table 1. It can be obtained that the average value of the clinical HAMD score is 20.29, the predicted clinical HAMD value is 21.01, and the predicted HAMD value is compared with the clinical HAMD score, and the predicted value is close to the true value, and the mean and variance of the error are 1.37 and 0.75, respectively.

[0029] Table 1 Evaluation results of HAMD scale values ;

[0030] Embodiment 3 The present invention also provides a method for evaluating the Hamilton Depression Rating Scale based on expression videos, such as Figure 4 As shown, a happy video is randomly selected from the positive emotion video library, and a sad video is randomly selected from the negative emotion video library, and both are edited into 90 seconds in length.

[0031] Then, the player was used to present the happy and sad videos to the subject, and the camera was used to synchronously capture the facial expressions of the subject while watching the happy and sad videos. The camera capture frequency was 80 frames per second, the image resolution was 1280×960, and the capture time for each emotion was 90 seconds. Each happy expression video captured included 7200 frames of expression images for 90 seconds, and each sad expression video captured included 7200 frames of expression images for 90 seconds.

[0032] Next, the data of each collected expression video was processed: starting from the first frame, sampling was performed every 10 frames to reduce the original video length by 50%; the positions of the key points of the face such as eyes, nose and mouth in each frame of the downsampled video were detected, and the image was cropped to 224×224 size according to the key point coordinates to reduce background noise, and then the images of all frames were aligned with the first frame. After preprocessing, each happy expression video had a total of 720 expression images, and each sad expression video had a total of 720 expression images.

[0033] Finally, the HAMD score was evaluated using the scale assessment module. Figure 5 As shown in the figure, feature extraction is performed on the collected happy expression videos and sad expression videos respectively; for the happy expression videos, static expression features are extracted , extract along Dynamic expression characteristics of the axis , extract along Dynamic expression characteristics of the axis ; Extract static expression features for sad expression videos , extract along Dynamic expression characteristics of the axis , extract along Dynamic expression characteristics of the axis .

[0034] Different from the second embodiment, this embodiment includes two processes: HAMD score evaluation of happy expression videos and HAMD score evaluation of sad expression videos. These two processes are performed independently in different channels, and the parameters are independent of each other. The final evaluation results include the HAMD prediction score of the happy expression video and the HAMD prediction score of the sad expression video.

[0035] Therefore, the present invention adopts the above-mentioned Hamilton Depression Rating Scale assessment method and system based on expression video, which can improve the assessment efficiency and assessment accuracy, and provide a scientific basis and objective standard for clinical depression diagnosis and efficacy evaluation.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A Hamilton Depression Rating Scale assessment method based on facial expression video, characterized in that: The following steps are involved: First, through the emotional video playback module, a sad video and a happy video are selected from the preset emotional database and played for the person being evaluated; Secondly, through the data collection module, collect the expression videos of the evaluated persons when they watch sad videos and happy videos; Next, the collected expression videos are preprocessed through the data processing model; Then, the static and dynamic features of the preprocessed expression video are extracted through the scale evaluation module to evaluate the HAMD scale value.

2. The Hamilton Depression Rating Scale evaluation method based on facial expression video according to claim 1, characterized in that: When the person being evaluated watches the emotional video, the camera is used to synchronously collect expression video information.

3. The Hamilton Depression Rating Scale evaluation method based on facial expression video according to claim 1, characterized in that: Preprocessing includes downsampling, face detection and registration, as follows: Downsampling means sampling the expression video at the same frame rate to reduce the length to half of the original expression video; Facial detection and registration refers to detecting the position of facial key points in each frame of the downsampled video, and performing image cropping and alignment operations based on the position of facial key points.

4. The Hamilton Depression Rating Scale assessment method based on facial expression video according to claim 1, characterized in that: Extract static and dynamic features, including: First, extract Static texture features of a plane ,as follows: ； Secondly, extract the time intervals of each row and column along the time axis. flat, The dynamic texture features of the plane are as follows: ； ； in, ; , , They are horizontal axis, vertical axis and time axis respectively; Indicates the pixel in the 3×3×3 window area except the center pixel Outside pixels, Represents the gray value of the center pixel, Indicates the first The value of the pixel, subscript , , Indicates the corresponding plane.

5. The Hamilton Depression Rating Scale evaluation method based on facial expression video according to claim 1, characterized in that: Evaluating the HAMD scale value includes calculating the evaluation value of the HAMD scale based on the extracted features through the state equation of the neural network, as follows: ； ； ； ； ； ； In the formula, is the evaluation value of the HAMD scale, Indicates input samples, represents the number of input signals of the input layer, ; represents the input signal; , Respectively represent the number of neurons in the first hidden layer and the second hidden layer, , ; , Respectively represent the connection weights between the input layer and the first hidden layer, and between the first hidden layer and the second hidden layer; , Respectively represent the input and output of the first hidden layer neurons; , They represent the input and output of the neurons in the second hidden layer, Represents the output of the output layer neurons; represents the transfer function.

6. A Hamilton Depression Rating Scale assessment system based on facial expression video, characterized in that: include: The emotional video playback module is connected to the data acquisition module and is used to play sad videos or happy videos; A data collection module, connected to the data processing module, is used to collect the expression videos of the evaluated person when watching the sad video and the happy video; A data processing module, connected to the scale evaluation module, is used to pre-process the collected expression videos; The scale evaluation module is used to extract the features of expression videos and evaluate the HAMD scale.

7. The Hamilton Depression Rating Scale assessment system based on facial expression video according to claim 6, characterized in that: The emotional video playback module includes randomly selecting a sad video or a happy video, editing the selected sad video or happy video into a fixed time, and providing it to the assessee through a player.

Images