Emotional disorder auxiliary diagnosis system based on attention analysis and micro-expression recognition

Abstract

The application belongs to the technical field of image recognition, and particularly relates to an emotion disorder auxiliary diagnosis system based on attention analysis and micro-expression recognition. The system comprises: a data acquisition module for acquiring and / or inputting a video; an attention analysis module for analyzing the attention of a subject according to the blinking and face positioning success rate of the subject in the video, and further obtaining a judgment of whether the video can be used for subsequent auxiliary diagnosis; and an auxiliary diagnosis module for performing micro-expression recognition on the video determined as usable, and obtaining a micro-expression recognition result for emotion disorder auxiliary diagnosis. The application firstly uses attention analysis to screen video data with auxiliary diagnosis value, and further performs micro-expression recognition on the video data, thereby improving the reliability of the video data for auxiliary diagnosis, and further improving the accuracy of the subsequent auxiliary diagnosis process, and having a good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an auxiliary diagnostic system for mood disorders based on attention analysis and micro-expression recognition. Background Technology

[0002] Micro-expressions are brief facial expressions that appear when people intentionally hide or unconsciously suppress their emotions. Micro-expressions contain a wealth of information about a person's actual emotions, applicable in fields such as healthcare, security, and interrogation. Individual emotions are more likely to be expressed through micro-expressions when in high-risk environments. However, the subtle differences in micro-expressions displayed by facial muscles make extracting this information extremely challenging. This presents two main technical challenges: first, making micro-expression features describable; and second, given that micro-expressions last only 1 / 25 to 1 / 5 of a second, micro-expression systems must be able to process and classify human emotions in real time.

[0003] Currently, the diagnosis of mood-related disorders (including major depressive disorder, bipolar disorder, and anxiety disorders) mainly relies on the patient's own clinical history and that of informed witnesses, combined with a professional psychiatric examination. Doctors make a comprehensive judgment based on diagnostic criteria (DSM-IV, ICD-10, etc.). Patients often experience significant mood swings and are prone to negative emotions such as anger, anxiety, depression, loneliness, and impulsive symptoms. Commonly used screening tools (such as psychological assessment scales) are symptom severity assessment scales. Diagnostic results are influenced by the patient's subjective expression and the doctor's clinical experience. Furthermore, there is a delay between the onset of symptoms and hospitalization, during which the patient's suffering and socioeconomic losses can be avoided through early detection. Microexpressions are an important part of emotional expression; patients with mood-related disorders often exhibit unusual microexpressions. Microexpressions can be objectively quantified using facial action coding systems to extract facial features for further evaluation.

[0004] Deep learning has been widely applied in various fields. For the recognition of micro-expressions or emotions, labeled data is typically used to train neural network models, which are then used to predict the patient's state. For example, CN114333002A invented a method for micro-expression recognition based on graph deep learning and 3D facial reconstruction. However, these deep learning methods have not been very effective in emotion diagnosis. This is because micro-expression recognition methods usually process a video of the subject's face. During the acquisition of these videos, the subject's face and some key features (such as the eyes) need to be kept in the correct positions, which requires the subject to maintain a high level of concentration. Attention is the core of human cognitive activity and plays a crucial role in higher cognitive activities such as reasoning, problem-solving, and learning. Impaired attention can be seen in many mental disorders, including depressive disorders, mania, anxiety disorders, schizophrenia, and organic mental disorders. Patients with emotion-related disorders often experience impaired attention. This often results in facial occlusion and incorrect positioning of key areas during video capture of patients with mood-related disorders, leading to significant errors in micro-expression recognition and further diagnosis of mood disorders based on the videos of these patients. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an auxiliary diagnostic system for mood disorders based on attention analysis and micro-expression recognition. The aim is to continuously and accurately identify facial expression changes in videos ranging from 1 second to 30 minutes, while ensuring the reliability of video data, in order to assist in achieving more accurate diagnosis of mood disorders.

[0006] A diagnostic aid system for mood disorders based on attention analysis and micro-expression recognition, comprising:

[0007] The data acquisition module is used to acquire and / or input video.

[0008] The attention analysis module is used to analyze the attention of the subjects based on their blinking and facial positioning success rates in the video, and then to determine whether the video can be used for subsequent diagnosis.

[0009] The auxiliary diagnostic module is used to perform micro-expression recognition on videos that are deemed usable, and obtain micro-expression recognition results for auxiliary diagnosis of mood disorders.

[0010] Preferably, in the attention analysis module, the video can be used for subsequent auxiliary diagnostic determination if the following two conditions are met:

[0011] 1) Facial localization success rate ≥ 90%;

[0012] 2) The time taken to blink 7 times is ≥15 seconds.

[0013] Preferably, the face localization success rate is calculated as the proportion of frames in the video where face localization is successful to the total number of frames in the video.

[0014] Preferably, the method for micro-expression recognition in the auxiliary diagnostic module includes the following steps:

[0015] Step 1: Input video, separate each frame of the video, and obtain the face in each frame to form a face frame sequence;

[0016] Step 2: Perform difference analysis on two adjacent frames in the face frame sequence to obtain a difference value sequence of the face frame sequence;

[0017] Step 3: Apply a Fourier transform with a Hanning window to the difference value sequence to generate an expression curve;

[0018] Step 4: In the expression curve, multiple expression intervals are divided according to the setting of peaks, troughs and thresholds;

[0019] Step 5: Apply the TVL1 optical flow algorithm to the face frame sequence to obtain an optical flow sequence;

[0020] Step 6: Iterate the face frame sequence and optical flow sequence of the expression interval trough, and feed them into the dual-flow model of spatial flow and temporal flow to obtain the expression result of that expression interval.

[0021] Preferably, in step 1, each frame of the video is processed into a 256x256 grayscale image.

[0022] Preferably, in step 1, the facial positioning coordinates [x,y,w,h] of each frame in the video are adjusted as follows: the lower frame coordinates are increased by 10 pixels, the left frame is shifted 10 pixels to the right, and the right frame is shifted 10 pixels to the left.

[0023] Preferably, in step 5, a valley sampling method is used, and the TVL1 optical flow algorithm is applied to the 3-frame sequence of the spatial stream.

[0024] Preferably, in step 6, the dual-stream model consists of a spatial CNN and a temporal CNN, and the spatial CNN and the temporal CNN are finally fused by a fully connected layer.

[0025] Preferably, it also includes a facial displacement analysis module, which is used to analyze the specific displacement of specific parts of the face based on the displacement of 68 key points on the face.

[0026] Preferably, it also includes an output module, which is used to output the judgment results of the attention analysis module and the micro-expression recognition results of the auxiliary diagnosis module.

[0027] The present invention also provides a computer-readable storage medium having a computer program stored thereon for implementing the above-described system.

[0028] This invention provides a system for assisting in the diagnosis of mood disorders. This system, for the first time, utilizes attention analysis to screen video data with diagnostic value and further performs micro-expression recognition on this video data to provide valuable diagnostic indicators for mood disorders. The advantage of this invention lies in improving the reliability of the video data used for auxiliary diagnosis, thereby enhancing the accuracy of subsequent auxiliary diagnostic processes, and it has excellent application prospects.

[0029] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions, or alterations can be made without departing from the basic technical concept of the present invention.

[0030] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the diagnostic process for mood disorders.

[0032] Figure 2 The images are RGB images of three consecutive image sequences of the subject.

[0033] Figure 3 The optical flow sequence is a sequence of three consecutive image sequences of the subject.

[0034] Figure 4 This is a schematic diagram of the two-stream model.

[0035] Figure 5 This is a schematic diagram showing the positions of 68 key points on the face in the facial displacement analysis module.

[0036] Figure 6 This is an example of using emotional disorder-assisted diagnosis in Example 2. Detailed Implementation

[0037] It should be noted that the algorithms for data acquisition, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structures and circuit connections not specifically described, can all be implemented using content already disclosed in the prior art.

[0038] Example 1: Mood Disorder Diagnostic System

[0039] The mood disorder auxiliary diagnostic system of this embodiment includes:

[0040] The data acquisition module is used to acquire and / or input video.

[0041] The attention analysis module is used to analyze the attention of the subject based on the blinking and face positioning success rate in the video, and then to determine whether the video can be used for subsequent auxiliary diagnosis.

[0042] The auxiliary diagnostic module is used to perform micro-expression recognition on videos that are deemed usable, and obtain micro-expression recognition results for auxiliary diagnosis of mood disorders;

[0043] The facial displacement analysis module is used to analyze the specific displacement of specific parts of the face based on the displacement of 68 key points on the face.

[0044] The output module is used to output the judgment results of the attention analysis module, the specific displacement of specific facial parts, and the micro-expression recognition results of the auxiliary diagnosis module.

[0045] Methods for diagnosing mood disorders using the above system, such as Figure 1 As shown.

[0046] In the attention analysis module, the video can be used for subsequent auxiliary diagnostic determination if the following two conditions are met:

[0047] 1) Face localization success rate ≥ 90%; the face localization success rate is calculated as the proportion of frames in the video in which face localization is successful to the total number of frames in the video.

[0048] 2) The time taken to blink 7 times is ≥15 seconds.

[0049] The method for micro-expression recognition in the auxiliary diagnostic module includes the following steps:

[0050] Step 1: Input video. The video length is not fixed; videos ranging from 1 second to 30 minutes are acceptable. In this example, the videos used for training the model range from 33 seconds to 5 minutes, and the acquisition method is as follows:

[0051] The corresponding stimulus videos can elicit corresponding facial expressions. Recording is automatically performed while the subjects watch the videos, with the recorded videos and stimulus videos being synchronized in time. The recorded videos are labeled with six expressions based on the stimulus video: "angry," "disgusted," "fearful," "happy," "sadness," and "neutral," resulting in several video clips.

[0052] During the "watching continuous videos" phase, a baseline approach was used, selecting videos in the following order: baseline-anger-baseline-neutral-baseline-fear-baseline-neutral-baseline-sadness-baseline-neutral-baseline-happiness-baseline-neutral-baseline-aversion-baseline-neutral. Relevant playback segments were selected according to their video categories (anger, fear, sadness, happiness, aversion, and neutral), using a 10-second image with a white background and a fixed cross in the center as the baseline video. While the subjects watched the videos, a computer webcam recorded changes in their facial expressions at a fixed frame rate and pixel size. Each frame of this video corresponded to the duration of the stimulus video, capturing dynamic changes in the subjects' facial expressions at rest and after stimulation.

[0053] After the video is acquired, each frame of the video is separated, and the face is captured in each frame to form a face frame sequence.

[0054] As a preferred embodiment, each frame of the video is processed into a 256x256 grayscale image. The faces are shown as complete facial outlines, and the facial positioning coordinates [x,y,w,h] are adjusted: the lower frame coordinates are increased by 10 pixels to frame the chin, the left frame is shifted 10 pixels to the right, and the right frame is shifted 10 pixels to the left. Ear features are not considered.

[0055] Step 2: Perform difference analysis on two adjacent frames in the face frame sequence to obtain a difference value sequence of the face frame sequence;

[0056] Step 3: Apply a Fourier transform with a Hanning window to the difference value sequence to generate an expression curve;

[0057] Step 4: In the expression curve, multiple expression intervals are divided according to the setting of peaks, troughs and thresholds;

[0058] Step 5: Apply the TV-L1 optical flow algorithm to the face frame sequence to obtain the optical flow sequence;

[0059] Facial expressions exhibit low velocity, constant brightness, and regional consistency. The temporal sequence is the optical flow of three consecutive image sequences for each facial expression interval in the spatial sequence. The optical flow represents the motion vector of facial expression changes. By observing the changes in grayscale distribution of different images in the facial expression sequence, a two-dimensional vector field is obtained.

[0060]

[0061] Where E(x,y,t) represents the gray level in the x and y directions at time t, and E(x+Δx,y+Δx,t+Δt) represents the gray level after a very small time interval Δt. Therefore, the gradient of the pixel gray level along the x, y, and t directions in the image is as follows:

[0062]

[0063] In this embodiment, the processed image is grayscaled using cv2.COLOR_BGR2GRAY, and then the TVL1 optical flow algorithm is applied to two adjacent frames.

[0064] F0 and F1 represent two consecutive frames of images, and a pixel on F0 is represented as P = (x, y). The energy function of the TV-L1 optical flow algorithm is expressed as:

[0065] I0 and I1 represent two consecutive frames of images. A pixel in I0 is represented as X = (x, y). The goal of optical flow is to calculate the visual difference between the two images, which is expressed by the energy function of the TV-L1 optical flow algorithm as follows:

[0066]

[0067] U = (u, v) is a two-dimensional motion field, where ▽u and ▽v are the gradients of the image in the x and y directions, respectively, and U0 means that the gradients in the x and y directions are 0. The parameter λ of the data term is used to adjust the weight values ​​of the data constraints and regularization constraints. The data constraints refer to the I0 and I1 brightness invariance constraints, which refer to the grayscale difference of the same pixel between two adjacent frames. Assuming that the motion is continuous, the second term is the motion regularization constraint.

[0068] In this algorithm, the method for minimizing the total variational optical flow energy function is based on a numerical analysis mechanism that solves the bidirectional problem of image denoising.

[0069] The first step is to perform linearization calculations on the pixels near X+U0 in image frame F1 as follows:

[0070]

[0071] To minimize computational complexity, a first-order Taylor expansion approximation of F1(X+U) was used to replace the function itself. However, this operation linearizes the nonlinear function, making it impossible to obtain an accurate solution. Therefore, a bidirectional solution method was adopted, which compensates for the error caused by the linear approximation by alternately updating the variable values.

[0072] Substituting equation (3) into equation (2), we get:

[0073]

[0074] make:

[0075] ρ(U)=I1(X+U0)+(U-U0)ΔI1-I0 (5)

[0076] Where U` is an external variable, equation (4) can be transformed into the equation that minimizes the convex function as follows:

[0077]

[0078] Where θ is a very small constant, and U' infinitely approximates U. Optimization is performed by alternately updating U' and U, and the equation can then be obtained by transforming it using the following threshold equation.

[0079]

[0080] This project employs a trough sampling method. The TVL1 optical flow algorithm is used on a 3-frame sequence of the spatial stream to obtain the corresponding optical flow sequence. An example of the optical flow sequence is shown below. Figure 2 , 3 As shown, Figure 2 The image consists of RGB images of three consecutive image sequences of the subject. Figure 3 This corresponds to the optical flow sequence. The subject is blinking within this facial expression range; the changes in the eyes can be observed from the optical flow sequence.

[0081] Step 6: Iterate the face frame sequence and optical flow sequence of the expression interval trough, and feed them into the dual-flow model of spatial flow and temporal flow to obtain the expression result of that expression interval.

[0082] The dual-stream model consists of a spatial CNN and a temporal CNN, with the spatial CNN and temporal CNN ultimately fused by a fully connected layer. Feature fusion can be achieved by adding the features according to weighted coefficients or by adding them with different weighted coefficients. This embodiment uses equal weighted coefficients, i.e., 1:1.

[0083] The model parameter settings are shown in the table below:

[0084]

[0085] The structural diagram of the dual-flow model in this embodiment is shown below. Figure 4 As shown, after feature fusion, the data passes through a fully connected layer to form the output layer. This output layer then undergoes a softmax classification function to obtain the probability of each class. The class with the highest probability is the final identified class.

[0086] Compared with different VGG, Inception, and RestNet models, our model has fewer parameters, lower computational cost, and shorter inference time, making it more suitable for scenarios with limited storage space and power consumption. Using the method in this embodiment, the accuracy on the test set is 0.91-0.99 on three datasets: CK+, ce, and jaffeface_7.

[0087] In the facial displacement analysis module, the displacement of 68 key facial points is used to determine micro-expressions in image frames. By comparing these micro-expressions with those of normal individuals in image frames, those exceeding a threshold range are used to aid in the diagnosis of certain emotional disorders. The 68 key facial points are as follows: Figure 5 As shown, the correspondence between the serial numbers and the body parts is as follows (numbering starts from 1): outermost left cheek 1-8, chin 9, outermost right cheek 10-17, left eyebrow 18-22, right eyebrow 23-27, bridge of nose 28-31, tip of nose 31, wing of nose 32-36, left eye 37-40, outer corner of left eye 37, inner corner of left eye 40, right eye 43-46, inner corner of right eye 43, outer corner of right eye 46, mouth 51-68, center of mouth 67, left corner of mouth 49, right corner of mouth 55.

[0088] The method for analyzing the specific displacement of specific facial regions using the facial displacement analysis module includes the following steps:

[0089] Step a: Take the second frame in the video as the resting frame when the subject is at rest, and extract 68 facial key points from the resting frame and the trough frames of all expression intervals.

[0090] Step b: Take the key point on the tip of the nose as the reference point. Assuming that its position does not change when the expression changes, calculate the Euclidean distance between all key points and this reference point as the displacement distance, and obtain a displacement vector with a length of 68.

[0091] Step c: Take the displacement vector of the resting frame as the first row, and from the second row onwards, take the displacement vector of each trough frame as a row to form matrix A;

[0092] Step d: Subtract the resting frame data from each trough frame data to obtain a matrix F, which shows the specific displacement of a particular part of the face.

[0093] The displacements and derivations for each key point are as follows. Assume {P0, P1, P2, ..., P67} represent a sequence of 68 facial key points, and {S1, S2, S3, ..., Sm} represent m intervals. Then the sum of displacements for the nth key point is:

[0094]

[0095] Where n = 0, 1, 2, ..., 67, i = 0, 1, 2, ..., m, and F i，n+1 F represents the value in the (n+1)th row and i-th column of the matrix. This value is the Euclidean distance between the facial key point in the trough frame of each expression interval and the tip of the nose at that moment. 0，n+1 This represents the value in the 0th row and (n+1)th column of the matrix result. This value is the initial distance between the facial key points and the tip of the nose in the resting state.

[0096] Taking the abnormal analysis of subject woman_7 (depressive disorder) watching the stimulating video "Mom, Love Me Again" tagged "sadness" with the video name "VID_11_20201031110900" as an example, the analysis report generated by the output module is as follows: Figure 6 As shown, the results indicate that the system accurately identifies the subject's current emotional state as "sadness".

[0097] As can be seen from the above embodiments, the system of the present invention achieves the purpose of using focus to screen videos, then performing micro-expression recognition on the videos, and using it to assist in the diagnosis of mood disorders, which has a good application prospect.

Claims

1. A diagnostic aid system for mood disorders based on attention analysis and micro-expression recognition, characterized in that, include: The data acquisition module is used to acquire and / or input video. The attention analysis module is used to analyze the attention of the subject based on the blinking and face positioning success rate in the video, and then to determine whether the video can be used for subsequent auxiliary diagnosis. The auxiliary diagnostic module is used to perform micro-expression recognition on videos that are deemed usable, and obtain micro-expression recognition results for auxiliary diagnosis of mood disorders; In the attention analysis module, the video can be used for subsequent diagnostic determination if the following two conditions are met: 1) Facial localization success rate ≥ 90%; 2) The time taken to blink 7 times is ≥15 seconds; The facial displacement analysis module is used to analyze the specific displacement of specific parts of the face based on the displacement of 68 key points on the face. The output module is used to output the judgment results of the attention analysis module, the specific displacement of specific facial parts, and the micro-expression recognition results of the diagnostic module.

2. The mood disorder auxiliary diagnostic system according to claim 1, characterized in that: The face localization success rate is calculated as the proportion of frames in the video where face localization is successful to the total number of frames in the video.

3. The mood disorder auxiliary diagnostic system according to claim 1, characterized in that: The method for micro-expression recognition in the auxiliary diagnostic module includes the following steps: Step 1: Input video, separate each frame of the video, and obtain the face in each frame to form a face frame sequence; Step 2: Perform difference analysis on two adjacent frames in the face frame sequence to obtain a difference value sequence of the face frame sequence; Step 3: Apply a Fourier transform with a Hanning window to the difference value sequence to generate an expression curve; Step 4: In the expression curve, multiple expression intervals are divided according to the setting of peaks, troughs and thresholds; Step 5: Apply the TVL1 optical flow algorithm to the face frame sequence to obtain an optical flow sequence; Step 6: Iterate the face frame sequence and optical flow sequence of the expression interval trough, and feed them into the dual-flow model of spatial flow and temporal flow to obtain the expression result of that expression interval.

4. The mood disorder auxiliary diagnostic system according to claim 3, characterized in that: In step 1, each frame of the video is processed into a 256x256 grayscale image.

5. The mood disorder auxiliary diagnostic system according to claim 3, characterized in that: In step 5, the valley sampling method is adopted, and the TVL1 optical flow algorithm is applied to the 3-frame sequence of the spatial flow.

6. The mood disorder auxiliary diagnostic system according to claim 3, characterized in that: In step 6, the dual-stream model consists of a spatial CNN and a temporal CNN, and the spatial CNN and temporal CNN are finally fused by a fully connected layer.

7. A computer-readable storage medium, characterized in that: It stores a computer program for implementing the system according to any one of claims 1-6.

Images