Personalized emotion automatic detection method based on patient before and after operation

Abstract

The invention discloses a personalized emotion automatic detection method based on a patient before and after an operation, and the method specifically comprises the following steps: 1, carrying out the preprocessing of a patient expression video; 2, discovering change of a human face action unit related to the emotion AU intensity difference; 3, humanized threshold value selection based on the emotion and human face action unit relation; after non-positive face frames of the video in the step (1) are removed, OpenFace mature toolkit software is used for intensity prediction and existence detection of a face action unit; and 4, judging a personalized emotion evaluation application scene through a threshold value of the AU. According to the method, data support such as an accurate emotion change trend chart is provided for clinical emotion management and operation recovery monitoring, meanwhile, the emotion state of a patient is scientifically and quantitatively displayed for a doctor, the doctor is assisted in performing subsequent decision diagnosis, and the treatment effect is assisted to be improved.

Description

Technical Field

[0001] The present invention relates to the field of medical auxiliary equipment, and in particular to an automatic detection method based on the personalized emotions of patients before and after surgery. Background Art

[0002] In order to meet the high requirements of the accuracy of patient emotion assessment in the medical field, especially before and after surgery, the patient's emotional perception has an important impact on their recovery process. In addition, some patients have a close relationship between emotions and facial expressions, and many emotions will cause specific changes in facial expressions.

[0003] These include:

[0004]

[0005] At present, most of the emotion assessment methods on the market rely on the subjective feedback of patients, and these methods have certain limitations in clinical practice. The subjective perception of patients varies greatly from person to person, resulting in inaccurate emotion assessment results, and patients may not be able to provide effective feedback in some cases (such as language barriers, inability to express). In recent years, with the rapid development of artificial intelligence and computer vision technology, emotion assessment technology based on automatic recognition of facial expressions has gradually received attention. Facial action unit coding technology can identify and quantify subtle movements of facial muscles, and infer patients' emotional state by analyzing their facial expression changes in different situations (and the corresponding changes in facial action units). This automated emotion change assessment method based on facial expressions can make up for the shortcomings of traditional methods and provide more objective assessment data. However, existing facial expression emotion recognition technology still faces several challenges. For example, the personalized emotional difference assessment of patients before and after surgery has not been fully addressed, and existing technologies have failed to automatically quantify the differences between patients' emotional levels by combining individual characteristics. In particular, during the recovery period of surgery, the dynamic changes in patients' emotional perception also need to be monitored and analyzed by more sophisticated technical means, which cannot better assist doctors in subsequent decision-making and diagnosis, resulting in doctors failing to achieve the best treatment effect on patients before and after surgery. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a method for automatically detecting the patient's personalized emotions before and after surgery, which provides accurate data support such as emotion change trend charts for clinical emotion management and surgical recovery monitoring, and scientifically quantifies the patient's emotional state for display to doctors, assists them in subsequent decision-making and diagnosis, and helps improve treatment.

[0007] To achieve the above purpose, the present invention provides a method for automatically detecting the personalized emotions of patients before and after surgery, wherein the specific steps are as follows:

[0008] Step 1: Preprocess the patient's expression video;

[0009] (1) Select patients who need surgery;

[0010] (2) Video recording of the facial expressions of patients who are about to undergo surgery, which should include the patient's expressions in different emotional states;

[0011] (3) Use the recorded video to estimate the patient's facial posture, align face detection, and perform cropping;

[0012] By extracting facial feature points and performing rotation correction, it is possible to accurately detect frontal faces, crop and align facial images, and ensure that the output images have a uniform angle and size.

[0013] Step 2: discover changes in facial action units related to the intensity differences of emotional AUs;

[0014] Step 3: Select a humanized threshold based on the relationship between emotion and facial action unit. After removing the non-positive face frames from the video in step 1 (1), the OpenFace mature toolkit software is used to predict the intensity and detect the presence of facial action units.

[0015] The intensity prediction range is 0-5;

[0016] The presence detection is presence = 1 or absence = 0;

[0017] Step 4: Determine the application scenario of personalized emotion assessment through the threshold of AU;

[0018] When the AU thresholds of the current patient are respectively in the stable and unstable emotional states, the AU threshold information of the current patient in the stable and unstable emotional states is calculated based on the above steps, and the difference between the AU information of the video frame recorded by the patient after the operation and the two thresholds can be judged according to the AU threshold information to determine the state of the patient; then, the evaluation results are displayed on a display board to the doctor, so as to help him monitor the patient's recovery degree and provide rich patient emotional information for his further diagnosis of emotional management.

[0019] In some implementations, the video recording in the above step 1 uses a fixed camera to capture video frames of the front part of the human face.

[0020] In some embodiments, in the above step 4, when the threshold values ​​of the AUs of the current patient in the emotionally stable and unstable states are respectively calculated, for this application scenario, it is necessary to sum the seven types of AU intensities of the patient in the emotionally stable and unstable states, that is, to obtain the total AU intensity threshold for evaluating the patient's emotions in the two states.

[0021] In some implementations, the preprocessing process of the patient's expression video frame in step 1 is as follows:

[0022] A. Facial feature point detection

[0023] Use the face detection toolkit to perform face detection and facial key point extraction on the input video frame; that is, call the face detection module and load the pre-trained model for face detection and feature point extraction; the model detects facial feature points, which correspond to the important facial areas of eyes, nose, and mouth;

[0024] B. Pose Estimation

[0025] From the feature points detected on the face, select key points related to posture estimation, such as the corners of the eyes, the tip of the nose, and the corners of the mouth; use these key points and the preset 3D facial model points to calculate the rotation vector and translation vector of the face through the posture estimation algorithm, and then obtain the pitch angle, yaw angle, and roll angle of the face;

[0026] C. Frontal face screening

[0027] According to the posture estimation results, the yaw angle and pitch angle are determined; when the yaw angle and pitch angle are less than a certain arc, the face is determined to be frontal;

[0028] D. Face cropping

[0029] The cropping area is determined based on the positions of the eyes, mouth, and chin of the face. The cropping area ensures complete coverage of the face while maintaining an even distribution of the face area. The cropped image output is used as input for subsequent recognition or analysis.

[0030] E. Output the cropped face image

[0031] The final output is an aligned and cropped face image, ensuring the uniform position and posture of the face in the image for use in subsequent tasks.

[0032] In some embodiments, the humanized threshold for the relationship between emotion and facial action unit in step 3 is selected as follows:

[0033] (1) Time series of AU data

[0034] Based on obtaining the AU information of each frame, these data can be regarded as a time series; the difference of the specified AU data between all frames is calculated; the data of each frame can be represented as a vector of a specific dimension;

[0035] (2) Convolution

[0036] The convolution kernel can be used to calculate the AU difference between the current frame and the previous and next frames. The function of this convolution kernel is: through the convolution operation, we can detect the change of AU data in time, similar to edge detection.

[0037] (3) Applying convolutional thinking to AU data

[0038] The intensity and frequency of each AU are convolved separately. In essence, the gradient of the frequency change of adjacent frames is calculated to capture the drastic changes in the current sliding window / time series (local AU intensity mutation frame); for each dimension of AU data (such as the intensity of AU01), the convolution kernel is used to calculate the difference between the current frame and the previous and next frames; assuming three frames of data:

[0039] Frame 1: [AU01_1,AU02_1,AU03_1,AU04_1…,AU01_f_1,..]

[0040] Frame 2: [AU01_2,AU02_2,AU03_2,AU04_2…,AU01_f_2,…]

[0041] Frame 3: [AU01_3,AU02_3,AU03_2,AU04_2…,AU01_f_3,…]

[0042] Convolution of the intensity of AU01 indicates the intensity change of AU01 in the second frame. Similar calculations are performed on all dimensions such as the intensity of AU02 and AU03 to obtain a new time series representing the intensity change of AU data in each frame.

[0043] (4) Convolution operation in sliding window

[0044] By using the convolution kernel, the convolution calculation of the AU intensity difference between the previous and next frames of each frame can be performed in each sliding window, and the AU intensity change information and expression change of each frame can be obtained; then, the frame with the largest AU intensity change in this window is the moment of emotional mutation in the window; that is, in each sliding window, the position of the frame corresponding to the peak of an AU change can be obtained from the expression video recorded by the patient, and the AU information on the corresponding frame can be obtained, and the mean / threshold can be calculated; the AU information corresponding to the frames at the moment of emotional mutation in this video can be combined to obtain the mean / threshold of the AU intensity of the current patient at all emotional mutation moments in the current video; similarly, the mean of the AU intensity change calculated in all frames other than these emotional mutation moments is used as the mean AU intensity (threshold) of the patient in a stable emotional state; the convolution formula is as follows:

[0045]

[0046] Among them, S(t) is the output value after convolution, which represents the total change value of the AU in the t-th frame; W(i) is the weight of the convolution kernel, which can be regarded as a filter to capture local changes; X(t+i) is the AU data of the t+i-th frame; k is the size of the convolution kernel.

[0047] The beneficial effect of the present invention is that it provides accurate data support such as emotion change trend charts for clinical emotion management and surgical recovery monitoring, and scientifically quantifies the patient's emotional state for doctors to assist them in making subsequent decision-making diagnoses and improve the treatment effect. Because it uses facial action unit coding technology, combined with the Sobel edge detection algorithm and the sliding window mechanism, the emotional mutation points in the patient's facial expression are detected to obtain the action coding threshold under the relevant emotional state, and finally the difference in emotional degree under different states before and after the operation can be automatically quantified. The technology can not only effectively deal with the background and face posture problems in the video frame, but also build a personalized emotion assessment model, provide accurate data support such as emotion change trend charts for clinical emotion management and surgical recovery monitoring, and scientifically quantify the patient's emotional state for doctors to assist them in making subsequent decision-making diagnoses. Specifically: (1) Facial action unit (AU) coding. Based on the facial frontal action unit coding technology, a specific tool is used to extract facial action units related to specific emotions, including contraction of the orbicularis oculi muscle and tightening of the eyebrows. By comparing the changes in the intensity activity values ​​of the frontal action units in the videos recorded before and after the surgery, the differences in the degree of emotions (such as pain) are quantified to reflect the degree of recovery. (2) Face processing technology in video frames. Advanced image processing technology is used to effectively remove the background other than the face in the video frame and crop only the front face. The uniformity of the face at various angles is ensured by geometric correction of the skewed face. At the same time, the non-frontal face filtering algorithm is applied to focus on analyzing the frontal face to obtain more accurate action unit (AU) encoding. (3) Sobel edge detection algorithm. The Sobel edge detection algorithm is a classic image processing algorithm used to detect the area with rapid changes in grayscale values ​​in an image, that is, the edge of the image. The algorithm calculates the local gradient of the pixels in the image and calculates the change rate in the horizontal and vertical directions respectively, thereby identifying the significant change points in the image. The idea of ​​the Sobel algorithm is introduced into the analysis of the changes in facial action units in the expression videos recorded by patients. By setting a sliding window mechanism and identifying the mutation point of the emotion level (the specific frame position), the AU information of the patient in the two states of emotional stability and instability (only two extreme levels are identified in the present invention) can be calculated. Thus, the AU threshold of the individual user of the current patient under the two emotions is determined. (4) Emotion assessment technology based on facial expressions. In the emotion assessment technology based on facial expressions, by analyzing the changes in the patient's facial expressions and using existing tools to evaluate the action unit information, quantitative results of emotional changes before and after surgery are provided. This technology accurately evaluates the patient's personalized emotional perception based on the dynamic changes of facial micro-expressions and action units. Therefore, with the development of intelligent medical care, it has broad application prospects in the future and can be used for postoperative rehabilitation management.By combining facial expression analysis with emotion assessment technology, more accurate and personalized emotional care can be achieved, improving the patient's medical experience. It provides accurate emotional change trend charts and other data support for clinical emotional management and surgical recovery monitoring, and scientifically quantifies the patient's emotional state for doctors to assist them in subsequent decision-making and diagnosis, and to help improve the effectiveness of treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flowchart for preprocessing patient expression video frames;

[0049] Figure 2 It is a chart showing the emotional AU information board;

[0050] Figure 3 This is a curve chart of the patient's mood changes after surgery. DETAILED DESCRIPTION

[0051] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0052] A method for automatically detecting personalized emotions of patients before and after surgery, the specific steps are as follows:

[0053] (1) Preprocessing method of patient expression video

[0054] After selecting a specific group of patients who need to undergo a certain type of surgery, a facial expression video is recorded among the group of people who are about to undergo the surgery. The video should include the patient's expressions under different emotional states. The recording process here includes capturing video frames of the front of the face with a fixed camera. The present invention provides a face detection and cropping method based on facial posture estimation and alignment, which utilizes facial feature point extraction and rotation correction to accurately detect the front face, and crop and align the face image to ensure that the output image has a uniform angle and size. See the patient expression video frame preprocessing process (such as Figure 1 shown):

[0055] ●Facial feature point detection

[0056] ■ Use the face detection toolkit to perform face detection and facial key point extraction on the input video frames. Specifically, call the face detection module and load the pre-trained model for face detection and feature point extraction. The model detects facial feature points, which correspond to important facial areas such as eyes, nose, and mouth.

[0057] ●Pose estimation

[0058] ■ From the detected feature points, select key points related to posture estimation (such as eye corners, nose tip, mouth corners, etc.). Using these key points and the preset 3D facial model points, the posture estimation algorithm is used to calculate the rotation vector and translation vector of the face, and then the pitch angle, yaw angle and roll angle of the face are obtained.

[0059] ●Front-facing face screening

[0060] ■Based on the posture estimation results, determine the yaw angle and pitch angle. When the yaw angle and pitch angle are less than a certain arc, the face is determined to be frontal.

[0061] ●Face cropping

[0062] ■Determine the cropping area based on the position of the eyes, mouth, and chin of the face. The cropping area ensures complete coverage of the face while maintaining an even distribution of the face area. The cropped image output is the input for subsequent recognition or analysis.

[0063] ● Output cropped face image

[0064] ■Finally output the aligned and cropped face image to ensure the uniform position and posture of the face in the image for use in subsequent tasks.

[0065] (2) The facial action units most relevant to the intensity difference of emotional AUs selected in the present invention

[0066] For specific emotional areas, it is necessary to first determine the action units that are more related to such emotional changes (such as pain levels) and facial expression changes through research or with the help of medicine. In the present invention, considering that the pain expressions produced by parturients during uterine contractions are more obvious and have certain emotional change characteristics, the video data of the relevant number of parturients who signed the relevant agreement in the two extreme emotions (labeled by medical experts) recorded on their faces in the state of waiting for delivery are used to calculate the AU information through the OpenFace software, and from the average quantified difference values ​​of the relevant AU intensities of these parturients under these two extreme pain levels, several AUs with obvious changes can be found.

[0067] (3) Personalized threshold selection based on the relationship between emotion and facial action units

[0068] Before the surgery, the patient was asked to record a video. After the non-positive face frames were removed from the video, the OpenFace mature toolkit software was used to predict the intensity of facial action units (range 0-5) and existence detection (existence = 1, absence = 0). The technology first accurately detects facial landmarks through the convolutional expert restricted local model (CE-CLM) and extracts the HOG features and geometric shape features of the face. Then, the linear kernel support vector machine (SVM) model is used to predict the continuous value of the intensity of each AU and whether it exists or not. At the same time, the method has been jointly trained on multiple face datasets to ensure a certain robustness and accuracy of the model.

[0069] Therefore, for each video frame in a patient's facial video, OpenFace software is used to detect the intensity and presence of a specified AU associated with a specific emotional change. In order to automatically obtain several moments (frame positions) in this video where emotional changes are obvious, the present invention borrows the idea of ​​sliding window and Sobel edge detection algorithm. The length of the sliding window is set to T frames (adjustable). Each time the window slides, a convolution operation based on the selected AU information will be performed in the window. In other words, the position of the frame with the largest emotional change (the most drastic change in AU data) must be found in each window. This process is similar to detecting "edges". By finding the frame with the largest change and the AU information corresponding to the frame, the position of the frame with sudden emotional changes can be determined. Then, together with the AU information of the two frames before and after this frame, the threshold (mean) of the AU information of the sudden emotional change in the current window can be calculated.

[0070] Time series of AU data

[0071] Based on the AU information of each frame, these data can be regarded as a time series. The difference of the specified AU data between all frames is calculated. The data of each frame can be represented as a vector of a specific dimension.

[0072] convolution

[0073] The convolution kernel can be used to calculate the AU difference between the current frame and the previous and next frames. The function of this convolution kernel is: through the convolution operation, we can detect the change of AU data in time, similar to edge detection.

[0074] Applying convolution ideas to AU data

[0075] The intensity and frequency of each AU are convolved separately. In essence, the gradient of the frequency change of adjacent frames is calculated to capture the drastic changes in the current sliding window / time series (local AU intensity mutation frame). For each dimension of AU data (such as the intensity of AU01), the convolution kernel is used to calculate the difference between the current frame and the previous and next frames. Assume three frames of data:

[0076] ■ Frame 1: [AU01_1,AU02_1,AU03_1,AU04_1…,AU01_f_1,..]

[0077] ■ Frame 2: [AU01_2,AU02_2,AU03_2,AU04_2…,AU01_f_2,…]

[0078] ■ Frame 3: [AU01_3,AU02_3,AU03_2,AU04_2…,AU01_f_3,…]

[0079] Convolution of the intensity of AU01 indicates the intensity change of AU01 in frame 2. Similar calculations are performed on all dimensions such as the intensity of AU02 and AU03 to obtain a new time series representing the intensity change of AU data in each frame.

[0080] Convolution operation in sliding window

[0081] By using the convolution kernel, the convolution calculation of the AU intensity difference between the previous and next frames of each frame can be performed in each sliding window, so the AU intensity change information (expression change) of each frame can be obtained. Then, the frame with the largest AU intensity change in this window is the moment of sudden emotion change in this window. In other words, in each sliding window, the position of the frame corresponding to the peak of an AU change can be obtained from the expression video recorded by the patient (as well as the AU information on the corresponding frame, the mean / threshold can be calculated). Then, by combining the AU information corresponding to the frames at all moments of sudden emotion change in this video, the mean / threshold of the AU intensity of the current patient at all moments of sudden emotion change in the current video can be obtained. Similarly, the mean of the AU intensity change calculated in all frames other than these frames at the moment of sudden emotion change is used as the mean AU intensity (threshold) of the patient in a stable emotional state. The convolution formula is as follows:

[0082]

[0083] Among them, S(t) is the output value after convolution, which represents the total change value of AU in the t-th frame. W(i) is the weight of the convolution kernel, which can be regarded as a filter to capture local changes. X(t+i) is the AU data of the t+i-th frame. k is the size of the convolution kernel.

[0084] (4) Application scenarios of personalized emotion assessment

[0085] The threshold of AU based on the patient's current emotional stability and instability

[0086] Based on the previous steps, the AU threshold information of the current patient in the stable and unstable emotional state is calculated, and the state of the patient can be judged according to the difference between the AU information of the video frame recorded after the operation and the two thresholds. The evaluation results are presented to the doctor to help him monitor the patient's recovery degree and provide rich patient emotional information for his further diagnosis of emotional management (such as Figure 2 The update frequency of the display board can be customized by the doctor, for example, every 20 seconds. Based on the sum of the thresholds of the patient's AUs in the emotionally stable and unstable states

[0087] For this application scenario, it is necessary to sum the seven types of AU intensities (excluding the number of occurrences) of the patient in stable and unstable emotional states, that is, to obtain the threshold for the sum of AU intensities in the two states (to evaluate the patient's emotions). As shown in the figure below, every 10 seconds, the doctor can observe the sum of the AU intensities of the current patient, and based on the baseline, the changing trend of the patient's emotions can be known. If the incoming video is of the patient after surgery, then the degree of recovery or effect of the patient can be known (such as Figure 3 As shown). The invention of a light-delivery personalized emotion change assessment system has flexible application scenarios. Videos of patient expressions in stable and unstable emotional states can be uploaded directly and simply without the need for manual cropping of the video clips in stable and unstable emotional states. The invention can automatically obtain the corresponding AU information and feature information of the two unique emotional states of the current patient. It is used for subsequent personalized emotion assessment of patients, making the emotion assessment based on patient expressions more targeted. Personalized emotion difference visualization is suitable for clinical emotion information display and personalized patient recovery curve auxiliary diagnosis function.

[0088] The present invention performs preprocessing directly based on the collected face video data, and uses the existing mature action unit analysis software OpenFace to achieve fast and lightweight action unit coding extraction for facial expression recognition under different emotional levels. At the same time, it only needs to adjust the relevant AU types corresponding to the face under different emotions to achieve the detection task of different degrees of emotional changes in patients. This method is free and flexible, and effectively avoids the dependence on independently trained deep learning models, reduces excessive dependence on the quality of training data sets, and ensures the accuracy and reliability of coding analysis. In actual medical systems, pain management is deeply integrated with clinical applications, and common means of solving pain include injecting analgesics or performing surgery on patients. Therefore, quantitative assessment of pain for individuals is of great significance. By comparing the difference curves of the patient's expression and action coding before and after surgery, the patient's pain level and state can be more accurately reflected, thereby providing a more personalized pain management solution.

[0089] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all belong to the protection scope of the invention.

Claims

1. A method for automatically detecting personalized emotions of patients before and after surgery, wherein: The specific steps are as follows: Step 1: Preprocess the patient's expression video; (1) Select patients who need surgery; (2) Video recording of the facial expressions of patients who are about to undergo surgery, which should include the patient's expressions in different emotional states; (3) Use the recorded video to estimate the patient's facial posture, align face detection, and perform cropping; By extracting facial feature points and performing rotation correction, it is possible to accurately detect frontal faces, crop and align facial images, and ensure that the output images have a uniform angle and size. Step 2: discover changes in facial action units related to the intensity differences of emotional AUs; Step 3: Select a humanized threshold based on the relationship between emotion and facial action unit. After removing the non-positive face frames from the video in step 1 (1), the OpenFace mature toolkit software is used to predict the intensity and detect the presence of facial action units. The intensity prediction range is 0-5; The presence detection is presence = 1 or absence = 0; Step 4: Determine the application scenario of personalized emotion assessment through the threshold of AU; When the AU thresholds of the current patient are respectively in the stable and unstable emotional states, the AU threshold information of the current patient in the stable and unstable emotional states is calculated based on the above steps, and the state of the patient can be determined by judging the difference between the AU information of the video frame recorded after the operation and the two thresholds according to the AU threshold information; then, the evaluation results are displayed on a display board to the doctor, so as to help him monitor the patient's recovery degree and provide rich patient emotional information for his further diagnosis of emotional management.

2. The method for automatically detecting the patient's personalized emotions before and after surgery according to claim 1, characterized in that: The video recording in step 1 above uses a fixed camera to capture video frames of the front face of the person.

3. The method for automatically detecting the patient's personalized emotions before and after surgery according to claim 1 or 2, characterized in that: In the above step 4, when the threshold values ​​of the AUs of the current patient in the stable and unstable emotional states are summed, for this application scenario, it is necessary to sum the seven types of AU intensities of the patient in the stable and unstable emotional states, that is, to obtain the total AU intensity threshold for evaluating the patient's emotions in the two states.

4. The method for automatically detecting the patient's personalized emotions before and after surgery according to claim 2, characterized in that: The preprocessing process of the patient's expression video frame in step 1 is as follows: A. Facial feature point detection Use the face detection toolkit to perform face detection and facial key point extraction on the input video frame; that is, call the face detection module and load the pre-trained model for face detection and feature point extraction; the model detects facial feature points, which correspond to the important facial areas of eyes, nose, and mouth; B. Pose Estimation From the feature points detected on the face, select key points related to posture estimation, such as the corners of the eyes, the tip of the nose, and the corners of the mouth; use these key points and the preset 3D facial model points to calculate the rotation vector and translation vector of the face through the posture estimation algorithm, and then obtain the pitch angle, yaw angle, and roll angle of the face; C. Frontal face screening According to the posture estimation results, the yaw angle and pitch angle are determined; when the yaw angle and pitch angle are less than a certain arc, the face is determined to be frontal; D. Face cropping The cropping area is determined based on the positions of the eyes, mouth and chin of the face; the cropping area ensures complete coverage of the face while maintaining a uniform distribution of the face area; the cropped image output is used as input for subsequent recognition or analysis; E. Output the cropped face image The final output is an aligned and cropped face image, ensuring the uniform position and posture of the face in the image for use in subsequent tasks.

5. The method for automatically detecting the patient's personalized emotions before and after surgery according to claim 1, characterized in that: The above step 3 selects the humanized threshold value for the relationship between emotion and facial action unit: (1) Time series of AU data Based on obtaining the AU information of each frame, these data can be regarded as a time series; the difference of the specified AU data between all frames is calculated; the data of each frame can be represented as a vector of a specific dimension; (2) Convolution The convolution kernel can be used to calculate the AU difference between the current frame and the previous and next frames; the function of this convolution kernel is: through the convolution operation, we can detect the change of AU data in time, similar to edge detection; (3) Applying convolutional thinking to AU data The intensity and frequency of each AU are convolved separately. In essence, the gradient of the frequency change of adjacent frames is calculated to capture the drastic changes in the current sliding window / time series (local AU intensity mutation frame); for each dimension of AU data, the convolution kernel is used to calculate the difference between the current frame and the previous and next frames; assuming three frames of data: Frame 1: [AU01_1,AU02_1,AU03_1,AU04_1…,AU01_f_1,..] Frame 2: [AU01_2,AU02_2,AU03_2,AU04_2…,AU01_f_2,…] Frame 3: [AU01_3,AU02_3,AU03_2,AU04_2…,AU01_f_3,…] Convolution of the intensity of AU01 indicates the intensity change of AU01 in the second frame. Similar calculations are performed on all dimensions of the intensity of AU02 and AU03 to obtain a new time series representing the intensity change of AU data in each frame. (4) Convolution operation in sliding window By using the convolution kernel, the convolution calculation of the AU intensity difference between the previous and next frames of each frame can be performed in each sliding window, and the AU intensity change information and expression change of each frame can be obtained; then, the frame with the largest AU intensity change in this window is the moment of emotional mutation in the window; that is, in each sliding window, the position of the frame corresponding to the peak of an AU change can be obtained from the expression video recorded by the patient, and the AU information on the corresponding frame can be obtained, and the mean / threshold can be calculated; the AU information corresponding to the frames at the moment of emotional mutation in this video can be combined to obtain the mean / threshold of the AU intensity of the current patient at all emotional mutation moments in the current video; similarly, the mean of the AU intensity change calculated in all frames other than these emotional mutation moments is used as the mean AU intensity (threshold) of the patient in a stable emotional state; the convolution formula is as follows: Among them, S(t) represents the output value after convolution, which represents the total change value of AU in the t-th frame; W(i) is the weight of the convolution kernel, which can be regarded as a filter to capture local changes; X(t+i) is the AU data of the t+i-th frame; k is the size of the convolution kernel.

Images