Stroke face auxiliary evaluation method and system based on face action unit detection

Abstract

The invention provides a stroke face auxiliary evaluation method and system based on face motion unit detection, and the method comprises the steps: collecting a face motion video of a patient, and recording a start frame and an end frame of each motion in the video; the actions comprise an eyeball horizontal rotation action, a tooth indication action, an eyebrow lifting action and an eye closing action; according to the start frame and the end frame of the horizontal rotation eyeball action, extracting a horizontal rotation eyeball action video, namely an eye rotation action video, and calculating to obtain a severity score of the staring condition of the patient by using a staring scoring algorithm; and respectively extracting videos of the three actions according to starting frames and ending frames of the tooth indicating action, the eyebrow lifting action and the eye closing action, and calculating to obtain a severity score of the facial paralysis condition of the patient by using a facial paralysis scoring algorithm. According to the method, the abnormal conditions of the face of the patient are captured through accurate detection and quantification of various face actions of the patient, the problems that a conventional evaluation mode is low in accuracy, incomplete in evaluation and the like are solved, and an objective, accurate and comprehensive stroke face evaluation effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of stroke facial auxiliary assessment, and in particular to a stroke facial auxiliary assessment method and system based on facial action unit detection. Background Art

[0002] Facial paralysis includes central facial paralysis and peripheral facial paralysis. Central facial paralysis is one of the serious sequelae after stroke. About 70% of stroke patients develop central facial paralysis after the disease, and 30% of them develop permanent facial paralysis. The clinical manifestations are mainly characterized by movement disorders of the facial expression muscles below the orbit on the opposite side of the lesion, and facial deviation is the main symptom. Peripheral facial paralysis, also known as Bell's palsy or facial neuritis, is a peripheral facial muscle paralysis caused by nonspecific inflammation of the facial nerve in the facial nerve canal. Peripheral facial paralysis includes the entire face. The general symptoms are facial deviation, inability to complete actions such as raising eyebrows, closing eyes, and puffing mouth. Both central and peripheral facial paralysis affect the patient's aesthetics and are often accompanied by symptoms such as dysarthria, drooling, pinching food, and swallowing and chewing weakness. It is a stubborn disease that is difficult to cure clinically.

[0003] If a timely diagnosis and the most accurate auxiliary assessment can be obtained before this, this situation can be avoided with a high probability and the possibility of recovery can be increased.

[0004] Patent document CN113782184A discloses a stroke auxiliary assessment system based on facial key points and feature pre-learning, including a data acquisition module, a key point positioning model training module, a feature pre-learning model training module, and a rapid diagnosis and assessment model construction module. First, feature pre-learning is performed in a public large-scale facial paralysis data set, and then targeted feature learning is performed on a small-sample stroke data set to obtain highly descriptive stroke facial change image features. Secondly, a facial multi-key point positioning algorithm is used to segment and locate the stroke facial area, segment the eye and mouth areas, and send the obtained eye and mouth areas to a convolutional neural network for feature extraction. Finally, the features of different areas are classified to obtain a quantitative standard for stroke facial assessment.

[0005] However, the model output only has abnormal categories, such as mild eyes and severe mouth, etc., with low precision and interpretability, and lacks detection of gaze abnormalities. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a method and system for auxiliary stroke facial assessment based on facial action unit detection.

[0007] A method for assisting facial assessment of stroke based on facial action unit detection provided by the present invention comprises:

[0008] Step S1: collecting a video of the patient's facial movements, and recording the start frame and the end frame of each movement in the video; the movements include horizontal eyeball rotation, tooth showing, eyebrow raising, and eye closing;

[0009] Step S2: extracting a horizontal eye movement video, i.e., an eye movement video, according to the start frame and the end frame of the horizontal eye movement, and using a gaze scoring algorithm to calculate a severity score of the patient's gaze condition;

[0010] Step S3: according to the start frames and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, respectively, extract the videos of the three actions, and use the facial paralysis scoring algorithm to calculate the severity score of the patient's facial paralysis.

[0011] Preferably, the horizontal eye rotation action refers to the patient's horizontal eye rotation to the extreme positions of the left and right sides of the eye socket, and the eye rotation from the leftmost end to the rightmost end and then back to the leftmost end is counted as one time, and is repeated 3 to 4 times;

[0012] The tooth-showing action refers to the patient grinning hard;

[0013] The eyebrow-raising action refers to the patient's eyebrows being raised upwards on both sides at the same time;

[0014] The eye-closing action refers to the patient closing both eyes forcefully at the same time.

[0015] Preferably, the severity score of the patient's gaze condition includes three scores, namely 0, 1 and 2, which respectively represent normal, partial gaze paralysis, passive gaze or complete gaze paralysis;

[0016] The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2 and 3, which respectively represent: normal, minimal facial paralysis, partial facial paralysis and complete facial paralysis.

[0017] Preferably, the step S2 comprises:

[0018] Step S2.1: taking the eye-rolling action video as input to select feasible frames and obtain a feasible frame set;

[0019] Step S2.2: extracting eye-open frames from each frame image of the feasible frame set through an eye-closing detection model to form an eye-open frame set;

[0020] Step S2.3: Detect facial feature points for each frame in the eye-opening frame set, and determine whether the eyeball of the person has moved by the eye area feature points therein. If so, execute step S2.4; if not, output 2 points. The method for determining whether the eyeball of the person has moved includes calculating the ratio of the distance between the pupil center point and the left and right eye corners by the eye area feature points, and calculating the standard deviation of the ratio in all images. If the standard deviation is less than the threshold, it indicates that the eyeball has not moved; otherwise, it indicates that the eyeball has moved;

[0021] Step S2.4: each frame of the eye-opening frame set is passed through the eye posture recognition model to obtain frame-by-frame eye posture recognition results for the left eye and the right eye, and then the number of frames in which the eyeball is located at the inner and outer positions of the eye socket respectively;

[0022] Step S2.5: Determine whether the number of frames is greater than the threshold N1. If so, output 0 points; if not, output 1 point.

[0023] Preferably, step S3 includes: tooth showing detection, eyebrow raising detection and eye closing detection, and outputs a specific score according to the abnormal results of tooth showing detection, eyebrow raising detection and eye closing detection respectively through facial paralysis scoring logic.

[0024] Preferably, the tooth-showing detection comprises: inputting the tooth-showing action video into the mouth corner movement detection algorithm to obtain each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence; calculating the left and right side strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting two thresholds A and B to judge the asymmetry degree of the left and right side actions, corresponding to the abnormal situation judgment of the tooth-showing action, which are normal tooth-showing / slightly abnormal tooth-showing / severely abnormal tooth-showing respectively;

[0025] The eyebrow raising detection comprises: inputting the eyebrow raising action video into the eyebrow raising detection algorithm, obtaining each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence; calculating the left and right side strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting a certain threshold C to judge the asymmetry degree of the left and right side actions, and judging the abnormal situation of the corresponding eyebrow raising action, namely, normal eyebrow raising / abnormal eyebrow raising;

[0026] The eye closure detection includes using a feasible frame selection algorithm on the eye closure action video, extracting all feasible frames from the video to form a feasible frame set, passing each frame image of the feasible frame set through an eye closure detection model, obtaining frame-by-frame eye closure state detection results for the left eye and the right eye, and then obtaining the sum of the number of frames for lightly closed and tightly closed eyes, setting a threshold D, and judging for each eye whether the sum of the number of frames is greater than the threshold D, corresponding to the abnormal conditions of the eye closure action, which are normal eye closure and abnormal eye closure, respectively.

[0027] Preferably, the mouth corner movement detection algorithm includes: obtaining all continuous feasible frame sequences in the video through a feasible frame selection algorithm, and then, for each sequence, extracting facial feature points for each frame image therein, using some feature points corresponding to the detected action, calculating the movement amplitudes of the some feature points on the left and right sides respectively, as the left and right action unit intensity outputs.

[0028] Preferably, the facial paralysis scoring logic is specifically as follows: if the teeth are normal, 0 points are output; if the teeth are slightly abnormal, 1 point is output; if the teeth are severely abnormal, eyebrow raising is normal, and eye closing is normal, 2 points are output; if the teeth are severely abnormal, eyebrow raising is abnormal, and eye closing is abnormal, 3 points are output.

[0029] According to the present invention, a facial stroke auxiliary assessment system based on facial action unit detection includes:

[0030] Module M1: Collect the patient's facial movement video and record the start frame and end frame of each action in the video; the actions include horizontal eye movement, tooth showing, eyebrow raising, and eye closing;

[0031] Module M2: extracting the horizontal eye movement video, i.e., the eye movement video, according to the start frame and the end frame of the horizontal eye movement, and using the gaze scoring algorithm to calculate the severity score of the patient's gaze condition;

[0032] Module M3: According to the start and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, the videos of the three actions are extracted respectively, and the facial paralysis scoring algorithm is used to calculate the severity score of the patient's facial paralysis.

[0033] Preferably, the horizontal eye rotation action refers to the patient's horizontal eye rotation to the extreme positions of the left and right sides of the eye socket, and the eye rotation from the leftmost end to the rightmost end and then back to the leftmost end is counted as one time, and is repeated 3 to 4 times;

[0034] The tooth-showing action refers to the patient grinning hard;

[0035] The eyebrow-raising action refers to the patient's eyebrows being raised upwards on both sides at the same time;

[0036] The eye-closing action refers to the patient closing both eyes forcefully at the same time.

[0037] The severity score of the patient's gaze condition includes three scores, 0, 1, and 2, which respectively represent normal, partial gaze paralysis, passive gaze, or complete gaze paralysis;

[0038] The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2 and 3, which respectively represent: normal, minimal facial paralysis, partial facial paralysis and complete facial paralysis.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention captures abnormal conditions of the patient's face by accurately detecting and quantifying a variety of facial movements of the patient, thereby solving the problems of low accuracy and incomplete evaluation of conventional evaluation methods, and achieving an objective, accurate and comprehensive stroke facial evaluation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0042] Figure 1 It is a schematic diagram of the working method of the present invention;

[0043] Figure 2 Schematic diagram of the gaze scoring algorithm flow of the present invention;

[0044] Figure 3 It is a schematic flow chart of a facial paralysis scoring algorithm of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0046] The present invention uses facial movement videos of stroke patients to achieve accurate detection of patients' facial movements and quantitative analysis of patients' gaze symptoms and facial paralysis symptoms, ultimately achieving auxiliary assessment of stroke conditions.

[0047] Embodiment 1

[0048] According to a stroke facial auxiliary assessment method based on facial action unit detection provided by the present invention, Figure 1 As shown, including:

[0049] Step S1: Collect the patient's facial movement video and record the start frame and end frame of each action in the video. The actions include horizontal eye rotation, tooth showing, eyebrow raising, and eye closing. The horizontal eye rotation refers to the patient's horizontal eye rotation to the extreme positions on the left and right sides of the eye sockets. The eyeball moves from the leftmost end to the rightmost end and then back to the leftmost end as 1 time, and repeats 3 to 4 times. The process of collecting horizontal eye rotation movements can be completed by the patient independently or under the guidance of medical staff. For example, use a small object with bright colors or bright light to move left and right in front of the patient's face to guide the patient to continue to stare at the object and complete the above-mentioned eye rotation process. The tooth showing action refers to the patient grinning hard. The eyebrow raising action refers to the patient's eyebrows being raised hard at the same time. The eye closing action refers to the patient's eyes being closed hard at the same time.

[0050] Step S2: extract the horizontal eye movement video, i.e., the eye movement video, according to the start frame and the end frame of the horizontal eye movement, and use the gaze scoring algorithm to calculate the severity score of the patient's gaze condition. The severity score of the patient's gaze condition includes three scores, namely, 0, 1, and 2, which respectively represent normal, partial gaze paralysis, passive gaze, or complete gaze paralysis. Step S2 includes:

[0051] Step S2.1: The eye-rolling action video is used as input to select feasible frames and obtain a feasible frame set. The feasible frame selection method includes using a feasible frame selection algorithm. The feasible frame selection algorithm includes multiple constraints, and images that meet all constraints are considered to be feasible frames. First, the facial feature points and head pose matrix of the character on the image are detected; the pixel distance between the left and right eye feature points on the image is calculated to constrain the distance between the face and the camera; the head pose matrix is ​​used to constrain the deflection angle of the face in the horizontal and vertical directions.

[0052] Step S2.2: Extract the frames with open eyes from each frame image of the feasible frame set through the eye closure detection model to form an eye open frame set. The eye closure detection model determines the eye closure state of the side through the input unilateral eye area image. Therefore, for the facial images in the feasible frame set, it is necessary to pre-capture the eye area image separately through facial feature point detection as the input of the eye closure detection model. There are many methods to implement the eye closure detection model, for example, it can be implemented by a method based on a convolutional neural network. The output of the eye closure detection model is the closed state of the eye. One solution is to set a total of 4 states, including tightly closed, lightly closed, slightly open, and open. The eye closure detection model using this solution can also be used in facial paralysis scoring algorithms and other related tasks. The focus is on setting some intermediate states between tightly closed and open to avoid unclear definitions.

[0053] Step S2.3: Detect facial feature points for each frame in the eye-opening frame set, and determine whether the eyeball of the person has moved by the eye area feature points therein. If so, execute step S2.4; if not, output 2 points. The method for determining whether the eyeball of the person has moved includes calculating the ratio of the distance between the pupil center point and the left and right eye corners by the eye area feature points, and calculating the standard deviation of the ratio in all images. If the standard deviation is less than a preset threshold, the threshold is set according to multiple conditions such as the use scenario and the use device, for example, it is set to 0.05, indicating that the eyeball has not moved; otherwise, it indicates that the eyeball has moved.

[0054] Step S2.4: Each frame of the eye-opening frame set is passed through the eye posture recognition model to obtain frame-by-frame eye posture recognition results for the left eye and the right eye, and then the frame numbers of the eyeballs located at the inner and outer positions of the eye sockets are obtained respectively. The eye posture recognition model determines the eye posture of the side by inputting a unilateral eye area image. The implementation method of the eye posture recognition model includes implementation based on a convolutional neural network. The output of the eye posture recognition model is the category of eye posture, and one scheme is to set a total of 3 posture categories, including central, inner, and outer, with the focus on including the two categories of inner and outer.

[0055] Step S2.5: Determine whether the number of frames satisfies a threshold value N1. If so, output a score of 0; if not, output a score of 1.

[0056] Specifically, in combination with Figure 2 The gaze scoring algorithm of the present invention is specifically described as follows: First, input the eye-turning action video, use the feasible frame selection algorithm to extract all feasible frames from the video to form a feasible frame set. Each frame of the feasible frame set is passed through the closed eye detection model to obtain whether the person in the image has closed eyes, and then extract the frames with open eyes to form an eye-opening frame set. Facial feature points are detected for each frame, and the eye area feature points are used to determine whether the person's eyeball has moved: if the eyeball on either side does not move, 2 points are output; if both eyeballs move, the following steps are continued. Each frame of the eye-opening frame set is passed through the eyeball posture recognition model to obtain the frame-by-frame eyeball posture recognition results for the left eye and the right eye, and then the frame numbers of the eyeball located at the inner and outer positions of the eye sockets are obtained respectively. For each eye, if the inner frame number and the outer frame number meet the conditions, it indicates that the eye movement on that side is normal, otherwise it is abnormal. At this time, if both eyes are normal, 0 points are output; if the eye movement on either side is abnormal, 1 point is output.

[0057] Step S3: According to the start and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, respectively, the videos of the three actions are extracted, and the facial paralysis scoring algorithm is used to calculate the severity score of the patient's facial paralysis. The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2, and 3, which respectively represent: normal, minimal facial paralysis (asymmetric smile), partial facial paralysis (central paralysis), and complete facial paralysis (peripheral paralysis).

[0058] The step S3 includes: tooth showing detection, eyebrow raising detection and eye closing detection, and outputs a specific score according to the abnormal results of tooth showing detection, eyebrow raising detection and eye closing detection respectively through facial paralysis scoring logic. Figure 3 As shown, the facial paralysis scoring logic is specifically as follows: if the teeth are normal, 0 points are output; if the teeth are slightly abnormal, 1 point is output; if the teeth are severely abnormal, eyebrow raising is normal, and eye closing is normal, 2 points are output; if the teeth are severely abnormal, eyebrow raising is abnormal, and eye closing is abnormal, 3 points are output.

[0059] The tooth-showing detection includes inputting the tooth-showing action video into the mouth corner movement detection algorithm to obtain each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence. The left and right side strength ratio of each sequence is calculated, and the maximum value of the ratio in all sequences is obtained. For the maximum value, two thresholds A and B are set to judge the asymmetry of the left and right side actions, corresponding to the abnormal situation judgment of the tooth-showing action, which are normal tooth showing / slightly abnormal tooth showing / severely abnormal tooth showing, respectively. The mouth corner movement detection algorithm mainly includes two steps: First, all continuous feasible frame sequences in the video are obtained through a feasible frame selection algorithm. Then, for each sequence, facial feature points are extracted for each frame image therein, and the partial feature points corresponding to the detected action (such as mouth corner movement action, eyebrow raising action, etc.) are used to calculate the movement amplitude of the partial feature points on the left and right sides, respectively, as the left and right side action unit strength output.

[0060] The eyebrow raising detection includes inputting the eyebrow raising action video into the eyebrow raising detection algorithm to obtain each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence. The left and right side strength ratio of each sequence is calculated, and the maximum value of the ratio in all sequences is obtained. For the maximum value, a certain threshold C is set to judge the asymmetry degree of the left and right side actions, and the corresponding abnormal situation of the eyebrow raising action is judged as normal eyebrow raising / abnormal eyebrow raising.

[0061] The eye closure detection includes using a feasible frame selection algorithm on the eye closure action video to extract all feasible frames from the video to form a feasible frame set. Each frame image of the feasible frame set is passed through the eye closure detection model to obtain the frame-by-frame eye closure state detection results for the left eye and the right eye, and then the sum of the frames of the eyes being lightly closed and tightly closed is obtained. A certain threshold D is set, and for each eye, it is determined whether the sum of the above frame numbers is greater than the threshold D, corresponding to the abnormal conditions of the eye closure action, which are normal eye closure and abnormal eye closure, respectively.

[0062] The purpose of detecting eyebrow raising and eye closing movements included in the present invention includes evaluating whether there is peripheral facial paralysis.

[0063] Embodiment 2

[0064] The present invention also provides a stroke face auxiliary assessment system based on facial action unit detection. The stroke face auxiliary assessment system based on facial action unit detection can be implemented by executing the process steps of the stroke face auxiliary assessment method based on facial action unit detection, that is, those skilled in the art can understand the stroke face auxiliary assessment method based on facial action unit detection as a preferred implementation of the stroke face auxiliary assessment system based on facial action unit detection.

[0065] According to the present invention, a facial stroke auxiliary assessment system based on facial action unit detection includes:

[0066] Module M1: Collect the patient's facial movement video and record the start and end frames of each movement in the video; the movements include horizontal eye rotation, tooth showing, eyebrow raising, and eye closing. The horizontal eye rotation refers to the patient's horizontal eye rotation to the extreme positions on the left and right sides of the eye sockets. The eyeball moves from the leftmost end to the rightmost end and then back to the leftmost end as 1 time, and repeats 3 to 4 times. The tooth showing movement refers to the patient grinning hard. The eyebrow raising movement refers to the patient's eyebrows on both sides being raised hard at the same time. The eye closing movement refers to the patient's eyes being closed hard at the same time.

[0067] Module M2: Extract the video of the horizontal eye rotation action, i.e., the eye rotation action video, according to the start frame and the end frame of the horizontal eye rotation action, and use the gaze scoring algorithm to calculate the severity score of the patient's gaze condition. The severity score of the patient's gaze condition includes three scores, namely 0, 1, and 2, which respectively represent normal, partial gaze paralysis, passive gaze, or complete gaze paralysis. The module M2 includes: Module M2.1: Use the eye rotation action video as input to select feasible frames to obtain a feasible frame set. Module M2.2: Extract the eye-opening frame from each frame image of the feasible frame set through the eye closure detection model to form an eye-opening frame set. Module M2.3: Detect facial feature points for each frame image in the eye-opening frame set, and judge whether the eyeball of the person moves through the eye area feature points therein. If so, execute module M2.4; if not, output 2 points. The system package for judging whether the eyeball of a person moves calculates the ratio of the distance between the center point of the pupil and the left and right corners of the eye through the feature points of the eye area, and calculates the standard deviation of the ratio in all images. If the standard deviation is less than the threshold, it indicates that the eyeball has not moved; otherwise, it indicates that the eyeball has moved. Module M2.4: Pass each frame of the eye-opening frame set through the eyeball posture recognition model to obtain the frame-by-frame eyeball posture recognition results for the left eye and the right eye, and then obtain the frame number of the eyeball located at the inner and outer positions of the eye socket respectively; Module M2.5: Determine whether the frame number is greater than the threshold value N1. If so, output 0 points; if not, output 1 point.

[0068] Module M3: According to the start and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, respectively, the videos of the three actions are extracted, and the facial paralysis scoring algorithm is used to calculate the severity score of the patient's facial paralysis. The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2, and 3, which respectively represent: normal, minimal facial paralysis, partial facial paralysis, and complete facial paralysis. The module M3 includes: tooth-showing detection, eyebrow-raising detection, and eye-closing detection. According to the abnormal results of the tooth-showing detection, eyebrow-raising detection, and eye-closing detection, the specific score is output through the facial paralysis scoring logic. The tooth-showing detection includes inputting the tooth-showing action video into the mouth corner movement detection algorithm to obtain each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence; calculating the left and right side strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting two thresholds A and B to judge the asymmetry of the left and right side actions, corresponding to the abnormal judgment of the tooth-showing action, which are normal tooth-showing / slightly abnormal tooth-showing / severely abnormal tooth-showing. The eyebrow raising detection includes inputting the eyebrow raising action video into the eyebrow raising detection algorithm to obtain each continuous feasible frame sequence in the video, and the corresponding left and right action unit strengths of each sequence; calculating the left and right side strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting a certain threshold C to judge the asymmetry of the left and right side actions, corresponding to the abnormal situation of the eyebrow raising action, which is normal eyebrow raising / abnormal eyebrow raising. The eye closing detection includes using a feasible frame selection algorithm for the eye closing action video, extracting all feasible frames from the video to form a feasible frame set, passing each frame image of the feasible frame set through the eye closing detection model, obtaining the frame-by-frame eye closing state detection results for the left eye and the right eye, and then obtaining the sum of the frames of the eyes lightly closed and tightly closed, setting a threshold D, and judging whether the sum of the frames is greater than the threshold D for each eye, corresponding to the abnormal situation of the eye closing action, which is normal eye closing and abnormal eye closing.

[0069] The facial paralysis scoring logic is specifically as follows: if the teeth are normal, 0 points are output; if the teeth are slightly abnormal, 1 point is output; if the teeth are severely abnormal, eyebrow raising is normal, and eye closing is normal, 2 points are output; if the teeth are severely abnormal, eyebrow raising is abnormal, and eye closing is abnormal, 3 points are output.

[0070] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0071] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for assisting facial assessment of stroke based on facial action unit detection, characterized in that: include: Step S1: collecting a video of the patient's facial movements, and recording the start frame and the end frame of each movement in the video; the movements include horizontal eyeball rotation, tooth showing, eyebrow raising, and eye closing; Step S2: extracting a horizontal eye movement video, i.e., an eye movement video, according to the start frame and the end frame of the horizontal eye movement, and using a gaze scoring algorithm to calculate a severity score of the patient's gaze condition; Step S3: according to the start frames and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, respectively, extract the videos of the three actions, and use the facial paralysis scoring algorithm to calculate the severity score of the patient's facial paralysis.

2. The method for assisting stroke facial assessment based on facial action unit detection according to claim 1, characterized in that: The horizontal eye movement refers to the patient's horizontal eye movement to the extreme positions of the left and right sides of the eye sockets. The eye moves from the leftmost end to the rightmost end and then back to the leftmost end, which is counted as one time, and is repeated 3 to 4 times. The tooth-showing action refers to the patient grinning hard; The eyebrow-raising action refers to the patient's eyebrows being raised upwards on both sides at the same time; The eye-closing action refers to the patient closing both eyes forcefully at the same time.

3. The method for assisting stroke facial assessment based on facial action unit detection according to claim 1, characterized in that: The severity score of the patient's gaze condition includes three scores, 0, 1, and 2, which respectively represent normal, partial gaze paralysis, passive gaze, or complete gaze paralysis; The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2 and 3, which respectively represent: normal, minimal facial paralysis, partial facial paralysis and complete facial paralysis.

4. The method for assisting stroke facial assessment based on facial action unit detection according to claim 1, characterized in that: The step S2 comprises: Step S2.1: taking the eye-rolling action video as input to select feasible frames and obtain a feasible frame set; Step S2.2: extracting eye-open frames from each frame image of the feasible frame set through an eye-closing detection model to form an eye-open frame set; Step S2.3: Detect facial feature points for each frame in the eye-opening frame set, and determine whether the eyeball of the person has moved by the eye area feature points therein. If so, execute step S2.4; if not, output 2 points. The method for determining whether the eyeball of the person has moved includes calculating the ratio of the distance between the pupil center point and the left and right eye corners by the eye area feature points, and calculating the standard deviation of the ratio in all images. If the standard deviation is less than the threshold, it indicates that the eyeball has not moved; otherwise, it indicates that the eyeball has moved; Step S2.4: each frame of the eye-opening frame set is passed through the eye posture recognition model to obtain frame-by-frame eye posture recognition results for the left eye and the right eye, and then the number of frames in which the eyeball is located at the inner and outer positions of the eye socket respectively; Step S2.5: Determine whether the number of frames is greater than the threshold N1. If so, output 0 points; if not, output 1 point.

5. The method for assisting stroke facial assessment based on facial action unit detection according to claim 1, characterized in that: The step S3 includes: tooth showing detection, eyebrow raising detection and eye closing detection. According to the abnormal situation results of tooth showing detection, eyebrow raising detection and eye closing detection respectively, a specific score is output through the facial paralysis scoring logic.

6. The method for assisting stroke facial assessment based on facial action unit detection according to claim 5, characterized in that: The tooth-showing detection comprises: inputting a tooth-showing action video into a mouth corner movement detection algorithm to obtain each continuous feasible frame sequence in the video, and the strength of the left and right action units corresponding to each sequence; calculating the left and right strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting two thresholds A and B to judge the asymmetry degree of the left and right actions, and corresponding to the abnormal situation judgment of the tooth-showing action, which are normal tooth-showing / slightly abnormal tooth-showing / severely abnormal tooth-showing; The eyebrow raising detection comprises: inputting the eyebrow raising action video into the eyebrow raising detection algorithm, obtaining each continuous feasible frame sequence in the video, and the left and right action unit strengths corresponding to each sequence; calculating the left and right side strength ratio of each sequence, and obtaining the maximum value of the ratio in all sequences; for the maximum value, setting a certain threshold C to judge the asymmetry degree of the left and right side actions, and judging the abnormal situation of the corresponding eyebrow raising action, namely, normal eyebrow raising / abnormal eyebrow raising; The eye closure detection includes using a feasible frame selection algorithm on the eye closure action video, extracting all feasible frames from the video to form a feasible frame set, passing each frame image of the feasible frame set through an eye closure detection model, obtaining frame-by-frame eye closure state detection results for the left eye and the right eye, and then obtaining the sum of the number of frames for lightly closed and tightly closed eyes, setting a threshold D, and judging for each eye whether the sum of the number of frames is greater than the threshold D, corresponding to the abnormal conditions of the eye closure action, which are normal eye closure and abnormal eye closure, respectively.

7. The method for assisting stroke facial assessment based on facial action unit detection according to claim 6, characterized in that: The mouth corner movement detection algorithm includes: obtaining all continuous feasible frame sequences in the video through a feasible frame selection algorithm, and then, for each sequence, extracting facial feature points for each frame image therein, using some feature points corresponding to the detected action, calculating the movement amplitude of the some feature points on the left and right sides respectively, as the left and right side action unit intensity output.

8. The method for assisting stroke facial assessment based on facial action unit detection according to claim 5, characterized in that: The facial paralysis scoring logic is specifically as follows: if the teeth are normal, 0 points are output; if the teeth are slightly abnormal, 1 point is output; if the teeth are severely abnormal, eyebrow raising is normal, and eye closing is normal, 2 points are output; if the teeth are severely abnormal, eyebrow raising is abnormal, and eye closing is abnormal, 3 points are output.

9. A stroke facial auxiliary assessment system based on facial action unit detection, characterized in that: include: Module M1: Collect the patient's facial movement video and record the start frame and end frame of each action in the video; the actions include horizontal eye movement, tooth showing, eyebrow raising, and eye closing; Module M2: extracting the horizontal eye movement video, i.e., the eye movement video, according to the start frame and the end frame of the horizontal eye movement, and using the gaze scoring algorithm to calculate the severity score of the patient's gaze condition; Module M3: According to the start and end frames of the tooth-showing action, eyebrow-raising action, and eye-closing action, the videos of the three actions are extracted respectively, and the facial paralysis scoring algorithm is used to calculate the severity score of the patient's facial paralysis.

10. The stroke facial auxiliary assessment system based on facial action unit detection according to claim 9, characterized in that: The horizontal eye movement refers to the patient's horizontal eye movement to the extreme positions of the left and right sides of the eye sockets. The eye moves from the leftmost end to the rightmost end and then back to the leftmost end, which is counted as one time, and is repeated 3 to 4 times. The tooth-showing action refers to the patient grinning hard; The eyebrow-raising action refers to the patient's eyebrows being raised upwards on both sides at the same time; The eye-closing action refers to the patient closing both eyes forcefully at the same time; The severity score of the patient's gaze condition includes three scores, 0, 1, and 2, representing normal / partial gaze paralysis, passive gaze, and complete gaze paralysis, respectively; The severity score of the patient's facial paralysis includes four scores, namely 0, 1, 2 and 3, which respectively represent: normal, minimal facial paralysis, partial facial paralysis and complete facial paralysis.

Images