A facial paralysis rehabilitation training method and system
Through deep learning to evaluate facial symmetry and generate animation images to guide healthy expressions, it solves the problem that facial paralysis patients find it difficult to accurately evaluate facial symmetry during the recovery process, and achieves efficient and accurate facial muscle training, reducing the workload of doctors.
Patent Information
- Application Number
- CN202210165199.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-17
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2042-02-17
AI Technical Summary
Patients with facial paralysis have difficulty accurately assessing facial symmetry during the recovery process, resulting in poor training results. The traditional method relies on the subjective judgment of the doctor and takes up a lot of outpatient time.
By collecting patient facial images, detecting key points, establishing a face coordinate system, evaluating facial symmetry using deep learning methods, and generating animation images to guide healthy expressions, and feedback the differences in facial muscle function in real time to help patients adjust the direction and strength of force.
It realizes automated facial symmetry evaluation and personalized training plan push, which reduces the workload of doctors, improves the accuracy and efficiency of rehabilitation training, and enhances the patient's compliance and training effect.
Smart Images
Figure CN114550873B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of facial paralysis assessment and rehabilitation, and in particular to a facial paralysis rehabilitation training method and system. Background Art
[0002] Patients with facial muscle and nerve damage will have distorted expressions, which seriously affects their daily lives. For patients, regular visits to the hospital for rehabilitation checks will cost a lot of time and money. In addition, when facing doctors, patients may not be able to show their best condition due to psychological factors such as nervousness and fear, making the assessment of rehabilitation status inaccurate. For doctors, assessing the rehabilitation status of patients and guiding rehabilitation training are repetitive basic work, which takes up a lot of outpatient time and reduces the number of patients received.
[0003] How to provide a training method and training system that can automatically evaluate the facial symmetry of patients, provide patients with training content suitable for their current condition in a timely manner based on the evaluation results, and provide patients with the correct expected effect of each facial expression based on images or sounds, guide patients to make symmetrical expressions, and timely and finely evaluate the actual training effect of patients, is a problem that needs to be solved urgently. Summary of the invention
[0004] The purpose of the present invention is to provide a facial paralysis rehabilitation training method and system, which can evaluate the facial symmetry of patients in real time and provide patients with suitable training plans; the present invention also uses the actual facial muscle function of the affected side as a reference to generate a visualized animated image to guide the expression of the healthy side, and helps patients generate a symmetrical smile through the "affected side guiding the healthy side" method; at the same time, the present invention can also timely feedback the difference in facial muscle function between the affected side and the healthy side through animated images, sounds, etc., to help patients adjust the direction and strength of facial muscle force in a timely manner.
[0005] In order to achieve the above object, the present invention provides a facial paralysis rehabilitation training method, comprising the steps of:
[0006] Collecting a facial image of a patient and detecting key points in the facial image;
[0007] Establishing a facial coordinate system, calculating the coordinate values of the key points in the facial coordinate system, and obtaining a symmetry assessment result of the patient's face through a deep learning method based on the coordinate values;
[0008] Pushing a training task including at least one training content to the patient based on the patient's recovery time and / or the symmetry assessment result;
[0009] Detect key points of the patient during training, project the key points of the affected side in the face coordinate system to the healthy side with the y-axis of the face coordinate system as the symmetry axis, and obtain corresponding projection points; based on the coordinate values of the key points in the face coordinate system and / or the healthy side projection points, generate an animated image for guiding the training of the affected side and / or the healthy side facial muscles;
[0010] Feedback the animated image to the patient in real time via a graphical interface client;
[0011] Provide patients with feedback on the training effects of quantitative assessments.
[0012] Optionally, the animated image includes a first animated image; the first animated image includes one or more virtual eyebrows, eyes, nose, and lips images generated based on the key points of the affected side and the projection points of the healthy side in the facial coordinate system; the first animated image changes with the facial expression of the affected side, and the first animated image is used to guide one or more of the starting time point, direction, speed, expected arrival position, eye opening degree, and eyebrow height of the corner of the mouth movement on the healthy side.
[0013] Optionally, the animated image also includes a second animated image; the second animated image includes one or more virtual eyebrows, eyes, nose, and mouth images generated based on key points of the affected side and the healthy side in the facial coordinate system; the second animated image changes with the facial expressions of the affected side and the healthy side.
[0014] Optionally, the animation image further includes a graphic, which changes its movement direction based on the displacement of key points of the corners of the mouth on both sides; the movement of the graphic is used to feedback the movement amplitude, movement speed and symmetry of the corners of the mouth on both sides.
[0015] Optionally, the training content is relaxation training; the training method further comprises:
[0016] Taking the facial image of the patient in a relaxed state as the first facial image; calculating the coordinate values of key points in the first facial image in a face coordinate system;
[0017] Acquire a facial image of the patient as a second facial image in real time; calculate the coordinate values of key points in the second facial image in a face coordinate system;
[0018] Comparing the displacement values of the corresponding key points in the first and second facial images in the face coordinate system; performing weighted calculation on the displacement values of various facial regions of the patient to obtain a muscle tension score value of the corresponding region;
[0019] The second facial image in the face coordinate system is fed back through the graphical interface client. If the muscle tension score value of a certain facial area is higher than the set muscle tension threshold, the area is shown on the second facial image.
[0020] Optionally, the training content includes strength training for the affected side of the mouth, and the training method further includes:
[0021] Calculate the displacement value of the key point of the corner of the mouth on the affected side; and feed back the image of the graphic or the real-time facial image of the patient to the patient through a graphical interface client;
[0022] Based on the displacement value, the patient is fed back a score of the degree of recovery of the affected side of the mouth angle;
[0023] The degree of eye opening on the affected side is calculated based on the key points of the upper and lower eyelid margins on the affected side. When the degree of eye opening on the affected side is less than the set eye opening threshold, a prompt or alarm is issued.
[0024] Optionally, the training content includes bilateral corner-of-mouth movement coordination training, and the training method further includes:
[0025] Irregularly giving the patient instructions to smile, laugh moderately, or laugh loudly;
[0026] Feedback a facial image of the patient in a face coordinate system through a graphical interface client, and superimpose a first animation image on the facial image; or feed back a second animation image through a graphical interface client, and superimpose the first animation image on the second animation image; or feed back an image of the graphic through a graphical interface client;
[0027] Based on the starting time of the movement of the key points of the corners of the mouth on both sides, the time taken to complete the command action, and the displacement value in the face coordinate system, the starting synchronization score, the command action completion speed score, and the movement amplitude score of the corners of the mouth on both sides are fed back to the patient;
[0028] Based on the displacement amplitude of the key points of the affected lip and the corresponding key points of the healthy lip, the directional scores of the movement of the corners of the mouth on both sides are fed back to the patient.
[0029] Optionally, the training content includes eyelid closure training on the affected side, and the training method further includes:
[0030] Based on the time taken for the affected side to complete the eyelid closure action, the affected side eyelid closure speed score was fed back to the patient;
[0031] Based on the coordinates of the key points of the upper and lower eyelid margins of the affected side in the face coordinate system, the degree of eye opening on the affected side is calculated; and based on the degree of eye opening on the affected side, the degree of eyelid closure on the affected side is fed back to the patient.
[0032] Optionally, the training content includes: bilateral eyelid closure training, and the training method includes:
[0033] Based on the time taken by the affected side and the healthy side to complete the eyelid closing action, the patient is fed back the eyelid closing time difference score of the two eyes, the eyelid closing speed score of the affected side, and the eyelid closing speed score of the healthy side;
[0034] Based on the coordinates of the key points of the upper and lower eyelid margins of the affected side in the face coordinate system, the degree of eye opening on the affected side is calculated; based on the degree of eye opening on the affected side, the degree of eyelid closure on the affected side is fed back to the patient.
[0035] The present invention also provides a facial paralysis rehabilitation training system, which is used to implement the facial paralysis rehabilitation training method according to the present invention, and the training system comprises:
[0036] An image acquisition module, used for acquiring facial images of patients with facial paralysis;
[0037] An image processing module, signal-connected to the image acquisition module, for detecting key points in the facial image and transforming the facial image into a face coordinate system;
[0038] The facial paralysis assessment module is connected to the image processing module by signal, and obtains the patient's facial symmetry assessment result through a deep learning method based on the key point coordinate values in the face coordinate system;
[0039] A training guidance module, signal connection image processing module and the facial paralysis assessment module; pushes training tasks to the patient based on the patient recovery time and / or the symmetry assessment result; the training guidance module also generates an animated image for guiding the training of the affected side and / or the healthy side facial muscles based on the key point coordinate values and / or the healthy side projection point coordinate values in the face coordinate system; by taking the y-axis of the face coordinate system as the symmetry axis, the key points of the affected side in the face coordinate system are projected to the healthy side to obtain the healthy side projection points;
[0040] The display module is signal-connected to the image acquisition module and the training guidance module, and is used to visually display the patient's facial image and / or the animation image.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The facial paralysis rehabilitation training method and system of the present invention can automatically detect the key points of the patient's face, obtain a reliable facial symmetry evaluation result through a deep learning method, and automatically push a training program suitable for the current condition to the patient based on the evaluation result. The present invention also uses the actual facial muscle function of the affected side as a reference to generate a visual animated image to guide the expression of the healthy side, and helps the patient to generate a symmetrical smile through the "affected side guiding the healthy side" method, helping the patient to perform accurate and correct training, and preventing the wrong training method from aggravating facial asymmetry. The present invention not only greatly reduces the workload of doctors, but also enables patients to conveniently obtain high-quality, efficient, and stable medical resources, save medical costs, and improve the compliance and effectiveness of rehabilitation training.
[0043] In the traditional facial muscle training method, without the assistance of a trainer, the patient can only get feedback information about the facial muscle training through a mirror. It is impossible to accurately know whether the training is standardized and whether the force amplitude is reasonable. The patient can only use subjective feelings as a judgment standard. Especially during the closed eyelid training process, it is even more impossible to self-test the training effect. The present invention can timely and finely evaluate the actual effect of each facial muscle training of the patient, and timely feedback the difference in muscle function between the affected side and the healthy side through visual graphics, different prompt sounds, etc., to help patients adjust the direction and strength of facial muscle force in time.
[0044] The present invention helps patients to repeatedly train and form new smiling habits, so that they can smile symmetrically without using a mirror for observation and adjustment. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the description. Obviously, the drawings described below are an embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work:
[0046] Figure 1 It is a flow chart of the facial paralysis rehabilitation training method of the present invention;
[0047] Figure 2 Schematic diagram of detected facial key points and cornea recognition circle in an embodiment of the present invention;
[0048] Figure 3 A schematic diagram of converting key points in an original facial image into a facial coordinate system in an embodiment of the present invention;
[0049] Figure 4 The Sunny-Brook facial nerve assessment table used in the symmetry assessment in the embodiment of the present invention;
[0050] Figure 5 It is a schematic diagram of the height of the eyebrow head, eyebrow peak and eyebrow tail in the facial image;
[0051] Fig. 6A This is a schematic diagram of a patient's facial image in a face coordinate system;
[0052] Figure 6B For Fig. 6A A first animation image schematic diagram is superimposed thereon;
[0053] Figure 6C For Fig. 6A A second animation image schematic diagram is superimposed thereon;
[0054] Fig.6D For Fig. 6A The schematic diagrams of the first and second animation images are superimposed thereon;
[0055] Fig. 7A In the embodiment of the present invention, a lever is used as a graphic schematic diagram in the animation image;
[0056] Figure 7B In the embodiment of the present invention, a ball is used as a graphic schematic diagram in the animation image;
[0057] Figure 8 This is a schematic diagram of the degree of eye opening on the affected side;
[0058] Fig. 9 It is a structural schematic diagram of the facial paralysis rehabilitation training system of the present invention;
[0059] In the figure: 1. Key points; 2. Corneal recognition circle;
[0060] 11. Image acquisition module; 12. Image processing module; 13. Facial paralysis assessment module; 14. Training guidance module; 15. Display module. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0062] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0063] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0064] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0065] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0066] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0067] After facial paralysis, most patients will have more or less sequelae, such as facial muscle linkage, partial muscle weakness, etc. The sequelae of facial paralysis are also called facial muscle dysfunction. In the case of facial muscle dysfunction, in order to obtain a roughly normal smile, the original way of smiling must be changed.
[0068] Practice has shown that through a large amount of correct facial muscle training, patients can form new muscle memory and effectively improve their facial expressions. However, the location and degree of facial muscle dysfunction in each patient are different. As the condition improves, the specific facial muscle training program also needs to be adaptively adjusted.
[0069] Due to poor facial proprioception, it is difficult to obtain effective position information during facial expression activities. When you first start facial muscle training, obtaining visual feedback is an indispensable condition (for example, obtaining visual feedback by looking in the mirror). Whether the smile in the mirror is the optimal smile within the current facial muscle strength range can only be judged by subjective feelings.
[0070] For patients, it takes a lot of time and money to go to the hospital regularly for rehabilitation check-ups and receive facial muscle training guidance. In addition, patients may not be able to perform at their best when facing doctors due to psychological factors such as nervousness and fear, which affects the doctor's accurate assessment of the level of facial paralysis and the correct guidance of rehabilitation training. Although some hospitals have dedicated rehabilitation trainers, one rehabilitation trainer cannot guide multiple patients in facial muscle rehabilitation training at the same time, and cannot meet the rehabilitation training needs of a large number of patients. Due to the lack of correct guidance, some patients use the wrong facial muscle training method, which aggravates the asymmetry of facial expressions.
[0071] The present invention provides a facial paralysis rehabilitation training method, such as Figure 1 As shown, it includes the steps:
[0072] S1, collecting a facial image of a patient and detecting key points 1 in the facial image;
[0073] like Figure 2As shown, the key points 1 in the facial image are distributed at the contours of the lips, eyebrows, nose, and eyes. In this embodiment, the detection of the key points 1 can be obtained by training various parts of the face through the open source machine learning platform dlib (this is a prior art and will not be described in detail here).
[0074] S2. Establishing a facial coordinate system, calculating the coordinate value of the key point 1 in the facial coordinate system, and obtaining a symmetry assessment result of the patient's face through a deep learning method based on the coordinate value;
[0075] In this embodiment, a facial coordinate system is established based on the corneal recognition circle 2 in the facial image. The Faster-RCNN algorithm (based on the target recognition algorithm) in the symbolic mathematics system (TensorFlow) can be used to train the left and right corneal recognition circles respectively to obtain the corneal recognition circle 2 in the facial image (this is a prior art and will not be described in detail here). It should be noted that the facial coordinate system can also be established by other facial parts (such as the positions of the two ears), which is only an example of the present invention and should not be used as a limitation of the present invention.
[0076] The face coordinate system is established by taking the line connecting the centers of the left and right corneal recognition circles as the x-axis, the midpoint of the line as the origin (0,0), and the perpendicular midline of the line as the y-axis. Figure 3 As shown, the original patient facial image can be transformed by translation and rotation of the coordinate system to obtain the facial image in the face coordinate system. The diameter of the corneal recognition circle 2 is used as the reference length and defined as 10 mm, and the pixel distance of the facial image is converted into the physical distance to obtain the coordinate value of each key point 1 in the face coordinate system.
[0077] In an embodiment of the present invention, the evaluation of facial symmetry includes the evaluation of forehead lifting symmetry and the evaluation of mouth grinning symmetry. The facial symmetry score can be obtained by weighted summing the forehead lifting symmetry score and the mouth grinning symmetry score.
[0078] In the forehead lifting symmetry assessment, multiple facial images of patients with facial paralysis lifting their foreheads were first obtained, and then the images were manually compared based on the following: Figure 4 The Sunny-Brook facial nerve assessment form shown here scores the forehead lift movement for each facial image. The score ranges from 1 to 5. The brow feature is recorded as F h , the brow peak feature is F p , the eyebrow tail feature is F t .like Figure 5 As shown, the height of the left eyebrow is d LH , the height of the left eyebrow peak is d Lp , the height of the left eyebrow tail is d LT ; Note the height of the right eyebrow as d RH , the height of the right eyebrow peak is d Rp , the height of the right eyebrow tail is d RT; Record the “affected side” item in the table as S. Figure 5 Only part of the patient's facial image is shown. When measuring the brow height, the diameter of the corneal recognition circle 2 is also defined as 10 mm, and the pixel distance of the brow height is converted into a physical distance. Three brow-raising features are obtained;
[0079]
[0080]
[0081]
[0082] In this embodiment, a BP neural network is used as a forehead lifting assessment model. The input layer of the forehead lifting assessment model includes three input nodes (corresponding to the features of the brow, brow peak, and eyebrow tail, respectively), and the output layer includes five output nodes. Each output node corresponds to a score. For example, when the output result is "00100", the score is 3 points. The features of the brow, brow peak, and eyebrow tail of each facial image are used as one input of the forehead lifting model to train the model. By inputting the facial images collected in real time into the trained forehead lifting assessment model, the symmetry evaluation value of the forehead lifting action can be obtained.
[0083] Then assess the symmetry of mouth movement:
[0084] Through manual methods, based on Figure 4 The Sunny-Brook facial nerve assessment table shown is the grin corner movement score for each facial image used for training. The score ranges from 1 to 5. The horizontal coordinate feature of the mouth corner is Fx, the vertical coordinate feature is Fy, and the mirror distance feature is Fm; the horizontal coordinate of the left mouth corner is x L , the ordinate of the left mouth corner is y L ; Let the horizontal coordinate of the right mouth corner be x R , the vertical coordinate of the right mouth corner is y R ;Record the mirror image distance of the mouth angle as dm (mm);Record the item "affected side" in the table as S;
[0085]
[0086]
[0087] F t =d m
[0088] Taking the y-axis of the face coordinate system as the axis of symmetry, the left corner of the mouth is projected onto the right face image, and the distance between the projection point of the left corner of the mouth and the right corner of the mouth is the corner of mouth mirror distance.
[0089] In this embodiment, BP neural network is used as the grin angle evaluation model. The input layer of the grin angle evaluation model includes three input nodes (corresponding to the horizontal coordinate feature of the mouth corner, the vertical coordinate feature of the mouth corner, and the mirror distance feature, respectively), and the output layer includes five output nodes (each output node corresponds to a score). The horizontal coordinate feature of the mouth corner, the vertical coordinate feature of the mouth corner, and the mirror distance feature of each facial image are used as one input of the grin angle evaluation model to train the model. By inputting the facial image collected in real time into the trained grin angle evaluation model, the symmetry evaluation value of the grin angle action can be obtained.
[0090] The method for evaluating the facial symmetry of patients is not the focus of the present invention. It should be noted that any other deep learning method that can evaluate the facial symmetry of patients is applicable to the present invention. The above content is only used as an example of the present invention and should not be used as a limitation of the present invention. Since the recovery of facial paralysis rehabilitation training of patients is relatively slow, they can also have regular follow-up visits and obtain the results of the facial symmetry evaluation of patients based on the doctor's manual evaluation.
[0091] S3. Based on the patient's recovery time and / or the symmetry assessment result, push a training task containing at least one training content to the patient.
[0092] S4, detecting the key points 1 of the patient during training, taking the y-axis of the face coordinate system as the axis of symmetry, projecting the key points of the affected side in the face coordinate system to the healthy side, and obtaining corresponding projection points; based on the coordinate values of the key points in the face coordinate system and / or the healthy side projection points, generating an animated image for guiding the training of the affected side and / or the healthy side facial muscles;
[0093] In one embodiment, the animated image includes a first animated image; the first animated image includes one or more of two virtual eyebrow, eye, nose, lip, and cornea recognition circle images generated based on the key points of the affected side and the projection points of the healthy side in a facial coordinate system; the first animated image changes with the facial expression of the affected side, and the first animated image guides one or more of the starting time point, direction, speed, expected arrival position, eye opening, and eyebrow height of the corner of the mouth movement on the healthy side.
[0094] Fig. 6A , a patient's facial image in a facial coordinate system is shown. Figure 6B Shown superimposed on Fig. 6A The first animated image on Figure 6B The black dots in Figure 6B The position of the middle black spot changes with the expression of the affected side). Figure 6B The first animation image in includes virtual lips, eyes, and cornea recognition circle images, which is only an example.
[0095] In another embodiment, the animated image further includes a second animated image; the second animated image includes one or more of virtual eyebrows, eyes, nose, mouth, and cornea recognition circle 2 images generated based on the key points of the affected side and the healthy side in the face coordinate system; the second animated image changes with the facial expressions of the affected side and the healthy side. Figure 6C Shown superimposed on Fig. 6A The second animated image on Figure 6C The dotted lines in Figure 6C The shape of the middle dotted line changes with the expression of the affected side). Figure 6C The second animation image in the includes virtual lips, eyes, and cornea recognition circle images, which is only used as an example. In another embodiment, the animation image further includes a graphic, and the graphic changes the movement direction based on the displacement of the key points of the corners of the mouth on both sides; the movement amplitude, movement speed and symmetry of the corners of the mouth on both sides are fed back through the movement of the graphic.
[0096] like Fig. 7A As shown, the figure can be a seesaw, and the two ends of the seesaw correspond to the left and right corners of the mouth respectively. When the left / right end of the seesaw moves upward, it means that the movement amplitude of the left / right corner of the mouth increases. When the seesaw is balanced, it means that the movement amplitude of the corners of the mouth on both sides is the same.
[0097] Figure 7B The figure can also be a rolling ball. In the initial state, the ball is at the zero point. When the ball rolls from the zero point to the right, it means that the left side of the mouth has a larger movement. The distance between the ball and the zero point indicates the difference in strength between the two sides of the mouth.
[0098] S5. The animation images and the training effects of the quantitative assessment are fed back to the patient in real time through the graphical interface client.
[0099] The training contents of the present invention include relaxation training, strength training of the affected side of the mouth corner, coordination training of bilateral mouth corner movements, eyelid closure training of the affected side, bilateral eyelid closure training, smile training under emotional stimulation, etc. The present invention can also be applied to other facial muscle training contents, and the enumeration of the training contents is only for example and not for limitation of the present invention. A training task may include multiple training contents, and the assessment results of facial symmetry may be different, and the included training contents may also be different. The various training contents can be combined arbitrarily, and the following combination method is only for example of the present invention and is not for limitation.
[0100] Embodiment 1
[0101] In this embodiment, the patient has a short recovery time, the affected side has no mobility, and the facial symmetry score is 0. Patients at this stage are prone to excessive muscle tension on the healthy side, so the training is mainly to relax the healthy side.
[0102] First, a facial image of the patient in a relaxed state is used as a first facial image; and the coordinate values of key points in the first facial image in a face coordinate system are calculated. The facial image of the patient in a relaxed state can be obtained by playing a radio program to relax the patient's expression, or the patient can provide a facial image of the patient in a relaxed state.
[0103] The patient is prompted to start relaxation training, and a facial image of the patient is collected in real time as a second facial image; and the coordinate values of key points in the second facial image are calculated in a face coordinate system.
[0104] Compare the displacement values of the corresponding key points in the first and second facial images in the face coordinate system; after weighted calculation of the displacement values of each area of the patient's face, obtain the muscle tension score value of the corresponding area. It should be further explained that the centers of the two corneal recognition circles in the first facial image correspond to the centers of the two corneal recognition circles in the second facial image.
[0105] The second facial image in the face coordinate system is fed back through the graphical interface client. If the muscle tension score value of a certain facial area is higher than the set muscle tension threshold, the area is shown on the second facial image.
[0106] In this embodiment, whether the muscle tension of the healthy side corner of mouth is too high can also be judged by a key point 1 of the healthy side corner of mouth. The coordinate value of the key point of the healthy side corner of mouth in the first facial image is recorded as (x1, y1), the coordinate value of the key point of the healthy side corner of mouth in the second facial image is recorded as (x2, y2), and the displacement value of the key point of the corner of mouth in the two images is recorded as L, L was used as the scoring value of the muscle tension at the angle of mouth of the healthy side.
[0107] In another embodiment, the displacement value L of the i-th key point of the healthy side eye contour can be calculated as i , and the periocular muscle tension score L′ is obtained by weighted calculation. ω i is the weight of the i-th key point 1 of the healthy eye, and A is the total number of key points on the healthy eye contour.
[0108] Embodiment 2
[0109] In this embodiment, the patient's affected side has only slight progress (for example, the range of motion of the corner of the mouth on the affected side is less than 3mm), the facial symmetry score is 1 to 2 points (this is only an example and not a limitation of the present invention), and the training content includes strength training of the corner of the mouth on the affected side. It is also possible to push strength training of the corner of the mouth on the affected side to the patient based on the patient's self-recovery time. Or, based on the surgery performed by the patient and the recovery time of the surgery, the patient (6 months after the masseter nerve repair surgery) is pushed strength training of the corner of the mouth on the affected side. The specific training steps include:
[0110] The patient is prompted to start strength training of the affected side corner of mouth, the displacement value of the key points of the affected side corner of mouth is calculated in real time, and the image of the graphic or the real-time facial image of the patient is fed back to the patient through the graphical interface client. In this embodiment, in order to better observe the movement amplitude of the affected side corner of mouth, when the image of the graphic is fed back, the position and shape of the graphic will not change with the movement of the healthy side corner of mouth.
[0111] The patient is fed back the degree of recovery of the affected corner of the mouth based on the displacement value of the key points of the affected corner of the mouth. For example, when the displacement value is 2mm, the voice or text feedback of the graphical interface client is "good"; when the displacement value exceeds 3mm, the voice or text feedback of the graphical interface client is "obvious smile achieved".
[0112] On the other hand, in order to prevent the patient from pulling the eye muscles on the affected side when exerting force on the affected side of the mouth, causing the facial muscles to be linked, this embodiment also calculates the degree of eye opening on the affected side based on the key points of the upper and lower eyelid margins on the affected side. When the degree of eye opening on the affected side is less than the set eye opening threshold, a prompt or alarm is issued. Figure 8 The distance between points a and b. Figure 8 In the figure, the y′ axis is parallel to the y axis of the face coordinate system and passes through the center of the cornea recognition circle on the affected side. Points a and b are the highest upper eyelid key point and the lowest lower eyelid key point on the y′ axis.
[0113] Embodiment 3
[0114] In this embodiment, the patient's affected side has made great progress, and the score of facial symmetry is not less than 3 points (this is only an example and not a limitation of the present invention), and the training content includes: relaxation training and bilateral mouth corner movement coordination training (the order and number of each training are not limited). This training content is also called the first stage smile training.
[0115] The specific steps of the bilateral mouth angle movement coordination training include:
[0116] Irregularly issue instructions to the patient to smile, laugh moderately, or laugh loudly. Instructions can be given through voice or different prompt sounds.
[0117] In this embodiment, if Figure 6B As shown, the real-time facial image of the patient in the face coordinate system can be fed back through the graphical interface client, and the first animation image can be superimposed on the facial image to guide the patient's training. It should be noted that the virtual eyes, nose, mouth, corneal recognition circle, etc. on the affected side in the first animation image correspond to the positions of the eyes, nose, mouth, and corneal recognition circle on the affected side in the real-time facial image. When the healthy side of the patient's mouth corner reaches the healthy side of the mouth corner position in the first animation image, it is a symmetrical smile.
[0118] In another embodiment, Fig.6DAs shown, the real-time facial image of the patient superimposed with the first animated image and the second animated image can be fed back through the graphical interface client to guide the patient training. It is easy to understand that the second animated image is an abstraction of the actual smile. For easy observation, as shown in FIG. Fig.6D As shown, only the second animated image of the healthy side can be displayed in the graphical interface client. When the corners of the mouth on the healthy side in the second animated image overlap with the corners of the mouth on the healthy side in the first animated image, it is a symmetrical smile. In another embodiment, the second animated image superimposed on the patient's real-time facial image and the first animated image can also be a second animated image of the whole face (including the healthy side and the affected side), in which case the first animated image of the affected side overlaps with the second animated image of the affected side. It is also possible not to display the patient's real-time facial image, but only to display the superimposed first and second animated images, so that the patient no longer relies on the feedback of the actual smile through the real-time facial image.
[0119] In another embodiment, a graphical interface client can be used to feed back a graphic (such as a seesaw or a ball) image to guide the patient's training. When the seesaw is in a horizontal state, or the ball is at the zero position, it is a symmetrical smile. Graphic feedback can also make the patient no longer rely on the real smile fed back by real-time facial images, and can increase the fun of training, which is especially suitable for children.
[0120] The training method of the present invention changes the previous training method of using the healthy side to guide the affected side. Based on the actual facial muscle function of the affected side, a visual animated image is generated for the patient to guide the patient to control the facial expression amplitude of the healthy side. The animated image provides the patient with objective and highly refined visual feedback, and facial muscle training does not rely on subjective evaluation.
[0121] The present invention can also remind patients of problems in real time during the training process, and perform quantitative scoring based on facial key points detected during each training session, to help patients understand their own conditions and training progress.
[0122] In this embodiment, based on the starting time of the movement of the key points of the corners of the mouth on both sides, the time taken to complete the command action, and the displacement value in the face coordinate system, the patient is fed back with the starting synchronization score, the command action completion speed score, and the movement amplitude score of the movement of the corners of the mouth on both sides.
[0123] In this embodiment, the directional score of the corners of the mouth on both sides is also fed back to the patient based on the displacement amplitude of the key points of the lip on the affected side and the corresponding key points of the lip on the healthy side. The coordinate value of the key points of the corners of the mouth on the healthy side before smiling is (x1, y1), and the coordinate value of the key points of the corners of the mouth on the healthy side after smiling is (x1′, y1′). Then the smiling angle of the key points of the corners of the mouth on the healthy side is The movement directionality score of the corners of the mouth on both sides is scored based on the difference in smile angles of the key points of the corners of the mouth on the healthy side and the affected side. The larger the difference, the lower the movement directionality score.
[0124] Embodiment 4
[0125] In this embodiment, the score of facial symmetry is not less than 4 points (this is only an example and is not a limitation of the present invention), and the training content includes: relaxation training and eyelid closure training on the affected side (the order and number of each training are not limited). Since there is usually a gap when the eyelid on the affected side is closed, eyelid closure training on the affected side can speed up the eye closure speed on the affected side and improve the degree of eyelid closure. Specific training methods include:
[0126] Instruct the patient to perform eyelid closure training on the affected side. Detect the eyelid key points on the affected side in real time and calculate the eye opening degree on the affected side. In this embodiment, if the eye opening degree value on the affected side no longer changes within the set time (the eye opening degree on the affected side at this time is the minimum value during the training process), it is determined that the eyelid closing action is completed. Based on the time taken for the affected side to complete the eyelid closing action, the eyelid closing speed score on the affected side is fed back to the patient.
[0127] Based on the degree of eye opening on the affected side, feedback is given on the degree of eye closure on the affected side. If the degree of eye opening on the affected side is less than 1 mm when the eye closure action is completed (this is only an example and is not a limitation of the present invention), a score of "80 points" is given through voice feedback. The degree of eye closure on the affected side can also be fed back to the patient in real time through a "beep" sound: the more rapid the "beep" sound, the better the eye closure; or the louder the sound, the better the eye closure.
[0128] At the same time, in order to prevent the affected side from closing the eye too hard, causing the corner of the mouth on the affected side to rise too much (facial muscle linkage), during the eyelid closing training on the affected side, the displacement amplitude of the corner of the mouth on the affected side will be detected in real time. When it exceeds the preset threshold, an alarm or prompt will be issued.
[0129] Embodiment 5
[0130] In this embodiment, the patient's affected side has obvious improvement, and the facial symmetry score is not less than 4 points (this is only an example and not a limitation of the present invention), and the training content includes: relaxation training, bilateral mouth corner movement coordination training and bilateral eyelid closure training (the order and number of each training are not limited). This training content is also called the second stage smile training.
[0131] The steps of bilateral eyelid closure training include:
[0132] Instruct the affected side to perform bilateral eyelid closure training.
[0133] Based on the time taken by the affected side and the healthy side to complete the eyelid closing action, the patient is fed back the eyelid closing time difference score, the eyelid closing speed score of the affected side, and the eyelid closing speed score of the healthy side. At the same time, based on the degree of eye opening on the affected side, the eyelid closing degree score of the affected side is also fed back to the patient.
[0134] Embodiment 6
[0135] In this embodiment, the facial symmetry score is not less than 5 points (this is only an example and not a limitation of the present invention), and the patient is trained to smile with emotion, which is also called the third stage smile training. The training content includes:
[0136] Instruct the patient to perform emotional smile training. The superimposed first animation image and real-time facial image are displayed through a graphical interface client. Based on the virtual eyes and virtual lips in the first animation image, the patient is guided to smile. While paying attention to the smile amplitude of the healthy side corner of the mouth, the patient also needs to pay attention to the squinting degree of the healthy side (because when people smile happily, their eyes will squint involuntarily).
[0137] After several training sessions, the patients were asked to listen to music, jokes, stories, etc. to induce an emotional smile (a smile of inner joy). The weighted sum of the starting synchronization score of the corners of the mouth, the speed score of the command action completion, the movement amplitude score, the direction score, and the eye opening score was calculated to obtain the score of the emotional smile and fed back to the patients.
[0138] The present invention also provides a facial paralysis rehabilitation training system for implementing the facial paralysis rehabilitation training method of the present invention. Fig. 9 As shown, the training system comprises:
[0139] An image acquisition module 11 is used to acquire facial images of patients with facial paralysis;
[0140] An image processing module 12, signal-connected to the image acquisition module 11, for detecting key points 1 in the facial image and transforming the facial image into a face coordinate system;
[0141] The facial paralysis assessment module 13 is connected to the image processing module 12 by signal, and obtains the patient's facial symmetry assessment result through a deep learning method based on the key point coordinate values in the face coordinate system;
[0142] The training guidance module 14 is connected to the image processing module 12 and the facial paralysis assessment module 13 by signals; based on the patient's recovery time and / or the symmetry assessment result, the training task is pushed to the patient; the training guidance module 14 also generates an animated image for guiding the training of the affected side and / or the healthy side facial muscles based on the key point coordinate values and / or the healthy side projection point coordinate values in the face coordinate system; wherein, by taking the y-axis of the face coordinate system as the symmetry axis, the key points of the affected side in the face coordinate system are projected to the healthy side to obtain the healthy side projection points;
[0143] The display module 15 is signal-connected to the image acquisition module 11 and the training guidance module 14, and is used to visually display the patient's facial image and / or the animation image.
[0144] The facial paralysis rehabilitation training system and method of the present invention can perform machine learning based on the patient's facial image, and can provide a relatively accurate quantitative evaluation result of facial symmetry without the doctor's subjective judgment. At the same time, it can judge the patient's disease progression based on the evaluation result and push corresponding rehabilitation training tasks to the patient.
[0145] The present invention also uses the actual facial muscle function of the affected side as a reference to generate a visualized animated image to guide the facial expression of the healthy side, and helps the patient to generate a symmetrical smile in the manner of "the affected side guides the healthy side". At the same time, the present invention can also timely feedback the difference in facial muscle function between the affected side and the healthy side through animated images, sounds, etc., to help the patient to adjust the direction and strength of the facial muscle force in time. The present invention can ultimately help the patient get rid of the dependence on visual feedback and make a satisfactory smile with emotion.
[0146] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0147] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A facial paralysis rehabilitation training method, characterized in that: Contains steps: Collecting a facial image of a patient and detecting key points in the facial image; Establishing a facial coordinate system, calculating the coordinate values of the key points in the facial coordinate system, and obtaining a symmetry assessment result of the patient's face through a deep learning method based on the coordinate values; Pushing a training task including at least one training content to the patient based on the patient's recovery time and / or the symmetry assessment result; Detect key points of the patient during training, project the key points of the affected side in the face coordinate system to the healthy side with the y-axis of the face coordinate system as the symmetry axis, and obtain corresponding projection points; based on the coordinate values of the key points in the face coordinate system and / or the healthy side projection points, generate an animated image for guiding the training of the affected side and / or the healthy side facial muscles; Feedback the animated images to the patient in real time through a graphical interface client, and quantitatively evaluate the training effect; The animated image includes a first animated image; the first animated image includes one or more virtual eyebrows, eyes, nose, and lips images generated based on the key points of the affected side and the projection points of the healthy side in a facial coordinate system; the first animated image changes with the facial expression of the affected side, and the first animated image guides one or more of the starting time point, direction, speed, expected arrival position, eye opening degree, and eyebrow height of the corner of the mouth movement on the healthy side.
2. The facial paralysis rehabilitation training method according to claim 1, characterized in that: The animated image also includes a second animated image; the second animated image includes one or more virtual eyebrows, eyes, nose, and mouth images generated based on the key points of the affected side and the healthy side in the face coordinate system; the second animated image changes with the facial expressions of the affected side and the healthy side.
3. The facial paralysis rehabilitation training method according to claim 2, characterized in that: The animation image further includes a graph, which changes its movement direction based on the displacement of key points of the corners of the mouth on both sides; the movement amplitude, movement speed and symmetry of the corners of the mouth on both sides are fed back through the movement of the graph.
4. The facial paralysis rehabilitation training method according to claim 1, characterized in that: The training content is relaxation training; the training method further comprises: Taking the facial image of the patient in a relaxed state as the first facial image; calculating the coordinate values of key points in the first facial image in a face coordinate system; Acquire a facial image of the patient as a second facial image in real time; calculate the coordinate values of key points in the second facial image in a face coordinate system; Comparing the displacement values of the corresponding key points in the first and second facial images in the face coordinate system; performing weighted calculation on the displacement values of various facial regions of the patient to obtain a muscle tension score value of the corresponding region; The second facial image in the face coordinate system is fed back through the graphical interface client. If the muscle tension score value of a certain facial area is higher than the set muscle tension threshold, the area is marked on the second facial image.
5. The facial paralysis rehabilitation training method according to claim 3, characterized in that: The training content includes strength training of the affected side corner of mouth, and the training method further includes: Calculate the displacement value of the key point of the corner of the mouth on the affected side; and feed back the image of the graphic or the real-time facial image of the patient to the patient through a graphical interface client; Based on the displacement value, providing the patient with feedback on the recovery degree of the affected side of the mouth angle; The degree of eye opening on the affected side is calculated based on the key points of the upper and lower eyelid margins on the affected side. When the degree of eye opening on the affected side is less than the set eye opening threshold, a prompt or alarm is issued.
6. The facial paralysis rehabilitation training method according to claim 3, characterized in that: The training content includes bilateral corner-of-mouth movement coordination training, and the training method further includes: Irregularly giving the patient instructions to smile, laugh moderately, or laugh loudly; Feedback a facial image of the patient in a face coordinate system through a graphical interface client, and superimpose a first animation image on the facial image; or feed back a second animation image through a graphical interface client, and superimpose the first animation image on the second animation image; or feed back an image of the graphic through a graphical interface client; Based on the starting time of the movement of the key points of the corners of the mouth on both sides, the time taken to complete the command action, and the displacement value in the face coordinate system, the starting synchronization score, the command action completion speed score, and the movement amplitude score of the corners of the mouth on both sides are fed back to the patient; Based on the displacement amplitude of the key points of the affected lip and the corresponding key points of the healthy lip, the directional scores of the movement of the corners of the mouth on both sides are fed back to the patient.
7. The facial paralysis rehabilitation training method according to claim 1, characterized in that: The training content includes eyelid closure training on the affected side, and the training method further includes: Based on the time taken for the affected side to complete the eyelid closure action, the affected side eyelid closure speed score was fed back to the patient; Based on the coordinates of the key points of the upper and lower eyelid margins of the affected side in the face coordinate system, the degree of eye opening on the affected side is calculated; and based on the degree of eye opening on the affected side, the degree of eyelid closure on the affected side is fed back to the patient.
8. The facial paralysis rehabilitation training method according to claim 1, characterized in that: The training content includes: bilateral eyelid closure training, and the training method includes: Based on the time taken by the affected side and the healthy side to complete the eyelid closing action, the patient is fed back the eyelid closing time difference score of the two eyes, the eyelid closing speed score of the affected side, and the eyelid closing speed score of the healthy side; Based on the coordinates of the key points of the upper and lower eyelid margins of the affected side in the face coordinate system, the degree of eye opening on the affected side is calculated; based on the degree of eye opening on the affected side, the degree of eyelid closure on the affected side is fed back to the patient.
9. A facial paralysis rehabilitation training system, used to implement the facial paralysis rehabilitation training method according to any one of claims 1 to 8, characterized in that: The training system comprises: An image acquisition module, used for acquiring facial images of patients with facial paralysis; An image processing module, signal-connected to the image acquisition module, for detecting key points in the facial image and transforming the facial image into a face coordinate system; The facial paralysis assessment module is connected to the image processing module by signal, and obtains the patient's facial symmetry assessment result through a deep learning method based on the key point coordinate values in the face coordinate system; A training guidance module, a signal connection image processing module and the facial paralysis assessment module; Pushing training tasks to the patient based on the patient's recovery time and / or the symmetry assessment result; The training guidance module also generates an animated image for guiding the training of the affected side and / or the healthy side facial muscles based on the key point coordinate values and / or the healthy side projection point coordinate values in the face coordinate system; the healthy side projection points are obtained by projecting the key points of the affected side in the face coordinate system to the healthy side with the y-axis of the face coordinate system as the axis of symmetry; The animated image includes a first animated image; the first animated image includes one or more of virtual eyebrows, eyes, nose, and lips images generated based on the key points of the affected side and the projection points of the healthy side in a face coordinate system; the first animated image changes with the facial expression of the affected side, and the first animated image guides one or more of the starting time point, direction, speed, expected arrival position, eye opening, and eyebrow height of the mouth corner movement of the healthy side; The display module is signal-connected to the image acquisition module and the training guidance module, and is used to visually display the patient's facial image and / or the animation image.
Citation Information
Patent Citations
Facial paralysis rehabilitation task auxiliary generation method and system based on medical big data
CN113362924A