Facial paralysis diagnosis rating method and system based on deep learning

Abstract

The application discloses a kind of facial paralysis diagnostic rating method and system based on deep learning, belong to facial paralysis identification technical field.Facial paralysis diagnostic rating method includes steps have S1: establish normal person under certain facial action motion model;S2: obtain the motion data of user under actual facial action;S3: the motion data of user is compared with normal person motion model, and comparison result is obtained;S4: the comparison result obtained in S3 is analyzed, and the facial paralysis degree conclusion of user is generated.Facial paralysis diagnostic rating system based on deep learning is used to realize the above method, including construction module, image acquisition module, diagnosis module, evaluation module and display module.The present scheme solves the problems of lack of intelligent facial paralysis identification scheme supporting multiple platforms and multiple models in the prior art, and lack of a unified high-precision and high-accuracy facial paralysis diagnostic rating system, reducing the human and material resources cost of frequent medical treatment of patients.

Description

Technical Field

[0001] This invention belongs to the field of facial paralysis recognition technology, specifically a facial paralysis diagnosis and rating method and system based on deep learning. Background Technology

[0002] Existing methods for identifying facial paralysis primarily rely on manual identification by doctors. In reality, patients need to make multiple trips to the hospital for diagnosis during the course of their illness, resulting in significant expenditure of human and material resources. Furthermore, due to the high degree of subjectivity in manual identification, different doctors have difficulty reaching a consensus on the diagnostic criteria for the degree of facial paralysis, which creates an insurmountable obstacle to the systematic diagnosis of facial paralysis.

[0003] Building upon the above, employing machine recognition methods is an inevitable direction for optimization. However, current machine recognition solutions are limited. Among the more mature and accurate solutions, such as the mobile rating device for facial paralysis recognition disclosed in the paper "Utilization of SmartphoneDepth Mapping Cameras for App-Based Grading of Facial Movement Disorders: Development and Feasibility Study," which uses iOS's Face ID technology to create a 3D model of the face for identification, this device has a significant limitation: it cannot be used on Android phones, computers, and older iPhones. Furthermore, most existing machine recognition technologies can only provide a rough diagnosis of the overall degree of facial paralysis, failing to pinpoint or predict the specific affected area. Consequently, they cannot consider or provide feedback on the crucial associated movements in facial paralysis, resulting in low accuracy and precision in facial paralysis recognition and hindering its application in subsequent treatment stages. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based method and system for diagnosing and rating facial paralysis. This method and system are not limited to a single device platform and can perform facial paralysis diagnosis and rating with high accuracy and precision. It can also predict and judge the possible affected areas of the patient. This solves the problems in the existing technology of lacking intelligent facial paralysis recognition solutions that support multiple platforms and multiple models, as well as the lack of a unified, high-precision, and high-accuracy facial paralysis diagnosis and rating system.

[0005] This invention is achieved using the following technical solution:

[0006] A deep learning-based diagnostic rating method for facial paralysis includes the following steps:

[0007] S1: Establish a motion model of a normal person under specific facial movements; that is, for a specific facial movement, establish a motion data model of the relevant muscles or muscle groups when a normal person performs the corresponding facial movement, where muscle groups refer to a collection of muscles.

[0008] S2: Obtain motion data of the user during actual facial movements; that is, for the actual facial movements performed, obtain the motion dataset of the relevant muscles or muscle groups when the user performs the corresponding facial movements, where the user refers to the user who uses this method to diagnose and rate facial paralysis, and can be a confirmed or unconfirmed facial paralysis patient.

[0009] S3: Compare the user's motion data with the motion model of a normal person to obtain the comparison results; that is, compare the motion datasets of muscles or muscle groups of the user and normal people under the same facial movements to find out the abnormal movement of muscles or muscle groups when the user performs a certain facial movement (or combination of movements).

[0010] S4: Analyze the comparison results obtained in S3 to generate a conclusion on the degree of facial paralysis of the user. The conclusion on the degree of facial paralysis mainly refers to the severity and probability of facial paralysis of the user, and also includes the possibility of having facial paralysis. The possibility of having facial paralysis is due to the fact that there is no unified standard for the diagnosis and rating of facial paralysis in the current technology. Therefore, this method integrates multiple authoritative standards and determines the possibility of having facial paralysis based on the actual situation of different users.

[0011] In the above scheme, steps S1-S4 enable the diagnosis and rating of the user's facial paralysis severity. In existing technologies for facial paralysis recognition, almost none use normal information modeling to process abnormal information. This scheme addresses the problems of insufficient action data and inconsistent action sequences for facial paralysis patients, which are currently the biggest limitations and influences on the accuracy of other schemes. For patients awaiting diagnosis of facial paralysis, it informs them whether they have symptoms and the possible severity, helping them seek timely treatment. For patients already diagnosed with facial paralysis, it allows them to determine whether their condition has improved or worsened based on the severity assessment, helping them take timely action. Therefore, this method enables users to self-diagnose and monitor their facial paralysis recovery process, while reducing the human and material costs associated with frequent outings for medical treatment and avoiding the psychological problems patients may face when seeking medical care outside the home.

[0012] Furthermore, the diagnostic rating method also includes step S5: obtaining information on the possible location of facial paralysis and related nerves of the user. Note that, since there is no unified standard or scheme for facial paralysis identification in the existing technology, the word "possibly" indicates that the conclusions obtained by this method do not represent industry standard conclusions, but are only for user reference and can serve as an auxiliary judgment for subsequent artificial etiological examination.

[0013] In the above scheme, this method can also, based on the comparison results obtained in S3, identify the specific muscles or muscle groups that are incorrectly or unable to move during a certain facial movement (or combination of movements), thereby achieving a preliminary analysis of the damaged areas of facial paralysis nerves and abnormal nerve repair. Therefore, this method, while achieving diagnosis and rating of the degree of facial paralysis, overcomes the limitations of existing technologies that only monitor the overall degree of facial paralysis, and achieves monitoring at the muscle level of facial paralysis, that is, down to the probability and severity of paralysis of each muscle or nerve, thus incorporating associated movements into the consideration and providing assistance for subsequent treatment; on this basis, because it comprehensively considers the movement of all major facial muscles under different movements, the accuracy of identifying associated movements is also excellent.

[0014] Furthermore, step S1 specifically includes the following sub-steps:

[0015] S11: Determine a specific facial movement plan; this plan should be selected by relevant personnel (physicians, scholars in the field, patients, program developers, etc.) in the early stages, and the selection criteria should be based on the ability to scientifically assess the degree of facial paralysis.

[0016] S12: Collect data samples of normal people performing specific facial movements in advance and label them. It should be noted that "normal people" here refers to a group, so there is no limit to the number of normal people samples. The standard for the number should be to establish a muscle movement model with a strong error tolerance in a scientific way. In addition, the data samples should be in video format, and the video should cover the entire process of performing specific facial movements.

[0017] S13: According to the given facial muscle segmentation standard, the facial muscles of the normal human data sample are segmented. The segmentation standard is also applied in step S2. The facial muscle segmentation standard is one of the existing mature standards. According to the standard, the corresponding area of ​​the face can be segmented. The specific segmentation method is one of the existing mature technologies.

[0018] S14: According to step S13, deep learning technology is used to obtain the real-time position of each muscle when a normal person performs a specific facial movement scheme from normal person images, so as to obtain the change trend of each muscle or muscle group when a normal person performs a specific movement.

[0019] S15: Perform parameter fitting on the action plan in S11 and the change trend obtained in S14, and then establish a muscle or muscle group movement model of a normal person under various specific facial movements through methods such as recurrent neural networks or machine learning. That is, a discrimination network for the change trend of corresponding muscles or muscle groups under various specific facial movements of a normal person. This movement model is used in step S3.

[0020] Furthermore, step S2 specifically includes the following sub-steps:

[0021] S21: Determine the actual facial movement plan; the actual facial movement plan is selected by the user.

[0022] S22: The user executes the action plan and collects self-action data, that is, during the execution, the user takes a picture of their face using a camera device to obtain a facial image of the user under actual facial movements; the picture can be taken in any well-lit place chosen by the user, thereby effectively protecting the user's privacy.

[0023] S23: Perform facial muscle segmentation on the facial image obtained in S22; the segmentation standard is the given facial muscle segmentation standard described in S13, that is, the segmentation standard here is always consistent with that in S13.

[0024] S24: According to step S23, deep learning technology is used to obtain the real-time position of each muscle when the user performs the actual action plan, so as to obtain the change trend of each muscle or muscle group when the user performs a certain action. This change trend is the motion data of the user under actual facial movements. This motion data is used in step S3.

[0025] Furthermore, in S21, the actual facial movement scheme is a specific facial movement in S1, or a movement or combination of movements involved in the user's existing facial paralysis symptoms, or other freely divisible movements or combinations of movements. The specific facial movements in S1 include movements or combinations of movements involved in the user's existing facial paralysis symptoms, as well as other freely divisible movements or combinations of movements. This is because the data in the normal person model must cover all movements that the user might perform.

[0026] Furthermore, in S3, the comparison results include the degree of motion, the number of motions, and the amplitude of motion;

[0027] The degree of motion is a discrete value of the motion amplitude at a certain stage of a certain action, defined by an individual. The specific calculation process for the discrete value of the degree of motion includes the following sub-steps:

[0028] S1 运动度: Calculate the difference between the user's actual range of motion and the standard range of motion of a normal person (the standard range of motion of a normal person is determined by referring to the normal person's motion model established in S1) during a certain phase of a certain action.

[0029] S2 运动度 : S1 运动度 The absolute value of the difference obtained is compared with the standard motion amplitude;

[0030] S3 运动度 : S2 运动度 The results of the comparison are matched with the established standards to obtain the discrete values ​​of the movement status of a certain muscle group. These discrete values ​​are used in step S4.

[0031] Furthermore, one method for calculating the number and amplitude of movements is to calculate the cumulative amplitude of the related muscles or muscle groups over the entire time domain of a certain movement.

[0032] A deep learning-based facial paralysis diagnostic rating system, applied to the aforementioned facial paralysis diagnostic rating method, includes:

[0033] Modules are built to create motion models for normal individuals;

[0034] The image acquisition module is used to acquire images of the user's facial movements.

[0035] The diagnostic module is used to compare the user's facial movement data with a normal person's movement model to determine any abnormalities in the muscles or muscle groups when the user performs facial movements.

[0036] The assessment module is used to determine the degree of facial paralysis in users and to obtain information on the possible locations of facial paralysis and related nerves.

[0037] The display module is used to realize human-computer interaction in the image acquisition module (i.e., the system instructs the user to take a picture and receives the picture completion information, the user selects the actual facial action plan, etc.), and displays the various conclusions obtained by the evaluation module.

[0038] In the above scheme, each module of the system corresponds to a step in the facial paralysis diagnosis and rating method, enabling the system to diagnose and rate the severity of facial paralysis in users and provide timely feedback on associated phenomena. For users of this system, the first step is to select an action plan based on their individual circumstances. Then, following the instructions, they take and upload images or videos of their actions. After a very short wait (less than one minute), they receive the diagnosis and rating results for the degree of facial paralysis. These results show the severity and probability of facial paralysis, the likelihood of its presence, and information about possible locations and related nerves. Based on this, undiagnosed patients can promptly identify their facial paralysis symptoms and severity; established patients can track their treatment progress and effectiveness. Furthermore, if deterioration occurs or new associated movements develop (incorrect muscle connections during nerve repair), they can also promptly detect these phenomena and seek medical attention.

[0039] The beneficial effects achieved by this invention are:

[0040] (1) The deep learning-based facial paralysis diagnosis and rating method uses normal information modeling to process abnormal information, that is, comparing the user's motion data with the normal person's motion model, to achieve accurate diagnosis and accurate rating of the degree of facial paralysis of the user; compared with traditional manual recognition technology or existing machine recognition technology, it breaks through the problem of insufficient motion data and inconsistent motion process of facial paralysis patients, and supports multiple platforms and multiple models, reducing the manpower and material costs of patients to frequently go out for medical treatment.

[0041] (2) Furthermore, this method takes into account medically important associated movements and achieves monitoring at the level of facial paralysis muscles. It comprehensively considers the movement of all major facial muscles under different actions, and has high precision and accuracy, which is helpful for the patient's subsequent treatment.

[0042] (3) The deep learning-based facial paralysis diagnosis and rating system can diagnose and rate the degree of facial paralysis in users and provide timely feedback on related phenomena. It has the advantages of easy operation for users, fast system calculation and processing speed, and good human-computer interaction performance. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps in the facial paralysis diagnosis and rating method of the present invention;

[0044] Figure 2 This is a flowchart of the sub-steps of step S1 in Examples 1 and 2;

[0045] Figure 3 This is a flowchart of the sub-steps of steps S2 to S4 and S5 in Example 1 / Example 2;

[0046] Figure 4This is a schematic diagram of the facial muscle division criteria used in Examples 1 and 2;

[0047] Figure 5 This is a schematic diagram of the Toronto Points of Reference;

[0048] Figure 6 This is a schematic diagram of the 98 facial feature points used in Examples 1 and 2. Detailed Implementation

[0049] To clearly illustrate the solution in this invention, further explanation is provided below with reference to the accompanying drawings:

[0050] Example 1

[0051] Please refer to Figures 1 to 6 The facial paralysis diagnosis and rating method based on deep learning described in this embodiment includes the following steps:

[0052] S1: Establish a motion model of a normal person under specific facial movements;

[0053] S2: Acquire motion data of the user's actual facial movements;

[0054] S3: Compare the user's motion data with the motion model of a normal person to obtain the comparison results;

[0055] S4: Analyze the comparison results obtained in S3 to generate a conclusion on the degree of facial paralysis of the user.

[0056] S1 specifically includes the following sub-steps:

[0057] S11: Referring to the more authoritative medical methods for diagnosing facial paralysis, specific facial movement patterns were determined to be the movement patterns used in the Toronto scoring system, such as... Figure 5 As shown, the procedure includes five actions: raising the forehead, gently closing the eyes, opening the mouth and smiling, baring the teeth, and sucking.

[0058] S12: Using existing datasets such as Genki4k, video clips were obtained of normal people performing five actions: raising their forehead, gently closing their eyes, opening their mouth and smiling, baring their teeth, and sucking.

[0059] S13: According to... Figure 4 The facial muscle segmentation criteria shown are used to segment facial muscles in all video segments obtained in step S12. The segmentation is implemented using a deep learning scheme for facial feature point annotation, specifically PFLD facial feature point detection, as described in this embodiment. Figure 6The 98 facial feature points shown are labeled in this way. In addition, for those skilled in the art, the division can also be achieved using the following three methods: First, using an open facial feature point detection interface to determine the muscle positions on the face in real time; second, establishing the recognition and judgment of muscle groups based on the recognition and judgment of facial action units (AUs) in the Facial Behavior Coding System (FACS). The principle is that the concepts of facial action units (AUs) and muscle groups (muscle combinations) under this method can be considered the same, both being recognized through neural networks; third, using the optical flow method to determine the muscle movement process based on the concept of optical flow in computer vision.

[0060] S14: Based on step S13, the muscle positions of each video segment under each action obtained in step S12 are determined in real time, thereby obtaining the real-time positions of each muscle when a normal person performs the Toronto scoring table action plan, and thus determining the changing trends of each muscle or muscle group when a normal person performs the Toronto scoring table action plan. In this embodiment, the PFLD facial feature point detection can achieve a real-time muscle position determination speed of over 60fps on a home computer, which is sufficient to obtain relatively accurate results, thereby further determining the changing trends of muscles or muscle groups through neural networks; in addition, for those skilled in the art, the real-time changing trends of muscles or muscle groups can also be determined by recognizing and judging the facial action unit AU.

[0061] S15: Using common recurrent neural networks such as RNNs, LSTs, and PERFORMERs, parameter fitting is performed on the action scheme in S11 and the change trend obtained in S14 to establish a muscle or muscle group movement model for normal individuals under various specific facial movements. This is essentially a discriminant network for identifying the change trends of corresponding moving muscles or muscle groups under various specific facial movements in normal individuals. Alternatively, those skilled in the art can also establish a normal individual movement model by using machine learning to determine simple area changes.

[0062] S2 specifically includes the following sub-steps:

[0063] S21: Determine the actual facial movement scheme. In this embodiment, the actual facial movement scheme is also the movement scheme in the Toronto scoring table. It should be noted that the actual facial movement scheme here is selected by the user and can be consistent with the specific facial movement scheme in S1, or the movement or movement combination involved in the user's existing facial paralysis symptoms, or other freely divided movement or movement combination.

[0064] S22: The user freely selects a well-lit environment, and then the user performs five actions: raising the forehead, gently closing the eyes, opening the mouth and smiling, baring the teeth, and sucking. During the performance, the user takes a picture of their face using a camera device to obtain a facial image of the user under the actual facial movements.

[0065] S23: Same as step S13, perform facial muscle segmentation on the facial image obtained in S22.

[0066] S24: Same as step S14, obtain the real-time position of each muscle when the user performs the actual facial movement plan, and then obtain the change trend of each muscle or muscle group when the user performs the actual facial movement plan. This change trend is the motion data of the user under the actual facial movement.

[0067] In step S3, the comparison results include degree of motion, number of motions, and amplitude of motion. For ease of explanation, degree of motion is defined as a discrete value of the amplitude of motion at a specific stage of a given action, artificially defined. This is because each action includes multiple motion stages, and the amplitude of motion varies in each stage. The specific calculation process for the discrete value of degree of motion includes the following sub-steps:

[0068] S1 运动度 : Calculate the difference between the user's actual range of motion and the standard range of motion of a normal person during a certain phase of a certain action;

[0069] S2 运动度 : S1 运动度 The absolute value of the difference obtained is compared with the standard motion amplitude;

[0070] S3 运动度 : S2 运动度 The results of the comparison are matched with the established standards to obtain the discrete values ​​of the movement status of a certain muscle group. These discrete values ​​are used in step S4.

[0071] To facilitate the explanation of the above calculation process, this embodiment uses actual data from normal individuals and users when performing the Toronto scoring table action plan. For example, the values ​​of some action data when normal individuals, Case 1, Case 2, and Case 3 perform the forehead-raising action are as follows:

[0072] In a normal individual, the standard number of movements of the right frontalis muscle is 3, and the standard range of motion is {[(+0.078521)-(+0.171291), (+0.487121)-(+0.68172)], [(-0.018521)-(+0.02871), (-0.187121)-(-0.291817)]}; In Case 1, the actual number of movements of the right frontalis muscle was 2, and the actual range of motion was {[+0.057918, +0.45 9181]}; Case 2: The actual number of movements of the right frontalis muscle was 1, and the actual range of motion was {[-0.037918, +0.259181]}; Case 3: When performing the head-raising movement, abnormal movement of the right upper lip muscles occurred, with an actual number of movements of 3 and an actual range of motion of {[+0.478121, +0.489181], [-0.387121, -0.319021], [+0.371121, +0.389111]}. The value of the number of movements represents the number of movement stages during the performance of the movement. For example, if the actual number of movements of the right frontalis muscle in Case 1 was 2, it means that Case 1 experienced a total of 2 movement stages when performing the head-raising movement. In this embodiment, based on the above data, the discrete values ​​of the degree of motion in each movement stage of Case 1 and Case 2 (Case 3 showed abnormal movement of the right upper lip levator muscle group, which can be preliminarily judged as synergistic movement; synergistic movement is calculated using a specialized calculation system) under the forehead-raising movement were calculated. For a more intuitive explanation, the following division can be made:

[0073] If the absolute value of the difference between the actual range of motion of the right frontalis muscle and the standard range of motion during the head-raising exercise is greater than or equal to 110% of the standard range of motion, then the range of motion is taken as -1.

[0074] If the absolute value of the difference between the actual range of motion of the right frontalis muscle and the standard range of motion during the forehead elevation exercise is greater than or equal to 70% of the standard range of motion, then the range of motion is taken as 0.

[0075] If the absolute value of the difference between the actual range of motion of the right frontalis muscle and the standard range of motion during the head-raising exercise is greater than or equal to 50% of the standard range of motion, then the range of motion is taken as 1.

[0076] If the absolute value of the difference between the actual range of motion of the right frontalis muscle and the standard range of motion during the head-raising exercise is greater than or equal to 30% of the standard range of motion, then the range of motion is taken as 2.

[0077] If the absolute value of the difference between the actual range of motion of the right frontalis muscle and the standard range of motion during the head-raising exercise is greater than or equal to 10% of the standard range of motion, then the range of motion is taken as 3.

[0078] In step S4, the conclusion on the degree of facial paralysis includes the severity of facial paralysis. In this embodiment, based on step S3, the severity of facial paralysis in the conclusion on the degree of facial paralysis is further subdivided into six types according to the Toronto scoring system: normal movement, almost complete movement, deviated movement, slight movement, no movement, and associated movement, according to four stages: none, mild, moderate, and severe. Among them, associated movement is independent of the other five types of movement and uses a special calculation system.

[0079] From an algorithmic perspective, the five severity levels—normal movement, almost complete movement, slight movement, minor movement, and no movement—can be understood as corresponding to a discrete form of movement normality. This correspondence is freely defined by experts (physicians, scholars in the field, patients, program developers, etc.). The discrete values ​​of movement normality are calculated by the ratio between the sum of movement degrees and the number of movements multiplied by a coefficient. The sum of movement degrees refers to the sum of the movement degree values ​​(floating-point values) across all movement phases when performing a certain action. The coefficient value is adapted to the corresponding relationship (as mentioned above: the five severity levels are understood as corresponding to a discrete form of movement normality, and this correspondence is freely defined by experts (physicians, scholars in the field, patients, program developers, etc.) and determined by experts according to a specific dataset. In this embodiment, the ratio of normal movement, almost complete movement, minor movement, no movement, and slight movement to deviation movement is defined as follows:

[0080] Normal exercise: The sum of positive exercise degree values ​​is greater than or equal to the number of exercises * 2.9, and less than the number of exercises * 3.2;

[0081] Almost complete motion: The sum of positive motion degree values ​​is greater than or equal to the number of motions * 2.6, and less than the number of motions * 2.9;

[0082] Mild exercise: The sum of positive exercise intensity values ​​is less than or equal to the number of exercises * 2.1, and greater than or equal to the number of exercises * 1.3;

[0083] No movement: The sum of the positive values ​​of movement degree is less than or equal to the number of movements * 1.3;

[0084] Offset motion: The degree of motion produces a negative value.

[0085] Substituting the data of all facial muscle groups involved in movement in Cases 1 and 2 into the aforementioned ratio relationship, we obtained the normal movement, almost complete movement, deviation movement, slight movement, and no movement status for each muscle group. For a specific movement, since it may involve multiple muscle groups, after obtaining the movement status of each muscle group, a certain coefficient was multiplied. The proportion of each muscle group's movement status to the overall movement status can be determined by facial anatomy and physician experience. Therefore, the forehead-raising movement in Case 1 was judged as initiating slight movement, and the forehead-raising movement in Case 2 was judged as no movement. Furthermore, because Case 3 exhibited abnormal movement of the right upper lip muscles during the forehead-raising movement, the forehead-raising movement in Case 3 was judged as producing associated movement.

[0086] It is important to note that in actual machine judgment and algorithmic processes, the normality of movement is not presented in a discrete form. For example, the five discrete levels in the program—normal movement, almost complete movement, offset movement, slight movement, and no movement—will be floating-point values ​​from 1 to 5. The value of the degree of movement, however, is also a floating-point value in the program, representing the ratio of the absolute value of the difference between the actual movement amplitude and the standard movement amplitude at a certain movement stage to the difference in the standard movement amplitude. Based on this, the normality of movement is calculated as the sum of the degrees of movement multiplied by the number of movements and a coefficient. Furthermore, the normality of associated movements is calculated as the product of the ratio of the amplitude of the associated movement to the maximum width of the face and a coefficient. In summary, the discrete values ​​of the degree of movement and the normality of movement mentioned in this article are for ease of explanation and understanding. In the actual operation of the facial paralysis diagnosis and rating method, relying on pixel-level detection, the obtained values ​​are not coarse discrete values. Therefore, this method has higher accuracy and precision compared to manual scoring and traditional scoring methods.

[0087] In step S3, the comparison results also include the number of movements and the amplitude of movements. One method for calculating the number of movements and the amplitude of movements is to calculate the cumulative amplitude of the related muscles or muscle groups in the entire action time domain under a certain action. Further, if deep learning technology is used in steps S13 and S23 to establish the recognition and judgment of muscle groups using facial action units (AU), then the intensity of the facial action unit AU can be calculated accordingly here. If deep learning technology is used in steps S13 and S23, the muscle group structure can also be annotated in real time using facial feature point detection methods to determine the shape and centroid, and to calculate muscle movement; or the concept of optical flow in computer vision can be used to directly calculate the related muscle groups. The calculation process is implemented using the optical flow method. Specifically: first, the changes in the pixels occupied by a certain muscle group in the image sequence in the action time domain and the correlation between adjacent frames are obtained, thereby finding the correspondence between the previous frame and the current frame, and thus calculating the movement information of the muscle groups between adjacent frames; furthermore, using the above... After identifying the muscle group using one of the three methods, the same action in the dataset and the test set (i.e., normal human action data in S12 and patient action data in S22) is scaled down to the same number of frames. After calculation, the motion information can be summarized as a k*m*n vector matrix, where k represents the number of muscle groups, n represents the total number of pixels occupied by a certain muscle group during the entire movement, and m represents the total number of frames of the entire movement. For this matrix, for a certain muscle group, the average motion value of the vectors in the range of the muscle group marked in step S13 / step S23 at a certain time is calculated. Then, the motion vectors describing the muscle group in m frames can be extracted from the n*m ​​matrix, that is, a vector array of m elements. Then, the m vectors are analyzed to obtain the sequence changes, and the number and amplitude of the movement can be extracted.For example, in Case 4, during the forehead-raising exercise, the frontalis muscle movement sequence is [(0,0),(0,+0.000125),(+0.001281,+0.0000291),(+0.01819,+0.00139),……,(+0.000013,+0.00000123),(0,0),……]. The sequence between the two (0,0) vectors can be considered as a movement towards the (+,+) direction (i.e., to the upper right), with an amplitude equal to the sum of the intermediate vectors of the two (0,0) vectors, i.e., (+0.371,+0.571). For the entire sequence, following the above calculation method, n such as (+0.3... If the movement process is (71, +0.571), then n is the number of movements of the frontalis muscle when Case 4 performs the forehead raising exercise. The range of values ​​such as (+0.371, +0.571) represents the range of motion for each movement stage. By statistically analyzing the number of movements and range of motion of all related muscle groups during the forehead raising exercise, we can obtain the number of movements and range of motion of all muscle groups in Case 4 when performing the forehead raising exercise. Through the above calculation method, we can also calculate the number of movements and range of motion of all muscle groups that should occur in a normal person under a certain movement. Therefore, we can compare the data of normal people with the data of Case 4 to obtain the abnormal movement information of Case 4, and calculate the normality of movement of Case 4.

[0088] In summary, through steps S1-S4, the degree of facial paralysis in users can be diagnosed and rated. This solution can solve the problems of insufficient movement data and inconsistent movement procedures for facial paralysis patients, and the accuracy and precision of the diagnostic rating results are high.

[0089] Example 2

[0090] Reference Figures 1 to 6 Based on Example 1, the deep learning-based facial paralysis diagnosis and rating method in this example further includes the following steps:

[0091] S5: Obtain information on the possible locations of facial paralysis and related nerves in the user. That is, identify the specific muscles or muscle groups that are incorrectly or unable to move during a certain facial movement (or combination of movements), thereby achieving a preliminary analysis of the damaged areas of the facial nerves and abnormal nerve repair.

[0092] For example, in this embodiment, Case 5 performs step S22, that is, Case 5 performs five actions respectively: raising the forehead, gently closing the eyes, opening the mouth and smiling, baring the teeth, and sucking. When the movement patterns of multiple movements are combined, according to the calculation method described in Example 1, the normality of movement in Case 5 is obtained as <1,5,6,48.567>, <0,98.198>, <1,5,-3,69.196>, <2,47,8,29,4,47.981>, <0,97.463>. The structural standard is <number of abnormal muscles, abnormal muscle 1 number, movement pattern of abnormal muscle 1 (negative numbers represent the occurrence of synergistic phenomena where movement should not occur, different numbers represent the matching pattern of the movement, for example, 1 is rapid contraction and continuous tension, 2 is the main muscle body rapidly jumping upward and then downward once, ...), abnormal muscle 2 number, movement pattern of abnormal muscle 2 ..., total normality score of movement>. The determination of the normality of movement can be freely determined by experts to divide the contribution relationship between the number and degree of movement of each muscle group.

[0093] First, as described in step S4, the obtained normality of movement is analyzed and calculated to generate a conclusion on the user's degree of facial paralysis. This conclusion includes the severity and probability of the user's facial paralysis, as well as the likelihood of having facial paralysis. After calculation, the most likely degree of facial paralysis in Case 5 is severe, and the conclusion "You may have severe facial paralysis, with a probability of 95.81% and a probability of severe facial paralysis of 78.56%" is fed back to the user. The calculation of the probability and degree of facial paralysis is based on the weights of each movement in the Toronto scoring table, combined with the proportional coefficients set by the physician. Furthermore, compared to methods such as the Toronto scoring table used for manual scoring, this system relies on pixel-level calculations, meaning that each value is a floating-point number rather than a simple discrete integer. Therefore, the relevant values ​​for facial paralysis have decimal parts compared to the Toronto scoring table.

[0094] Then, as described in step S5, the obtained normality of movement is analyzed and calculated to obtain the possible damaged muscles and related nerves of the user's facial paralysis, serving as an auxiliary judgment for subsequent artificial etiology examination. The specific calculation method is to comprehensively analyze all normality of movement involving a certain muscle to obtain the abnormal probability of that muscle and nerve. For example, for the frontalis muscle (number 5) in case 5, it is necessary to first extract the normality of movement involving the frontalis muscle from <1,5,6,48.567>, <0,98.198>, <1,5,-3,69.196>, <2,47,8,29,4,47.981>, <0,97.463>, and then substitute it into the calculation scheme: 1-{[SUM(correlation coefficient between related muscles and this movement * normality of movement *))+coefficient of associated movement *SUM(correlation coefficient between associated muscles and this movement * normality of movement that produces associated movement)] / total percentage should be the size under normal conditions (no associated movement)}, that is, 1-{[(0.832*48.56 7)+(0.3414*98.198)(The model in Action 2 also involves the frontalis muscle)+0.148*(1-0.042)*69.196] / [0.832*100+0.3414*100+0.958*100]}=0.56, that is, under this calculation scheme, the probability of abnormal frontalis muscle in Case 5 is 56.0%. Among them, the correlation coefficient between a certain muscle and a certain action can be obtained by calculating the Spearman rank correlation coefficient for each muscle group and each action in step S3. This method is also applicable to calculating abnormal muscle movement, that is, to determine whether a certain movement should occur (if it should not occur but occurs, it is an associated movement).

[0095] For cases 2 and 3 in Example 1, the above-mentioned principles and methods were used for calculation. It was determined that case 2 was more likely to have a conduction problem in the nerve related to the right frontalis muscle, and case 3 had a motor nerve misalignment to the right upper lip (significant synergistic movement).

[0096] In summary, the facial paralysis diagnosis and rating method in this scheme can also monitor the level of facial paralysis muscles based on the comparison results obtained in step S3, that is, to the probability of paralysis of each muscle or nerve. Therefore, on the basis of realizing the diagnosis and rating of the degree of facial paralysis, it breaks through the limitation of the existing technology that only monitors the overall degree of facial paralysis.

[0097] Example 3

[0098] A deep learning-based facial paralysis diagnosis and rating system, applied to the facial paralysis diagnosis and rating methods described in Examples 1 and 2, includes:

[0099] Modules are built to create motion models for normal individuals;

[0100] The image acquisition module is used to acquire images of the user's facial movements.

[0101] The diagnostic module is used to compare the user's facial movement data with a normal person's movement model to determine any abnormalities in the muscles or muscle groups when the user performs facial movements.

[0102] The assessment module is used to determine the degree of facial paralysis in users and to obtain information on the possible locations of facial paralysis and related nerves.

[0103] The display module is used to realize human-computer interaction in the image acquisition module (i.e., the system instructs the user to take a picture and receives the picture completion information, the user selects the actual facial action plan, etc.), and displays the various conclusions obtained by the evaluation module.

[0104] For Case 5 in Example 2, the construction module is used to establish a motion model of a normal person performing the action plan in the Toronto scoring table; the image acquisition module is used to acquire facial motion images (in video clip form) of Case 5 when performing five actions: raising the forehead, gently closing the eyes, opening the mouth and smiling, baring the teeth, and sucking; the diagnosis module is used to compare Case 5's facial motion data with the normal person's motion model to determine the abnormalities of muscles or muscle groups when Case 5 performs the aforementioned five actions; the assessment module is used to determine the degree of facial paralysis in Case 5 and to acquire information on possible facial paralysis sites and related nerves; the display module is used to instruct Case 5 to select an action plan, record an action video, and view the diagnosis and rating conclusion, namely, "You may have severe facial paralysis, with a 95.81% probability of having facial paralysis, a 78.56% probability of severe facial paralysis, and a 56.0% probability of frontalis muscle abnormality."

[0105] In summary, each module of the system corresponds to a step in the facial paralysis diagnosis and rating method, enabling the system to diagnose and rate the severity of facial paralysis in users and provide timely feedback on associated phenomena. For users of this system, the first step is to select an action plan based on their individual circumstances. Then, following the instructions, they take and upload images or videos of their actions. After a very short wait (less than one minute), they receive the diagnosis and rating results for the degree of facial paralysis. These results show the severity and probability of facial paralysis, the likelihood of its presence, and information about possible locations and related nerves. Based on this, undiagnosed patients can promptly identify their facial paralysis symptoms and severity; established patients can track their treatment progress and effectiveness. Furthermore, if deterioration occurs or new associated movements develop (incorrect muscle connections during nerve repair), they can also promptly detect these phenomena and seek medical attention.

[0106] Of course, the above description is only a preferred embodiment of the present invention and should not be considered as limiting the scope of the embodiments of the present invention. The present invention is also not limited to the above examples, and all equivalent changes and improvements made by those skilled in the art within the scope of the present invention should fall within the patent coverage of the present invention.

Claims

1. A deep learning-based facial paralysis diagnostic rating method, characterized by, Includes the following steps: S1: Establish a motion model of a normal person under specific facial movements; S1 specifically includes the following sub-steps: S11: Determine a specific facial movement plan; S12: Pre-collect or gather data samples of normal people performing specific facial movement schemes; S13: According to the given facial muscle classification criteria, the facial muscles of the normal human data sample are classified, and the classification criteria are also applied in step S2. S14: Based on step S13, obtain the real-time position of each muscle when a normal person performs a specific facial movement scheme, thereby obtaining the trend of change of each muscle or muscle group when a normal person performs a specific movement. S15: Perform parameter fitting on the action plan in S11 and the change trend obtained in S14 to establish a muscle or muscle group movement model for normal people under various specific facial movements. This movement model is used in step S3. S2: Acquire motion data of the user's actual facial movements; S2 specifically includes the following sub-steps: S21: Determine the actual facial movement plan; In S21, the actual facial motion scheme is the specific facial motion in S1; S22: The user executes the action plan and takes a picture of their face using a camera device during the execution, thereby obtaining a facial image of the user under actual facial movements; S23: Perform facial muscle segmentation on the facial image obtained in S22; S24: According to step S23, obtain the real-time position of each muscle when the user performs the actual action plan, so as to obtain the change trend of each muscle or muscle group when the user performs a certain action. The change trend is the motion data of the user under actual facial movements. The motion data is used in step S3. S3: Compare the user's motion data with the motion model of a normal person to obtain the comparison results; In S3, the comparison results include degree of motion, number of motions, and range of motion; The degree of motion is a discrete value of the motion amplitude at a certain stage of a certain action, defined by human intervention. The specific calculation process for the discrete value of the motion amplitude includes the following sub-steps: A: Calculate the difference between the user's actual range of motion and the standard range of motion of a normal person during a certain phase of a certain action; B: Compare the absolute value of the difference obtained in step A with the standard motion amplitude; C: Match the results of the comparison in step B with the established standard to obtain the discrete value of the amplitude of a certain muscle group's movement. This discrete value is used in step S4. The value of the number of motions represents the number of motion stages involved in performing an action; S4: Analyze the comparison results obtained in S3 to generate a conclusion on the degree of facial paralysis of the user; S5: Obtain information about the possible location of facial paralysis and related nerves in the user.

2. A deep learning-based facial paralysis diagnostic grading system for implementing the deep learning-based facial paralysis diagnostic grading method of claim 1, characterized by, include: Modules are built to create motion models for normal individuals; The image acquisition module is used to acquire images of the user's facial movements. The diagnostic module is used to compare the user's facial movement data with a normal person's movement model to determine any abnormalities in the muscles or muscle groups when the user performs facial movements. The assessment module is used to determine the degree of facial paralysis in users and to obtain information on the possible locations of facial paralysis and related nerves. The display module is used to enable human-computer interaction in the image acquisition module and to show the conclusions obtained by the evaluation module.

Images