Method of evaluating facial symmetry using landmark vectors in dynamic video, diagnostic evaluation method for grading facial paralysis using the same

Abstract

The present invention relates to a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images and a method for diagnosing the grade of facial paralysis using the same. More specifically, the invention relates to a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images and a method for diagnosing the grade of facial paralysis using the same, wherein the direction and displacement of landmarks in a video frame in an expression state are calculated using landmarks in a video frame in an expression state and landmark vectors in a video frame in an expression state, and facial symmetry is determined through a Movement Symmetry Score (MSS) calculated using the direction and displacement of landmarks in a video frame in an expression state, thereby enabling quantitative evaluation of the degree (grade) of facial paralysis in a patient with facial paralysis.

Description

Technology Field

[0001] The present invention relates to a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images and a method for diagnosing the grade of facial paralysis using the same. More specifically, the invention relates to a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images and a method for diagnosing the grade of facial paralysis using the same, wherein the direction and displacement of landmarks in a video frame in an expression state are calculated using landmarks in a video frame in an expression state and landmark vectors in a video frame in an expression state, and facial symmetry is determined through a Movement Symmetry Score (MSS) calculated using the direction and displacement of landmarks in a video frame in an expression state, thereby enabling quantitative evaluation of the degree (grade) of facial paralysis in a patient with facial paralysis. Background Technology

[0002] Facial paralysis refers to a condition in which facial muscle symmetry is impaired due to damage to the facial nerve. If not properly treated in the early stages, facial paralysis can lead to chronic sequelae such as psychological anxiety and depression. For this reason, it is crucial to accurately diagnose facial paralysis in the early stages, assess its progression, and provide timely rehabilitation treatment.

[0003] Methods used to evaluate facial paralysis include the House-Brackmann (HB) method and the Facial Nerve Grading System (FNGS) 2.0. In clinical practice, various visual evaluation methods are typically utilized, but most are subjective. Known objective methods for evaluating facial paralysis include tracking the face with optical markers, using optical scans (3D scans), and analyzing electromyography signals. However, these methods require additional equipment and a limited environment, which can cause discomfort to patients with facial paralysis.

[0004] Recently, research has been conducted on methods to evaluate facial paralysis in an unrestrained environment. One method for evaluating facial paralysis in an unrestrained environment is the analysis of RGB images. Most RGB image analysis methods conduct evaluations through AI-based learning and utilize 68 landmarks in static images.

[0005] The following is a brief introduction to prior studies evaluating facial paralysis.

[0006] 3D facial landmark detection methods are utilized for the evaluation of facial paralysis. 3D facial landmarks are a method for detecting the locations of key facial areas within an image. Previous research proposed a method that extracts a 2D heatmap of facial landmarks from a 2D image and then expands it into 3D information (see Non-Patent Literature 1). However, this approach requires a significant amount of computation. Studies that directly estimate 3D facial landmarks without going through a 2D heatmap (see Non-Patent Literatures 2 and 3) are also known. Methods that estimate 3D facial landmarks by referencing scanning data demonstrate faster speeds than methods that extract 2D heatmaps, but they are limited by the requirement of a scanning device. Another study directly estimates 3D facial landmarks from images. However, most methods focus on a limited number of 38 landmarks. More landmark information is required to accurately measure the progression of facial paralysis. Kartynnik et al. (see Non-patent Literature 4) proposed a 3D face landmark detector for 3D face mesh representation. It uses Blazeface (see Non-patent Literature 5) to detect faces and subsequently extract 3D landmarks. This method operates in real-time and detects 468 face landmarks and an additional 10 iris landmarks. MediaPipe provides this data as a modularized library.

[0007] In addition, numerical approaches can be utilized to evaluate facial paralysis. Research on numerical approaches to facial paralysis is primarily being conducted on still images. Guarin et al. (see Non-patent Literature 6) extracted and analyzed 68 facial landmarks from 2D images. Based on these extracted landmarks, a series of indicators for evaluating facial symmetry and function were estimated. Recently, a numerical approach to measuring facial paralysis using a new registration method utilizing 3D facial landmarks (see Non-patent Literatures 7 and 8) has been introduced. 478 3D facial landmarks were extracted from 2D RGB images captured at different times. To compare the same patient with facial paralysis at different times, iterative Closest Point (ICP) registration was performed, and facial distance and angle symmetry were calculated. This method numerically evaluates the degree of facial paralysis and compares it with data from different time points. Prior art literature

[0008] Patent Document 1 : KR 10-2003-0041131 A, May 23, 2003. Patent Document 2 : KR 10-2003-0029638 A, April 14, 2003. Patent Document 3 : KR 10-2023-0130926 A, September 12, 2023. Patent Document 4 : KR 10-2023-0076741 A, May 31, 2023.

[0009] 비특허문헌 1 : Tu, Xiaoguang; Zhao, Jian; Xie, Mei; Jiang, Zihang; Balamurugan, Akshaya; Luo, Yao; Zhao, Yang; He, Lingxiao; Ma, Zheng; Feng, Jiashi. "3D face recon struction from a single image assisted by 2D face im ages in the wild." IEEE Transactions on Multimedia, vol. 23, pp. 1160-1172, 202비특허문헌 2 : Bulat, A.; Tzimiropoulos, G. “Hierarchical binary CNNs for landmark localization with limited resources.” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, pp. 343?356, 2018비특허문헌3 : Colaco, SJ; Han, DS “Deep Learning-Based Facial Landmarks Localization Using Compound Scaling.” IEEE Access, vol. 10, 2022, pp. 7653?7663.비특허문헌 4 : Kartynnick, Y.; Ablavatsky, A.; Grishchenko, I.; Grund-mann, M. “Real-time facial surface geometry from monocular video on mobile GPUs.” arXiv preprint arXiv:1907.06724, 2019.비특허문헌 5 : Bazarevsky, Valentin; Kartynnik, Yuri; Vakunov, Andrey; Raveendran, Karthik; Grundmann, Matthias. “Blazeface: Sub-millisecond neural face detection on mobile gpus." arXiv preprint arXiv:1907.05047, 2019.비특허문헌 6 : Guarin, D. L., Dusseldorp, J., Hadlock, T. A., & Jowett, N. "A machine learning approach for automated facial measurements in facial palsy." JAMA Facial Plastic Surgery, vol. 20, no. 4, 2018, pp. 335-337.비특허문헌 7 : Woo. Soo Hyun et al., "Artificial intelligence-based numerical analysis of the quality of facial reanimation: Acomparative retrospective cohort study between one stage dual innervation and single innervation," Journal of Cranio-Maxillofacial Surgery, vol. 51, no. 5, pp. 265-271, Jul. 2023., DOI:10.1016 / j.jcms.2023.05.012비특허문헌 8 : Kim. Junsik et al., "Numerical approach to facial palsy using a novel registration method with 3D facial landmark," Sensors, vol. 22, no. 17, pp. 6636, Sep. 2022., DOI:10.3390 / s22176636. 해결하려는 과제

[0010] The first problem (objective) to be solved by the present invention is to provide a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images, which can quantitatively evaluate the degree (grade) of facial paralysis in a patient by determining the left-right symmetry of the patient's face using a numerical approach based on the landmark vectors of a video frame in a neutral state and a video frame in an expression state, and a method for diagnosing and evaluating the grade of facial paralysis using the same.

[0011] Furthermore, the problems to be solved by the present invention are not limited to the problems described above, and various additional problems to be solved may be provided through the technologies described in the embodiments below. means of solving the problem

[0012] The present invention, according to an embodiment for achieving the above-mentioned objective, provides a method for evaluating facial left-right symmetry using landmark vectors of a dynamic video image, comprising: a process of calculating, respectively, landmark vectors of a video frame in a neutral state and a video frame in a facial expression state by utilizing the location of landmarks of a video frame in a facial expression state and the location of landmarks of a video frame in a neutral state, which are detected in a dynamic video image obtained by capturing the frontal view of a patient's face and continuously capturing a state of change from a neutral state to various expressions; a process of calculating the direction and amount of movement of landmarks of a video frame in a facial expression state using the landmark vectors of the video frame in a neutral state and the landmark vectors of the video frame in a facial expression state, and calculating a Movement Symmetry Score (MSS) using the calculated direction and amount of movement of landmarks of a video frame in a facial expression state; and a process of determining facial symmetry using the calculated Movement Symmetry Score.

[0013] In addition, the process of calculating the above displacement symmetry score can be calculated using the [mathematical formula] below.

[0014] [Mathematical Formula]

[0015]

[0016]

[0017]

[0018] Here, i is the landmark number of the video frame of the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is i It can represent the sum of the differences in left-right vertical directionality of the nth landmark.

[0019] In addition, in the process of determining the facial symmetry, the displacement symmetry score may have a value between 0 and 100, and it may be determined that the closer the value is to 100, the higher the symmetry.

[0020] In addition, the video frame of the expression state can be detected through a process of comparing the landmark of the video frame of the neutral state with the landmarks of subsequent video frames, and determining the subsequent video frame having the largest landmark movement relative to the landmark of the video frame of the neutral state among the subsequent video frames as the video frame of the expression state.

[0021] Additionally, the process of determining the video frame of the expression state may include: a process of calculating the landmark vector of the video frame of the expression state and the landmark vector of the subsequent video frames by utilizing the location of the landmark of the video frame of the expression state and the location of the landmark of the subsequent video frames; and a process of comparing the calculated landmark vector of the video frame of the expression state and the landmark vector of the subsequent video frames to detect the subsequent video frame among the subsequent video frames that has the largest total movement amount of the landmark based on the landmark vector of the video frame of the expression state, and determining it as the video frame of the expression state.

[0022] In addition, the landmark vector of the above-mentioned neutral image frame and the landmark vector of the above-mentioned subsequent image frame can be calculated using the [mathematical formula] below.

[0023] [Mathematical Formula]

[0024]

[0025] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( It can be expressed as the value excluding ).

[0026] In addition, the amount of movement of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0027] [Mathematical Formula]

[0028]

[0029] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression statei It can indicate the location of the th landmark.

[0030] In addition, the total movement amount of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0031] [Mathematical Formula]

[0032]

[0033] In addition, among the aforementioned subsequent video frames, the subsequent video frame with the largest total movement of the landmark based on the landmark vector of the aforementioned video frame in a neutral state can be detected using the [mathematical formula] below.

[0034] [Mathematical Formula]

[0035]

[0036] In addition, the landmarks of the video frames in the above-mentioned expressionless state and the landmarks of subsequent video frames may total 478 by combining 468 face landmarks and 10 eye landmarks.

[0037] In addition, according to another embodiment for achieving the above-mentioned purpose, the present invention provides a method for diagnosing and evaluating the grade of facial paralysis, comprising: a process of acquiring dynamic video images by continuously capturing a state of change from a neutral state to various expressions while capturing the frontal view of a patient's face with facial paralysis; a process of extracting landmarks from the video frames of the neutral state and subsequent video frames, respectively, in the acquired dynamic video images; a process of comparing the landmarks of the extracted subsequent video frames with the landmarks of the video frames of the neutral state and determining the subsequent video frame having the largest landmark movement based on the landmarks of the video frames of the neutral state among the subsequent video frames as the video frames of the expression state; a process of aligning and registering the landmarks of the video frames of the neutral state and the landmarks of the video frames of the expression state; and a process of determining facial symmetry using a method that evaluates left-right facial symmetry by utilizing the landmark vectors of the dynamic video images. Effects of the invention

[0038] As described above, according to the method for evaluating facial left-right symmetry using landmark vectors of dynamic video images according to an embodiment of the present invention, 3D facial landmarks are extracted from a dynamic video (RGB video) to detect a video frame in an expression state, and then a landmark vector is calculated with a video frame in a neutral state to calculate a displacement symmetry score. By using the calculated displacement symmetry score to determine facial symmetry of a patient with facial paralysis, the degree of facial paralysis in the patient with facial paralysis can be quantitatively evaluated, and the degree of facial paralysis before and after treatment can be compared to contribute to evaluating the effectiveness of treatment, and thereby, it is possible to track how the degree of facial paralysis changes over time. Brief explanation of the drawing

[0039] FIG. 1 is a flowchart illustrating a method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the process of determining an image frame of an expression state according to an example of the present invention. FIG. 3 is a flowchart illustrating the process of comparing a set reference landmark of an image frame in a neutral state with landmarks of subsequent image frames according to an example of the present invention. FIG. 4 is a diagram showing a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention. FIG. 5 is a conceptual diagram showing the grade diagnosis evaluation process of facial paralysis according to an embodiment of the present invention. Figure 6 is a diagram showing the status of participants evaluated by a specialist clinician using the HB method and FNGS 2.0. Fig. 7 is a diagram showing the major landmark areas analyzed for facial expression evaluation. Figure 8 is an example diagram showing a graph of the landmark movement volume of one of the participants. FIG. 9 is a diagram showing an image frame of a detected facial expression state. FIG. 10 is a diagram showing the result of alignment between landmarks in a neutral state and a facial expression state. Figure 11 is a box plot showing the symmetry of movement amounts of participants according to three facial expressions. Figure 12 is a diagram showing the correlation between the Movement Symmetry Score (MSS) and the HB diagnosis result along with a linear regression line. Figure 13 is a diagram showing the correlation between the displacement symmetry score (MSS) and the FNGS 2.0 diagnostic results along with a linear regression line. Specific details for implementing the invention

[0040] The advantages and features of the present invention and the methods for achieving them will become clear from the embodiments described below in detail in conjunction with the accompanying drawings. Throughout this specification, the same reference numerals refer to the same components. Additionally, where "A and / or B" is written, it may mean both A and B, or either A or B. Furthermore, the size and shape of each component shown in the drawings may be exaggerated for convenience of explanation and are not intended to be limiting.

[0041] Changes in facial expressions of patients with facial paralysis can provide crucial information for analyzing the severity of the condition. Dynamic video footage obtained by capturing a patient's face typically begins with the patient in a neutral state and subsequently displays various expressions. However, not all dynamic video footage begins with a neutral expression and ends with an expression. In fact, in some dynamic video footage, the patient may not be in an expression state even in the final frame (remaining in a neutral state). To address this issue, the present invention proposes a method for detecting expressions in dynamic video footage and a method for evaluating facial symmetry by utilizing landmark vectors of the dynamic video footage.

[0042] Hereinafter, embodiments of the present invention will be described in detail using the attached drawings.

[0043] FIG. 1 is a flowchart illustrating a method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention.

[0044] Referring to FIG. 1, a method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention includes a process (S1) of detecting an image frame of an expression state in a dynamic video, and a process (S2) of evaluating left-right symmetry of the face using landmark vectors of the dynamic video image.

[0045] The process of detecting a video frame of an expression state in a dynamic video (S1) is a process of detecting video frames of an expression state and a non-expression state in a dynamic video, and includes a process of acquiring a dynamic video image by photographing the face of a patient with facial paralysis using a shooting device (not shown) (S11), a process of extracting landmarks from the acquired dynamic video image (S12), and a process of determining a video frame of an expression state using the extracted landmarks (S13).

[0046] The process of acquiring dynamic video images (S11) involves capturing the face of a patient with facial paralysis using a capturing device (such as a camera) to acquire dynamic video images. At this time, the capturing device captures the frontal view of the patient's face and continuously captures the state of change from an initial expressionless state to various expressions to acquire dynamic video images.

[0047] The process of extracting landmarks (S12) extracts landmarks including face landmarks and eye landmarks from video frames of dynamic video images. Here, the landmarks may be 3D landmarks. Landmarks are extracted from video frames of an initial neutral expression state and video frames of expression states captured thereafter, respectively. For example, a total of 478 landmarks can be extracted by combining 468 face landmarks and 10 eye landmarks from video frames of dynamic video images, i.e., RGB video images. In an embodiment of the present invention, the process of determining video frames of expression states (S13) can be performed using face landmarks.

[0048] FIG. 2 is a flowchart showing the process of determining an image frame of a facial expression state according to an example of the present invention.

[0049] Referring to FIGS. 1 and 2, the main objective of the process for determining an image frame of an expression state (S13) according to an example of the present invention is to accurately extract the image frame in which the change in expression is greatest. To this end, the process includes a process (S131) ​​of setting a landmark extracted from an initial image frame (Ft0) of a patient with facial paralysis in a neutral expression state as a reference landmark, a process (S132) of comparing the reference landmark set in the initial image frame (Ft0) in a neutral expression state with the landmarks of image frames (Ft1~FtN (N is an integer)) (hereinafter referred to as 'subsequent image frames') taken after the image frame (Ft0), and a process (S133) of detecting a subsequent image frame having the largest landmark movement (amount of movement) based on the reference landmark set among the subsequent image frames (Ft1~FtN) as a result of the comparison and determining it as an image frame of an expression state.

[0050] FIG. 3 is a flowchart illustrating the process of comparing a set reference landmark of an image frame in a neutral state with landmarks of subsequent image frames according to an example of the present invention.

[0051] As shown in FIG. 3, the process (S132) of comparing a reference landmark of a non-expression state image frame and a landmark of subsequent image frames according to an example of the present invention includes a process (S1321) of calculating a landmark vector using the position of a set reference landmark of a non-expression state image frame (Ft0) and the position of a landmark of subsequent image frames (Ft1~FtN), and a process (S1322) of comparing the vector of a set reference landmark of a non-expression state image frame (Ft0) and the vector of a landmark of subsequent image frames (Ft1~FtN).

[0052] As shown in FIG. 2, in the process (S133) of determining the image frame having the largest landmark movement (amount of movement) among subsequent image frames (Ft1~FtN) as the image frame of the expression state, the vector of a set reference landmark of the image frame of the expression state (Ft0) and the vector of a landmark of the subsequent image frames (Ft1~FtN) are compared with each other (S1322), and then the subsequent image frame having the largest landmark movement (amount of movement) based on the vector of a set reference landmark of the image frame of the expression state (FtO) is detected and determined as the image frame of the expression state.

[0053] In order to analyze landmark movement (amount of movement), an embodiment of the present invention calculates landmark vectors between consecutive video frames of a video image, for example, a viewpoint S And the terminal t The landmark vector operation in can be expressed as shown in [Equation 1] below.

[0054]

[0055] Here, i Vector of the th landmark is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( It can be expressed as the value excluding ).

[0056] To determine the video frame representing the facial expression state among the subsequent video frames (Ft1~FtN), the video frame exhibiting the greatest change in facial expression is extracted. In other words, the video frame with the largest landmark movement is extracted. To extract the video frame with the largest landmark movement, the displacement of the set reference landmark in the neutral-expression video frame and the displacement of the landmarks extracted from each of the subsequent video frames (Ft1~FtN) are calculated.

[0057] For example, among subsequent video frames (Ft1~FtN), k In the nth subsequent video frame i When calculating the movement amount of the i-th landmark, it can be expressed as [Equation 2] below. Here, k is included within the range of 0 to N, and i is as a landmark number, (Here, 468 is the number of extracted face landmarks) has a range.

[0058]

[0059] Here, Is k In the nth subsequent video frame i Indicates the location of the th landmark, is in a blank-faced video frame (FtO). i Indicates the location of the th landmark.

[0060] The amount of movement of landmarks in subsequent video frames (Ft1~FtN) can be calculated by subtracting the position of the landmark in each subsequent video frame (Ft1~FtN) and the position of the reference landmark set in the video frame (FtO) in a neutral state, as shown in [Equation 2] above.

[0061] In this way, the total displacement of each landmark in each subsequent video frame can be calculated using the displacement for each landmark in each subsequent video frame calculated individually. The total displacement of each landmark in each subsequent video frame ( ) is the sum of the movement amounts of all landmarks in the corresponding subsequent video frame, and can be calculated as shown in [Equation 3] below. For example, if the number of landmarks in each subsequent video frame is M (where M is an integer), the total movement amount of each landmark in the subsequent video frame can be calculated by summing the movement amounts of all M landmarks.

[0062]

[0063] The total movement of landmarks for subsequent video frames (Ft1~FtN) is calculated using [Equation 2] and [Equation 3] above. Based on the total movement of landmarks for each of the subsequent video frames calculated in this way, the video frame with the largest total movement of landmarks among the subsequent video frames is selected. F Find . This can be expressed as a mathematical formula as shown in [Equation 4] below.

[0064]

[0065] Here, F represents the video frame with the maximum total movement amount of the landmark among all subsequent video frames (Ft1~FtN), for example, among the subsequent video frames (Ft1~FtN). k It indicates that the total movement of the landmark in the i-th video frame is maximum. Through this method, the video frame of the facial expression state at the point in time with the greatest change in expression can be reliably extracted.

[0066] FIG. 4 is a schematic diagram showing a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention, and FIG. 5 is a schematic diagram showing the facial paralysis grade diagnosis evaluation process using the facial paralysis grade diagnosis evaluation system shown in FIG. 4, for each component.

[0067] Referring to FIGS. 4 and 5, a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention includes a landmark detection unit (11), an expression state detection unit (12), a landmark registration unit (13), a landmark vector calculation unit (14), and a displacement symmetry calculation unit (15). Additionally, it may further include a facial symmetry judgment unit (16) that determines the left-right facial symmetry of a patient with facial paralysis using a displacement symmetry score calculated through the displacement symmetry calculation unit (15).

[0068] The landmark detection unit (11) extracts landmarks including face landmarks and eye landmarks from dynamic video images obtained by continuously capturing the face of a patient with facial paralysis. The landmark detection unit (11) extracts landmarks from the image frame (FtO) in a neutral state and subsequent image frames (Ft1~FtN) in an expression state, respectively. For example, a total of 478 3D landmarks, including 468 face landmarks and 10 eye landmarks, can be extracted from each image frame of the dynamic video image, i.e., the RGB video image.

[0069] The facial expression state detection unit (12) sets a landmark detected in a video frame (Ft0) of a neutral expression state as a reference landmark, and compares the reference landmark of the video frame (Ft0) of the neutral expression state set in this way with the landmarks detected in subsequent video frames (Ft1~FtN) respectively. Then, based on the comparison result, it detects the video frame among the subsequent video frames (Ft1~FtN) that has the largest landmark movement, i.e., the amount of landmark movement, based on the set reference landmark. Afterward, the detected subsequent video frame is determined as the video frame of the facial expression state.

[0070] The landmark register (13) registers the landmarks of the image frame (FtO) of the expression state detected by the expression state detection unit (12) and the landmarks of the image frame of the expression state determined by the expression state detection unit (12) by aligning them. Here, the reason for aligning and registering the reference landmarks of the image frame (FtO) of the expression state and the landmarks of the image frame of the expression state determined by the expression state detection unit (12) is to correct the shaken images during the process in which a patient with facial paralysis changes from an expressionless face to an expressionful face.

[0071] The landmark vector calculation unit (14) calculates the landmark vector of the image frame in a neutral state and the landmark vector of the image frame in a positive state by using the location of the landmark of the image frame (Ft0) in a neutral state registered in the landmark matching unit (13) and the location of the landmark of the image frame in a positive state determined through the previous expression state detection unit (12).

[0072] The movement amount symmetry calculation unit (15) calculates the landmark movement amount and analyzes the directionality using the landmark vector extracted from the image frame of the neutral state and the landmark vector extracted from the image frame of the facial state calculated through the landmark vector calculation unit (14), and calculates the movement symmetry score (MSS) using this.

[0073] The facial symmetry judgment unit (16) quantitatively judges and evaluates the symmetry of the facial patient's face using the movement amount symmetry score (MSS) calculated by the movement amount symmetry calculation unit (15).

[0074] Referring to FIGS. 1 and FIGS. 4, the process (S2) of evaluating facial left-right symmetry using landmark vectors of dynamic video images involves aligning and registering the landmarks of the image frame (FtO) in a neutral state with the landmarks of the image frame in a facial state (S21). Here, the reason for aligning the landmarks of the image frame (FtO) in a neutral state with the landmarks of the image frame in a facial state is to correct the shaken images during the process of a patient with facial paralysis changing from a neutral state to a facial state.

[0075] Next, as shown in FIGS. 1 and 4, the landmark vector of the image frame in the neutral state and the landmark vector of the image frame in the facial state are calculated using the location of the landmark of the image frame in the neutral state and the location of the landmark of the image frame in the facial state (S22).

[0076] Next, landmark displacement is calculated and directionality is analyzed using the landmark vectors of the video frames of the neutral and facial expression states calculated in process 'S22' (S23). At this time, the calculation of landmark vectors and displacement, and the directionality analysis, can be performed in the same way as the landmark vectors, displacement calculation, and directionality analysis used to analyze the movement of landmarks in FIGS. 2 and FIGS. 3. Accordingly, processes 'S22' and 'S23' may be omitted, and the landmark vectors, displacement, and directionality information calculated through the landmark vectors, displacement calculation, and directionality analysis shown in FIGS. 2 and FIGS. 3 may be used as is.

[0077] Next, as shown in FIG. 1, a Movement Symmetry Score (MSS) is calculated to determine left-right symmetry using the calculated landmark movement amount and directionality (S24).

[0078] Landmarks represent key feature points of the face. Their positions change according to changes in facial expressions. Facial symmetry can be evaluated by analyzing the patterns of these changes.

[0079] In the embodiments of the present invention, horizontal and vertical movements are analyzed separately. When analyzing horizontal movements, in the embodiments of the present invention, if the right and left parts of the face move in opposite directions, it is considered symmetric. When facial expression changes occur for the evaluation of facial paralysis, horizontal changes occur symmetrically with respect to the midline of the face. Therefore, when the X-axis movements observed on the right and left sides with respect to the midline of the face are added together, it can be considered symmetrical if the sum approaches '0'. On the other hand, a different approach is required when analyzing vertical movements. If the right and left parts of the face move in the same direction, it is considered symmetric. When the Y-axis movements are identical, the values ​​must be subtracted so that the result approaches '0'. By analyzing movements for each axis in this manner, the following [Equation 5] and [Equation 6] can be derived.

[0080] In [Equation 5] and [Equation 6] below, i is a landmark number, and It has a range of. Here, 468 is the number of extracted face landmarks.

[0081]

[0082] Here, i is the landmark number of the video frame of the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the nth landmark.

[0083]

[0084] Here, Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the differences in left-right vertical orientation of the nth landmark.

[0085] Based on this result, a score representing the degree of symmetry, namely the translation symmetry score (MSS), is calculated. The translation symmetry score (MM) can be expressed as shown in [Equation 7] below.

[0086]

[0087] The displacement symmetry score (MSS) ranges from 0 to 100. A value closer to '100' indicates higher symmetry, which allows for the quantitative evaluation of facial symmetry.

[0088] Experiments were conducted to analyze the correlation between the extracted displacement symmetry score (MSS), the HB method, and FNGS 2.0, and 76 patients with facial paralysis and 13 normal individuals participated.

[0089] Figure 6 is a diagram showing the status of participants evaluated by specialist clinicians using the HB method and FNGS 2.0, and represents the results of analyzing the participants' diagnostic results by facial expression. In this case, data was collected based on the frontal face. The diagnostic results of all participants are as shown in [Table 1] below.

[0090]

[0091]

[0092] The experimental data consists of video data capturing various facial expression changes of the patients. These facial expression changes consist of 'smiling broadly,' 'closing eyes,' and 'raising eyebrows.'

[0093] In this experimental example, to analyze landmark changes, landmarks in a neutral state and landmarks in a facial state detected in a dynamic video were aligned. At this time, the alignment method used was, for example, the alignment method proposed by Kim. Junsik et al. (see Non-Patent Literature 8).

[0094] Figure 7 is a diagram showing major landmark areas analyzed for facial expression evaluation. In this experimental example, a set of landmarks in specific areas is used to analyze numerical values ​​according to each facial expression. For instance, 'eyebrows', 'eyes', 'mouth', etc. were referenced as landmark areas for each facial expression.

[0095] The experimental results according to the present experimental example are described below.

[0096] In this experimental example, video frames in a facial expression state were first searched in the video, and then the change in landmarks between the initial video frame in a neutral state and subsequent video frames in a facial expression state was measured to detect the video frame in which the landmark has the maximum amount of movement.

[0097] Figure 8 is an example graph showing the landmark movement amount of one of the participants.

[0098] In Fig. 8, the red line represents the video frame at the maximum amount of movement. Video frames with values ​​greater than or equal to the green line represent video frames when a facial expression is made. Video frames with values ​​less than or equal to the yellow line represent video frames in a neutral state. The remaining video frames (video frames between the green line and the yellow line) represent video frames in which a change in facial expression is in progress from a neutral state.

[0099] FIG. 9 is a diagram showing an image frame of an expression state detected through a detection method according to an embodiment of the present invention, and FIG. 9 shows an image frame of an expression state detected through video images of some patients. After detecting an image frame of an expression state, landmarks of the corresponding image frame are extracted. Then, alignment is performed to compare the landmarks of a neutral expression state with the landmarks of an expression state. FIG. 10 shows the result of the alignment.

[0100] [Table 2] below displays the internal Root Mean Square Error (RMSE) values ​​for facial landmarks of various expressions, classified by state before and after registration. It can be seen that the registration process successfully achieved better alignment as the RMSE value decreased after registration.

[0101]

[0102] Figure 11 is a box plot showing the symmetry of movement amounts of participants according to three facial expressions.

[0103] Referring to Figure 11, the green line inside the box represents the median of each data group. The top and bottom boundaries of the box represent the 50% range of data around the median. The whiskers outside the box represent the full range of data, and the other points represent data considered outliers. The detailed results for each participant are presented in [Table 3] below.

[0104]

[0105]

[0106] The results above have been normalized from 1 to 100 for intuitive interpretation, where 100 represents perfect symmetry. All values ​​have been rounded to the second decimal place.

[0107] The Pearson correlation between the displacement symmetry score (MSS) (see [Equation 7]) calculated through the facial paralysis grade diagnosis evaluation method proposed in an embodiment of the present invention and the HB method and FNGS 2.0 diagnosis results was analyzed. FIG. 12 is a figure showing the correlation between the displacement symmetry score (MSS) and the HB diagnosis results along with a linear regression line, and FIG. 13 is a figure showing the correlation between the displacement symmetry score (MSS) and the FNGS 2.0 diagnosis results along with a linear regression line.

[0108] Referring to Figures 12 and 13, a lower diagnostic grade after diagnosis with the HB method and FNGS 2.0 indicates normality. Conversely, a higher displacement symmetry score (MSS) indicates normality. Therefore, a negative correlation should be shown. Consequently, the two diagnostic results and the displacement symmetry score (MSS) showed a high correlation. In the relationship between the displacement symmetry score (MSS) of the eyebrow-raising expression, the HB method and FNGS 2.0 produced coefficients of 0.596 and 0.609, respectively.

[0109] As described above, preferred embodiments of the present invention have been explained and illustrated using specific terms; however, such terms are used solely to clarify the invention. Furthermore, it is obvious that various modifications and changes may be made to the embodiments and described terms of the present invention without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. Explanation of the symbols

[0110] 11: Landmark detection unit 12: Facial expression state detection unit 13 : Landmark Register 14 : Landmark Vector Calculation Unit 15: Movement Amount Symmetry Calculation Unit 16: Facial Symmetry Judgment Unit

Claims

Claim 1 A process of calculating the landmark vectors of the image frame in the expression state and the image frame in the expression state, respectively, through a landmark vector calculation unit by utilizing the location of the landmark of the image frame in the expression state and the location of the landmark of the image frame in the expression state detected in the dynamic video image obtained by continuously capturing the frontal view of the face of a patient with facial paralysis and the state of change from a blank state to various expressions through a shooting device; a process in which a movement symmetry calculation unit calculates the direction and movement amount of the landmark of the image frame in the expression state using the landmark vector of the image frame in the expression state and the landmark vector of the image frame in the expression state calculated by the landmark vector calculation unit, and calculates a Movement Symmetry Score (MSS) using the calculated direction and movement amount of the landmark of the image frame in the expression state; and a process in which a face symmetry determination unit determines face symmetry using the movement symmetry score calculated by the movement symmetry calculation unit. A method for evaluating facial left-right symmetry using a landmark vector of a dynamic video image, comprising: a facial expression state detection unit that compares the landmark of the video frame in the expressionless state with the landmark of subsequent video frames and determines the subsequent video frame having the largest landmark movement based on the landmark of the video frame in the expressionless state among the subsequent video frames as the video frame in the expressionless state. Claim 2 In claim 1, the displacement symmetry calculation unit is a method for evaluating facial left-right symmetry by utilizing landmark vectors of a dynamic video image, wherein the displacement symmetry score is calculated using the [Mathematical Formula] below. [Mathematical Formula] Here, i is the landmark number of the video frame of the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is i Represents the sum of the differences in left-right vertical orientation of the nth landmark. Claim 3 A method for evaluating left-right facial symmetry using landmark vectors of a dynamic video image, wherein the facial symmetry determination unit determines that the displacement symmetry score has a value between 0 and 100, and that the closer the value is to 100, the higher the symmetry. Claim 4 delete Claim 5 A method for evaluating facial left-right symmetry using landmark vectors of a dynamic video image, wherein the facial expression state detection unit comprises: a process of calculating a landmark vector of a video frame in a neutral state and a landmark vector of subsequent video frames, respectively, by utilizing the location of a landmark of a video frame in a neutral state and the location of a landmark of subsequent video frames; and a process of comparing the calculated landmark vector of a video frame in a neutral state and the landmark vector of subsequent video frames to detect the subsequent video frame among the subsequent video frames that has the largest total displacement of a landmark based on the landmark vector of a video frame in a neutral state, and determining it as a video frame in a facial expression state. Claim 6 In claim 1, the facial expression state detection unit evaluates facial left-right symmetry by utilizing landmark vectors of dynamic video images, wherein the landmark vector of the image frame in the neutral expression state and the landmark vector of the subsequent image frame are calculated using the following [Mathematical Formula]. [Mathematical Formula] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( Displayed as the value excluding ). Claim 7 In claim 5, the facial expression state detection unit evaluates facial left-right symmetry by utilizing a landmark vector of a dynamic video image, wherein the amount of movement of each landmark of the subsequent video frames is calculated using the [Mathematical Formula] below. [Mathematical Formula] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression state i Indicates the location of the th landmark. Claim 8 In claim 7, the facial expression state detection unit evaluates facial left-right symmetry by utilizing a landmark vector of a dynamic video image, wherein the total displacement of each landmark of the subsequent video frames is calculated using the [Mathematical Formula] below. [Mathematical Formula] Claim 9 In claim 8, the facial expression state detection unit detects the subsequent video frame with the largest total displacement of landmarks among the subsequent video frames based on the landmark vector of the neutral expression state video frame using the following [Mathematical Formula], a method for evaluating facial left-right symmetry using landmark vectors of dynamic video images. [Mathematical Formula] Claim 10 In claim 1, the face symmetry determination unit evaluates left-right symmetry of the face by utilizing a dynamic video image landmark vector totaling 478, which combines 468 face landmarks and 10 eye landmarks, for the landmarks of the image frame in the neutral state and the landmarks of subsequent image frames. Claim 11 delete

Images