Method of detecting image frames of facial expression and expressionless state in dynamic video, diagnostic evaluation method and system for grading facial paralysis using the same

Abstract

The present invention relates to a method and system for diagnosing and evaluating the grade of facial paralysis using a numerical approach, and more specifically, to a method for detecting image frames of expression and expressionless states in a dynamic video obtained by continuously capturing the face of a patient with facial paralysis changing from an expressionless state to various expressions, and diagnosing and evaluating the grade of facial paralysis in a patient with facial paralysis based on 3D facial landmarks extracted from the detected image frames of expressionless states, and a method and system for diagnosing and evaluating the grade of facial paralysis using the same.

Description

Technology Field

[0001] The present invention relates to a method and system for diagnosing and evaluating the grade of facial paralysis using a numerical approach, and more specifically, to a method for detecting image frames of expression and expressionless states in a dynamic video obtained by continuously capturing the face of a patient with facial paralysis changing from an expressionless state to various expressions, and diagnosing and evaluating the grade of facial paralysis in a patient with facial paralysis based on 3D facial landmarks extracted from the detected image frames of expressionless states, and a method and system for diagnosing and evaluating the grade of facial paralysis using the same. Background Technology

[0002] Facial paralysis refers to a condition in which facial muscle symmetry is impaired due to damage to the facial nerve. If not properly treated in the early stages, facial paralysis can lead to chronic sequelae such as psychological anxiety and depression. For this reason, it is crucial to accurately diagnose facial paralysis in the early stages, assess its progression, and provide timely rehabilitation treatment.

[0003] Methods used to evaluate facial paralysis include the House-Brackmann (HB) method and the Facial Nerve Grading System (FNGS) 2.0. In clinical practice, various visual evaluation methods are typically utilized, but most are subjective. Known objective methods for evaluating facial paralysis include tracking the face with optical markers, using optical scans (3D scans), and analyzing electromyography signals. However, these methods require additional equipment and a limited environment, which can cause discomfort to patients with facial paralysis.

[0004] Recently, research has been conducted on methods to evaluate facial paralysis in an unrestrained environment. One method for evaluating facial paralysis in an unrestrained environment is the analysis of RGB images. Most RGB image analysis methods conduct evaluations through AI-based learning and utilize 68 landmarks in static images.

[0005] The following is a brief introduction to prior studies evaluating facial paralysis.

[0006] 3D facial landmark detection methods are utilized for the evaluation of facial paralysis. 3D facial landmarks are a method for detecting the locations of key facial areas within an image. Previous research proposed a method that extracts a 2D heatmap of facial landmarks from a 2D image and then expands it into 3D information (see Non-Patent Literature 1). However, this approach requires a significant amount of computation. Studies that directly estimate 3D facial landmarks without going through a 2D heatmap (see Non-Patent Literatures 2 and 3) are also known. Methods that estimate 3D facial landmarks by referencing scanning data demonstrate faster speeds than methods that extract 2D heatmaps, but they are limited by the requirement of a scanning device. Another study directly estimates 3D facial landmarks from images. However, most methods focus on a limited number of 38 landmarks. More landmark information is required to accurately measure the progression of facial paralysis. Kartynnik et al. (see Non-patent Literature 4) proposed a 3D face landmark detector for 3D face mesh representation. It uses Blazeface (see Non-patent Literature 5) to detect faces and subsequently extract 3D landmarks. This method operates in real-time and detects 468 face landmarks and an additional 10 iris landmarks. MediaPipe provides this data as a modularized library.

[0007] In addition, numerical approaches can be utilized to evaluate facial paralysis. Research on numerical approaches to facial paralysis is primarily being conducted on still images. Guarin et al. (see Non-patent Literature 6) extracted and analyzed 68 facial landmarks from 2D images. Based on these extracted landmarks, a series of indicators for evaluating facial symmetry and function were estimated. Recently, a numerical approach to measuring facial paralysis using a new registration method utilizing 3D facial landmarks (see Non-patent Literatures 7 and 8) has been introduced. 478 3D facial landmarks were extracted from 2D RGB images captured at different times. To compare the same patient with facial paralysis at different times, iterative Closest Point (ICP) registration was performed, and facial distance and angle symmetry were calculated. This method numerically evaluates the degree of facial paralysis and compares it with data from different time points. Prior art literature

[0008] Patent Document 1 : KR 10-2003-0041131 A, May 23, 2003. Patent Document 2 : KR 10-2003-0029638 A, April 14, 2003. Patent Document 3 : KR 10-2023-0130926 A, September 12, 2023. Patent Document 4 : KR 10-2023-0076741 A, May 31, 2023.

[0009] 비특허문헌 1 : Tu, Xiaoguang; Zhao, Jian; Xie, Mei; Jiang, Zihang; Balamurugan, Akshaya; Luo, Yao; Zhao, Yang; He, Lingxiao; Ma, Zheng; Feng, Jiashi. "3D face recon struction from a single image assisted by 2D face im ages in the wild." IEEE Transactions on Multimedia, vol. 23, pp. 1160-1172, 202비특허문헌 2 : Bulat, A.; Tzimiropoulos, G. “Hierarchical binary CNNs for landmark localization with limited resources.” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, pp. 343?356, 2018비특허문헌3 : Colaco, SJ; Han, DS “Deep Learning-Based Facial Landmarks Localization Using Compound Scaling.” IEEE Access, vol. 10, 2022, pp. 7653?7663.비특허문헌 4 : Kartynnick, Y.; Ablavatsky, A.; Grishchenko, I.; Grund-mann, M. “Real-time facial surface geometry from monocular video on mobile GPUs.” arXiv preprint arXiv:1907.06724, 2019.비특허문헌 5 : Bazarevsky, Valentin; Kartynnik, Yuri; Vakunov, Andrey; Raveendran, Karthik; Grundmann, Matthias. “Blazeface: Sub-millisecond neural face detection on mobile gpus." arXiv preprint arXiv:1907.05047, 2019. Non-patent document 6: Guarin, DL, Dusseldorp, J., Hadlock, TA, & Jowett, N. "A machine learning approach for automated facial measurements in facial palsy." JAMA Facial Plastic Surgery, vol. 20, no. 4, 2018, pp. 335-337. Non-patent document. 7: Woo Hyun et al., “Artificial intelligence-based numerical analysis of facial reanimation: A comparative retrospective cohort study between one stage dual innervation and single innervation,” Journal of Cranio-Maxillofacial Surgery, vol 5, 265-271, Jul. DOI:10.1016 / j.jcms.2023.05.012Non-patent document 8: Kim. Junsik et al., "Numerical approach to facial palsy using a novel registration method with 3D facial landmark," Sensors, vol. 22, no. 17, pp. 6636, Sep. 2022.,DOI:10.3390 / s22176636. The problem to be solved

[0010] To solve the various problems of the prior art, the present invention proposes a numerical approach utilizing a dynamic video-based measurement method as a method for diagnosing and evaluating the grade of facial paralysis.

[0011] The first problem (objective) to be solved by the present invention is to provide a method for detecting image frames of expression and expressionless states in dynamic video, which can stably detect image frames of expressionless states in dynamic video obtained by photographing the face of a patient with facial paralysis changing from an expressionless state to various expressions, in order to evaluate the grade diagnosis of facial paralysis using a numerical approach utilizing a dynamic video-based measurement method.

[0012] In addition, as a second problem to be solved by the present invention, a method and system for diagnosing and evaluating the grade of facial paralysis in a patient with facial paralysis can stably diagnose and evaluate the grade of facial paralysis based on 3D facial landmarks extracted from video frames of facial expression states detected through a method for detecting video frames of facial expression and non-expression states in dynamic video.

[0013] Furthermore, the problems to be solved by the present invention are not limited to the problems described above, and various additional problems to be solved may be provided through the technologies described in the embodiments below. means of solving the problem

[0014] The present invention, according to an embodiment for achieving the above-mentioned purpose, provides a method for detecting expression and expressionless image frames in a dynamic video, comprising: a process of acquiring dynamic video images by continuously capturing a state of change from an expressionless state to various expressions while capturing the frontal view of a patient's face with facial paralysis; a process of extracting landmarks from image frames of an expressionless state and subsequent image frames, respectively, in the acquired dynamic video images; and a process of comparing the landmarks of the extracted subsequent image frames with the landmarks of the image frames of the expressionless state and determining the subsequent image frame having the largest landmark movement based on the landmarks of the image frames of the expressionless state among the subsequent image frames as the image frame of the expressionless state.

[0015] Additionally, the process of determining the video frame of the expression state may include: a process of calculating the landmark vector of the video frame of the expression state and the landmark vector of the subsequent video frames by utilizing the location of the landmark of the video frame of the expression state and the location of the landmark of the subsequent video frames; and a process of comparing the calculated landmark vector of the video frame of the expression state and the landmark vector of the subsequent video frames to detect the subsequent video frame among the subsequent video frames that has the largest total movement amount of the landmark based on the landmark vector of the video frame of the expression state, and determining it as the video frame of the expression state.

[0016] In addition, the landmark vector of the above-mentioned neutral image frame and the landmark vector of the above-mentioned subsequent image frame can be calculated using the [mathematical formula] below.

[0017] [Mathematical Formula]

[0018]

[0019] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( It can be expressed as the value excluding ).

[0020] In addition, the amount of movement of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0021] [Mathematical Formula]

[0022]

[0023] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression statei It can indicate the location of the th landmark.

[0024] In addition, the total movement amount of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0025] [Mathematical Formula]

[0026]

[0027] In addition, among the aforementioned subsequent video frames, the subsequent video frame with the largest total movement of the landmark based on the landmark vector of the aforementioned video frame in a neutral state can be detected using the [mathematical formula] below.

[0028] [Mathematical Formula]

[0029]

[0030] In addition, in the process of extracting each of the above landmarks, a total of 478 landmarks can be extracted by combining 468 face landmarks and 10 eye landmarks.

[0031] In addition, the present invention according to another embodiment for achieving the above-mentioned purpose provides a method for diagnosing and evaluating the grade of facial paralysis, comprising: a process of determining a video frame in an expression state using a method for detecting video frames in an expression state and a non-expression state in the dynamic video; a process of aligning and registering a landmark of a video frame in a non-expression state and a landmark of a video frame in an expression state; a process of calculating a landmark vector of a video frame in a non-expression state and a landmark vector of a video frame in an expression state, respectively, by utilizing the location of a landmark of a video frame in a non-expression state and the location of a landmark of a video frame in an expression state; a process of calculating the direction and amount of movement of a landmark of a video frame in an expression state using the landmark vector of a video frame in a non-expression state and the landmark vector of a video frame in an expression state, and calculating a Movement Symmetry Score (MSS) using the calculated direction and amount of movement of a landmark of a video frame in an expression state; and a process of determining facial symmetry using the calculated Movement Symmetry Score.

[0032] In addition, the process of calculating the above displacement symmetry score can be calculated using the [mathematical formula] below.

[0033] [Mathematical Formula]

[0034]

[0035]

[0036]

[0037] Here, i is the landmark number of the video frame in the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Isi Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is i It can represent the sum of the differences in left-right vertical directionality of the nth landmark.

[0038] In addition, the above displacement symmetry score has a value between 0 and 100, and the closer the value is to 100, the higher the symmetry can be.

[0039] In addition, according to another embodiment for achieving the above-mentioned purpose, the present invention comprises: a landmark detection unit that extracts landmarks from a video frame in a neutral state and subsequent video frames, respectively, in a dynamic video image obtained by continuously capturing the frontal view of a patient's face with facial paralysis and the state of changing from a neutral state to various expressions; a facial expression state detection unit that compares the landmarks of subsequent video frames extracted by the landmark detection unit with the landmarks of the video frame in the neutral state and determines the subsequent video frame having the largest landmark movement based on the landmarks of the video frame in the neutral state among the subsequent video frames as the video frame in the facial expression state; a landmark registration unit that registers the landmarks of the video frame in the facial expression state determined by the facial expression state detection unit and the landmarks of the video frame in the neutral state by aligning them; and a landmark vector calculation unit that calculates the landmark vector of the video frame in the neutral state and the landmark vector of the video frame in the facial expression state, respectively, by utilizing the location of the landmark of the video frame in the neutral state and the location of the landmark of the video frame in the facial expression state registered in the landmark registration unit. A facial paralysis grade diagnosis evaluation system is provided, comprising: a movement symmetry calculation unit that calculates the direction and amount of movement of a landmark in an image frame in an expression state using the landmark vector of an image frame in an expression state calculated by the landmark vector calculation unit and the landmark vector of an image frame in an expression state, and calculates a Movement Symmetry Score (MSS) using the calculated direction and amount of movement of the landmark in an image frame in an expression state; and a facial symmetry judgment unit that determines facial symmetry using the movement symmetry score calculated by the movement symmetry calculation unit.

[0040] In addition, the facial expression state detection unit can calculate the landmark vector of the facial expression state video frame and the landmark vector of the subsequent video frames by utilizing the location of the landmark of the facial expression state video frame and the location of the landmark of the subsequent video frames, respectively, and compare the calculated landmark vector of the facial expression state video frame and the landmark vector of the subsequent video frames to detect the subsequent video frame among the subsequent video frames that has the largest total movement amount of the landmark based on the landmark vector of the facial expression state video frame and determine it as the facial expression state video frame.

[0041] In addition, the landmark vector of the above-mentioned neutral image frame and the landmark vector of the above-mentioned subsequent image frame can be calculated using the [mathematical formula] below.

[0042] [Mathematical Formula]

[0043]

[0044] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( It can be expressed as the value excluding ).

[0045] In addition, the amount of movement of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0046] [Mathematical Formula]

[0047]

[0048] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression state i It can indicate the location of the th landmark.

[0049] In addition, the total movement amount of each landmark in the aforementioned subsequent video frames can be calculated using the [mathematical formula] below.

[0050] [Mathematical Formula]

[0051]

[0052] In addition, among the aforementioned subsequent video frames, the subsequent video frame with the largest total movement of the landmark based on the landmark vector of the aforementioned video frame in a neutral state can be detected using the [mathematical formula] below.

[0053] [Mathematical Formula]

[0054]

[0055] In addition, the above displacement symmetry calculation unit can be calculated using the following [mathematical formula].

[0056] [Mathematical Formula]

[0057]

[0058]

[0059]

[0060] Here, i is the landmark number of the video frame in the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is iIt can represent the sum of the differences in left-right vertical directionality of the nth landmark.

[0061] In addition, the face symmetry judgment unit has a displacement symmetry score between 0 and 100, and the closer the value is to 100, the higher the symmetry. Effects of the invention

[0062] As described above, according to the method for detecting image frames of facial expressions and neutral expressions in dynamic video according to an embodiment of the present invention, by stably detecting image frames of facial expressions in a dynamic video obtained by photographing the face of a patient with facial paralysis changing from a neutral expression to various expressions, the grade diagnosis of facial paralysis can be objectively evaluated using a numerical approach utilizing a dynamic video-based measurement method. Brief explanation of the drawing

[0063] FIG. 1 is a flowchart illustrating a method for detecting an image frame of an expression state according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the process of determining an image frame of an expression state according to an example of the present invention. FIG. 3 is a flowchart illustrating the process of comparing a set reference landmark of an image frame in a neutral state with landmarks of subsequent image frames according to an example of the present invention. FIG. 4 is a diagram showing a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention. FIG. 5 is a conceptual diagram showing the grade diagnosis evaluation process of facial paralysis according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention. Figure 7 is a diagram showing the status of participants evaluated by a specialist clinician using the HB method and FNGS 2.0. Fig. 8 is a diagram showing the major landmark areas analyzed for facial expression evaluation. Figure 9 is an example diagram showing a graph of the landmark movement volume of one of the participants. FIG. 10 is a diagram showing an image frame of a detected facial expression state. FIG. 11 is a diagram showing the result of alignment between landmarks in a neutral state and a facial expression state. Fig. 12 is a box plot showing the symmetry of movement amounts of participants according to three facial expressions. Figure 13 is a diagram showing the correlation between the Movement Symmetry Score (MSS) and the HB diagnosis results along with a linear regression line. Figure 14 is a diagram showing the correlation between the displacement symmetry score (MSS) and the FNGS 2.0 diagnostic results along with a linear regression line. Specific details for implementing the invention

[0064] The advantages and features of the present invention and the methods for achieving them will become clear from the embodiments described below in detail together with the accompanying drawings. Throughout this specification, the same reference numerals refer to the same components. Additionally, where "A and / or B" is written, it may mean both A and B, or either A or B. Furthermore, the size and shape of each component shown in the drawings may be exaggerated for convenience of explanation and are not intended to be limiting.

[0065] Changes in facial expressions of patients with facial paralysis can provide crucial information for analyzing the severity of the condition. Dynamic video footage obtained by capturing a patient's face typically begins with the patient in a neutral state and subsequently displays various expressions. However, not all dynamic video footage begins with a neutral expression and ends with an expression. In fact, in some dynamic video footage, the patient may not be in an expression state even in the final frame (remaining in a neutral state). To address this issue, an embodiment of the present invention proposes a method for detecting expressions in dynamic video footage and utilizes this to propose a method and system for diagnosing and evaluating the grade of facial paralysis.

[0066] Hereinafter, embodiments of the present invention will be described in detail using the attached drawings.

[0067] FIG. 1 is a flowchart illustrating a method for detecting image frames of facial expression states according to an embodiment of the present invention.

[0068] Referring to FIG. 1, a method for detecting video frames of an expression state according to an embodiment of the present invention provides a method for detecting video frames of an expression state and a non-expression state in a dynamic video.

[0069] A method for detecting an image frame of an expression state according to an embodiment of the present invention includes a process (S1) of obtaining a dynamic video image by photographing the face of a patient with facial paralysis using a shooting device (not shown), a process (S2) of extracting landmarks from the obtained dynamic video image, and a process (S3) of determining an image frame of an expression state using the extracted landmarks.

[0070] The process of acquiring dynamic video images (S1) involves capturing the face of a patient with facial paralysis using a capturing device (such as a camera) to acquire dynamic video images. At this time, the capturing device captures the frontal view of the patient's face and continuously captures the state of change from an initial expressionless state to various expressions to acquire dynamic video images.

[0071] The process of extracting landmarks (S2) extracts landmarks including face landmarks and eye landmarks from video frames of dynamic video images. Here, the landmarks may be 3D landmarks. Landmarks are extracted from video frames of an initial neutral expression state and video frames of expression states captured thereafter, respectively. For example, a total of 478 landmarks can be extracted by combining 468 face landmarks and 10 eye landmarks from video frames of dynamic video images, i.e., RGB video images. In an embodiment of the present invention, the process of determining video frames of expression states (S3) can be performed using face landmarks.

[0072] FIG. 2 is a flowchart showing the process of determining an image frame of a facial expression state according to an example of the present invention.

[0073] Referring to FIGS. 1 and 2, the main objective of the process for determining an image frame of an expression state (S3) according to an example of the present invention is to accurately extract the image frame in which the change in expression is greatest. To this end, the process includes a process (S31) of setting a landmark extracted from an image frame (Ft0) of an initial expressionless state of a patient with facial paralysis as a reference landmark, a process (S32) of comparing the set reference landmark of the image frame (Ft0) of the initial expressionless state with the landmarks of image frames (Ft1~FtN (N is an integer)) (hereinafter referred to as 'subsequent image frames') taken after the image frame (Ft0), and a process (S33) of detecting a subsequent image frame that has the largest landmark movement (amount of movement) based on the set reference landmark among the subsequent image frames (Ft1~FtN) as a result of the comparison and determining it as an image frame of an expression state.

[0074] FIG. 3 is a flowchart illustrating the process of comparing a set reference landmark of an image frame in a neutral state with landmarks of subsequent image frames according to an example of the present invention.

[0075] As shown in FIG. 3, the process (S32) of comparing a reference landmark of a non-expression state image frame and a landmark of subsequent image frames according to an example of the present invention includes a process (S321) of calculating a landmark vector using the position of a set reference landmark of a non-expression state image frame (Ft0) and the position of a landmark of subsequent image frames (Ft1~FtN), and a process (S322) of comparing the vector of a set reference landmark of a non-expression state image frame (Ft0) and the vector of a landmark of subsequent image frames (Ft1~FtN).

[0076] As shown in FIG. 2, in the process (S33) of determining the image frame having the largest landmark movement (amount of movement) among subsequent image frames (Ft1~FtN) as the image frame of the expression state, the vector of a set reference landmark of the image frame of the expression state (Ft0) and the vector of a landmark of the subsequent image frames (Ft1~FtN) are compared with each other (S322), and then the subsequent image frame having the largest landmark movement (amount of movement) based on the vector of a set reference landmark of the image frame of the expression state (FtO) is detected and determined as the image frame of the expression state.

[0077] In order to analyze landmark movement (amount of movement), an embodiment of the present invention calculates landmark vectors between consecutive video frames of a video image, for example, a viewpoint S And the terminal t The landmark vector operation in can be expressed as shown in [Equation 1] below.

[0078]

[0079] Here, i Vector of the th landmark is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( It can be expressed as the value excluding ).

[0080] To determine the video frame representing the facial expression state among the subsequent video frames (Ft1~FtN), the video frame exhibiting the greatest change in facial expression is extracted. In other words, the video frame with the largest landmark movement is extracted. To extract the video frame with the largest landmark movement, the displacement of the set reference landmark in the neutral-expression video frame and the displacement of the landmarks extracted from each of the subsequent video frames (Ft1~FtN) are calculated.

[0081] For example, among subsequent video frames (Ft1~FtN), k In the nth subsequent video frame i When calculating the movement amount of the i-th landmark, it can be expressed as [Equation 2] below. Here, k is included within the range of 0 to N, and i is as a landmark number, (Here, 468 is the number of extracted face landmarks) has a range.

[0082]

[0083] Here, Is k In the nth subsequent video frame i Indicates the location of the th landmark, is in a blank-faced video frame (FtO) i Indicates the location of the th landmark.

[0084] The amount of movement of landmarks in subsequent video frames (Ft1~FtN) can be calculated by subtracting the position of the landmark in each subsequent video frame (Ft1~FtN) and the position of the reference landmark set in the video frame (FtO) in a neutral state, as shown in [Equation 2] above.

[0085] In this way, the total displacement of each landmark in each subsequent video frame can be calculated using the displacement for each landmark in each subsequent video frame calculated individually. The total displacement of each landmark in each subsequent video frame ( ) is the sum of the movement amounts of all landmarks in the corresponding subsequent video frame, and can be calculated as shown in [Equation 3] below. For example, if the number of landmarks in each subsequent video frame is M (where M is an integer), the total movement amount of each landmark in the subsequent video frame can be calculated by summing the movement amounts of all M landmarks.

[0086]

[0087] The total movement of landmarks for subsequent video frames (Ft1~FtN) is calculated using [Equation 2] and [Equation 3] above. Based on the total movement of landmarks for each of the subsequent video frames calculated in this way, the video frame with the largest total movement of landmarks among the subsequent video frames is selected. F Find . This can be expressed as a mathematical formula as shown in [Equation 4] below.

[0088]

[0089] Here, F represents the video frame with the maximum total movement amount of the landmark among all subsequent video frames (Ft1~FtN), for example, among the subsequent video frames (Ft1~FtN). k It indicates that the total movement of the landmark in the i-th video frame is maximum. Through this method, the video frame of the facial expression state at the point in time with the greatest change in expression can be reliably extracted.

[0090] FIG. 4 is a schematic diagram showing a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention, and FIG. 5 is a schematic diagram showing the facial paralysis grade diagnosis evaluation process using the facial paralysis grade diagnosis evaluation system shown in FIG. 4, for each component.

[0091] Referring to FIGS. 4 and 5, a facial paralysis grade diagnosis evaluation system according to an embodiment of the present invention includes a landmark detection unit (11), an expression state detection unit (12), a landmark registration unit (13), a landmark vector calculation unit (14), and a displacement symmetry calculation unit (15). Additionally, it may further include a facial symmetry judgment unit (16) that determines the left-right facial symmetry of a patient with facial paralysis using a displacement symmetry score calculated through the displacement symmetry calculation unit (15).

[0092] The landmark detection unit (11) extracts landmarks including face landmarks and eye landmarks from dynamic video images obtained by continuously capturing the face of a patient with facial paralysis. The landmark detection unit (11) extracts landmarks from the image frame (FtO) in a neutral state and subsequent image frames (Ft1~FtN) in an expression state, respectively. For example, a total of 478 3D landmarks, including 468 face landmarks and 10 eye landmarks, can be extracted from each image frame of the dynamic video image, i.e., the RGB video image.

[0093] The facial expression state detection unit (12) sets a landmark detected in a video frame (Ft0) of a neutral expression state as a reference landmark, and compares the reference landmark of the video frame (Ft0) of the neutral expression state set in this way with the landmarks detected in subsequent video frames (Ft1~FtN) respectively. Then, based on the comparison result, it detects the video frame among the subsequent video frames (Ft1~FtN) that has the largest landmark movement, i.e., the amount of landmark movement, based on the set reference landmark. Afterward, the detected subsequent video frame is determined as the video frame of the facial expression state.

[0094] The landmark register (13) registers the landmarks of the image frame (FtO) of the expression state detected by the expression state detection unit (12) and the landmarks of the image frame of the expression state determined by the expression state detection unit (12) by aligning them. Here, the reason for aligning and registering the reference landmarks of the image frame (FtO) of the expression state and the landmarks of the image frame of the expression state determined by the expression state detection unit (12) is to correct the shaken images during the process in which a patient with facial paralysis changes from an expressionless face to an expressionful face.

[0095] The landmark vector calculation unit (14) calculates the landmark vector of the image frame in a neutral state and the landmark vector of the image frame in a positive state by using the location of the landmark of the image frame (Ft0) in a neutral state registered in the landmark matching unit (13) and the location of the landmark of the image frame in a positive state determined through the previous expression state detection unit (12).

[0096] The movement amount symmetry calculation unit (15) calculates the landmark movement amount and analyzes the directionality using the landmark vector extracted from the image frame of the neutral state and the landmark vector extracted from the image frame of the facial state calculated through the landmark vector calculation unit (14), and calculates the movement symmetry score (MSS) using this.

[0097] The facial symmetry judgment unit (16) quantitatively judges and evaluates the symmetry of the facial patient's face using the movement amount symmetry score (MSS) calculated by the movement amount symmetry calculation unit (15).

[0098] FIG. 6 is a flowchart schematically illustrating a method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention.

[0099] Referring to FIGS. 4 and 6, the method for diagnosing and evaluating the grade of facial paralysis according to an embodiment of the present invention includes a process of obtaining a dynamic video image by photographing the face of a patient with facial paralysis (S1), a process of extracting landmarks from the dynamic video image (S2), and a process of detecting and determining an image frame of an expression state using the extracted landmarks (S3).

[0100] Next, the landmarks of the image frame in the neutral state (FtO) and the landmarks of the image frame in the facial state are aligned and registered (S4). Here, the reason for aligning the landmarks of the image frame in the neutral state (FtO) and the landmarks of the image frame in the facial state is to correct the shaken images during the process of a patient with facial paralysis changing from a neutral state to a facial state.

[0101] Next, as shown in FIG. 6, the landmark vector of the image frame in the neutral state and the landmark vector of the image frame in the facial state are calculated respectively using the landmark position of the image frame in the neutral state (Ft0) and the landmark position of the image frame in the facial state (S5).

[0102] Next, the landmark displacement amount is calculated and the directionality is analyzed using the landmark vectors of the video frames of the neutral and facial expression states calculated in process 'S5' (S6). At this time, the calculation of the landmark vector and displacement amount and the directionality analysis can be performed in the same way as the landmark vector, displacement amount calculation, and directionality analysis used to analyze the movement of the landmark in FIGS. 2 and FIGS. 3. Accordingly, processes 'S5' and 'S6' may be omitted, and the landmark vector, displacement amount, and directionality information calculated through the landmark vector, displacement amount calculation, and directionality analysis shown in FIGS. 2 and FIGS. 3 may be used as is.

[0103] Next, as shown in FIG. 6, a Movement Symmetry Score (MSS) is calculated to determine left-right symmetry using the calculated landmark movement amount and directionality (S7).

[0104] Landmarks represent key feature points of the face. Their positions change according to changes in facial expressions. Facial symmetry can be evaluated by analyzing the patterns of these changes.

[0105] In the embodiments of the present invention, horizontal and vertical movements are analyzed separately. When analyzing horizontal movements, the embodiments of the present invention consider it symmetrical if the right and left parts of the face move in opposite directions. During facial expression changes for the evaluation of facial paralysis, horizontal changes occur symmetrically with respect to the midline of the face. Therefore, it can be considered symmetrical if the sum of the X-axis movements observed on the right and left sides relative to the midline of the face approaches '0'. On the other hand, a different approach is required when analyzing vertical movements. If the right and left parts of the face move in the same direction, it is considered symmetrical. When the Y-axis movements are identical, the values ​​must be subtracted so that the result approaches '0'. By analyzing movements for each axis in this manner, the following [Equation 5] and [Equation 6] can be derived.

[0106] In [Equation 5] and [Equation 6] below, i is a landmark number, and It has a range of. Here, 468 is the number of extracted face landmarks.

[0107]

[0108] Here, i is the landmark number of the video frame in the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the nth landmark.

[0109]

[0110] Here, Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the differences in left-right vertical orientation of the nth landmark.

[0111] Based on this result, a score representing the degree of symmetry, namely the translation symmetry score (MSS), is calculated. The translation symmetry score (MM) can be expressed as shown in [Equation 7] below.

[0112]

[0113] The displacement symmetry score (MSS) ranges from 0 to 100. A value closer to '100' indicates higher symmetry, which allows for the quantitative evaluation of facial symmetry.

[0114] Experiments were conducted to analyze the correlation between the extracted displacement symmetry score (MSS), the HB method, and FNGS 2.0, and 76 patients with facial paralysis and 13 normal individuals participated.

[0115] Figure 7 is a diagram showing the status of participants evaluated by specialist clinicians using the HB method and FNGS 2.0, and represents the results of analyzing the participants' diagnostic results by facial expression. In this case, data was collected based on the frontal face. The diagnostic results of all participants are as shown in [Table 1] below.

[0116]

[0117]

[0118] The experimental data consists of video data capturing various facial expression changes of the patients. These facial expression changes consist of 'smiling broadly,' 'closing eyes,' and 'raising eyebrows.'

[0119] In this experimental example, to analyze landmark changes, landmarks in a neutral state and landmarks in a facial state detected in a dynamic video were aligned. At this time, the alignment method used was, for example, the alignment method proposed by Kim. Junsik et al. (see Non-Patent Literature 8).

[0120] Figure 8 is a diagram showing major landmark areas analyzed for facial expression evaluation. In this experimental example, a set of landmarks in specific areas is used to analyze numerical values ​​according to each facial expression. For instance, 'eyebrows', 'eyes', 'mouth', etc. were referenced as landmark areas for each facial expression.

[0121] The experimental results according to the present experimental example are described below.

[0122] In this experimental example, video frames in a facial expression state were first searched in the video, and then the change in landmarks between the initial video frame in a neutral state and subsequent video frames in a facial expression state was measured to detect the video frame in which the landmark has the maximum amount of movement.

[0123] Figure 9 is an example graph showing the landmark movement amount of one of the participants.

[0124] In Fig. 9, the red line represents the video frame at the maximum amount of movement. Video frames with values ​​greater than or equal to the green line represent video frames when a facial expression is made. Video frames with values ​​less than or equal to the yellow line represent video frames in a neutral state. The remaining video frames (video frames between the green line and the yellow line) represent video frames in which a change in facial expression is in progress from a neutral state.

[0125] FIG. 10 is a diagram showing an image frame of an expression state detected through a detection method according to an embodiment of the present invention, and FIG. 10 shows an image frame of an expression state detected through video images of some patients. After detecting an image frame of an expression state, landmarks of the corresponding image frame are extracted. Then, alignment is performed to compare the landmarks of a neutral expression state with the landmarks of an expression state. FIG. 11 shows the result of the alignment.

[0126] [Table 2] below displays the internal Root Mean Square Error (RMSE) values ​​for facial landmarks of various expressions, classified by state before and after registration. It can be seen that the registration process successfully achieved better alignment as the RMSE value decreased after registration.

[0127]

[0128] Figure 12 is a box plot showing the symmetry of movement amounts of participants according to three facial expressions.

[0129] Referring to Figure 12, the green line inside the box represents the median of each data group. The top and bottom boundaries of the box represent the 50% range of data around the median. The whiskers outside the box represent the full range of data, and the other points represent data considered outliers. The detailed results for each participant are presented in [Table 3] below.

[0130]

[0131]

[0132] The results above have been normalized from 1 to 100 for intuitive interpretation, where 100 represents perfect symmetry. All values ​​have been rounded to the second decimal place.

[0133] The Pearson correlation between the displacement symmetry score (MSS) (see [Equation 7]) calculated through the facial paralysis grade diagnosis evaluation method proposed in an embodiment of the present invention and the HB method and FNGS 2.0 diagnosis results was analyzed. FIG. 13 is a figure showing the correlation between the displacement symmetry score (MSS) and the HB diagnosis results along with a linear regression line, and FIG. 14 is a figure showing the correlation between the displacement symmetry score (MSS) and the FNGS 2.0 diagnosis results along with a linear regression line.

[0134] Referring to Figures 13 and 14, a lower diagnostic grade after diagnosis with the HB method and FNGS 2.0 indicates normality. Conversely, a higher displacement symmetry score (MSS) indicates normality. Therefore, a negative correlation should be shown. Consequently, the two diagnostic results and the displacement symmetry score (MSS) showed a high correlation. In the relationship between the displacement symmetry score (MSS) of the eyebrow-raising expression, the HB method and FNGS 2.0 produced coefficients of 0.596 and 0.609, respectively.

[0135] As described above, preferred embodiments of the present invention have been explained and illustrated using specific terms; however, such terms are used solely to clarify the invention. Furthermore, it is obvious that various modifications and changes may be made to the embodiments and described terms of the present invention without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. Explanation of the symbols

[0136] 11: Landmark detection unit 12: Facial expression state detection unit 13 : Landmark Register 14 : Landmark Vector Calculation Unit 15: Movement Amount Symmetry Calculation Unit 16: Facial Symmetry Judgment Unit

Claims

Claim 1 A method for detecting expression and expressionless image frames in a dynamic video, comprising: a process of acquiring dynamic video images by continuously capturing a state of change from an expressionless state to various expressions using a capturing device while capturing the frontal view of a patient's face with facial paralysis; a process of extracting landmarks from image frames of an expressionless state and subsequent image frames, respectively, in the dynamic video images acquired through the capturing device using a landmark detection unit; and a process of using an expression state detection unit to compare the landmarks of subsequent image frames extracted through the landmark detection unit with the landmarks of the expressionless image frames, and determining the subsequent image frame having the largest landmark movement based on the landmarks of the expressionless image frames among the subsequent image frames as the expressionless image frame. Claim 2 A method for detecting video frames of expression and expressionless states in dynamic video, wherein the process of determining the video frame of the expressionless state comprises: a process in which the expression state detection unit calculates the landmark vector of the video frame of the expressionless state and the landmark vector of the subsequent video frames by utilizing the location of the landmark of the video frame of the expressionless state and the location of the landmark of the subsequent video frames; and a process in which the expression state detection unit compares the calculated landmark vector of the video frame of the expressionless state and the landmark vector of the subsequent video frames to detect the subsequent video frame among the subsequent video frames that has the largest total amount of movement of the landmark based on the landmark vector of the video frame of the expressionless state and determines it as the video frame of the expressionless state. Claim 3 In claim 2, the facial expression state detection unit calculates the landmark vector of the image frame in the expressionless state and the landmark vector of the subsequent image frame using the [Mathematical Formula] below, a method for detecting image frames in an expressionless state and image frames in a dynamic video. [Mathematical Formula] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( Displayed as the value excluding ). Claim 4 In claim 2, the facial expression state detection unit calculates the amount of movement of each landmark of the subsequent video frames using the [Mathematical Formula] below, a method for detecting video frames of facial expression and neutral expression states in dynamic video. [Mathematical Formula] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression state i Indicates the location of the th landmark. Claim 5 In claim 4, the facial expression state detection unit calculates the total movement amount of each landmark of the subsequent video frames using the [Mathematical Formula] below, a method for detecting video frames of facial expression and neutral expression states in dynamic video. [Mathematical Formula] Claim 6 A method for detecting image frames of expression and neutral expression states in dynamic video, wherein the expression state detection unit detects the subsequent image frame with the largest total displacement of landmarks among the subsequent image frames based on the landmark vector of the neutral expression state image frame using the [Mathematical Formula] below. [Mathematical Formula] Claim 7 A method for detecting video frames of expression and neutral expression states in a dynamic video, wherein the landmark detection unit extracts a total of 478 landmarks by combining 468 face landmarks and 10 eye landmarks during the process of extracting each of the landmarks. Claim 8 A process for determining a video frame in an expression state using a method for detecting video frames in an expression state and a neutral state in a dynamic video according to any one of claims 1 to 7; a process in which a landmark register registers the landmarks of the video frame in a neutral state and the landmarks of the video frame in an expression state by aligning them; a process in which a landmark vector calculation unit calculates the landmark vector of the video frame in a neutral state and the landmark vector of the video frame in an expression state, respectively, by utilizing the location of the landmark of the video frame in a neutral state and the location of the landmark of the video frame in an expression state registered in the landmark register; a process in which a movement symmetry calculation unit calculates the direction and movement amount of the landmarks of the video frame in an expression state using the landmark vector of the video frame in a neutral state and the landmark vector of the video frame in an expression state calculated by the landmark vector calculation unit, and calculates a Movement Symmetry Score (MSS) using the calculated direction and movement amount of the landmarks of the video frame in an expression state. A method for diagnosing and evaluating the grade of facial paralysis, comprising: a process of determining facial symmetry using a movement amount symmetry score calculated by a movement amount symmetry calculation unit using a facial symmetry judgment unit. Claim 9 In claim 8, the displacement symmetry calculation unit calculates the displacement symmetry score using the following [mathematical formula] in a method for diagnosing and evaluating the grade of facial paralysis. [Mathematical Formula] Here, i is the landmark number of the video frame in the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is i Represents the sum of the differences in left-right vertical orientation of the nth landmark. Claim 10 In claim 8, the displacement symmetry score has a value between 0 and 100, and the closer the value is to 100, the higher the symmetry is, in a method for diagnosing and evaluating the grade of facial paralysis. Claim 11 A landmark detection unit that extracts landmarks from video frames in a neutral state and subsequent video frames, respectively, in dynamic video images obtained by continuously capturing the frontal view of a patient's face and the state of change from a neutral state to various expressions; an expression state detection unit that compares the landmarks of subsequent video frames extracted by the landmark detection unit with the landmarks of the video frames in the neutral state and determines the subsequent video frame having the largest landmark movement based on the landmarks of the video frames in the neutral state among the subsequent video frames as the video frame in the expression state; a landmark registration unit that aligns and registers the landmarks of the video frame in the expression state determined by the expression state detection unit with the landmarks of the video frames in the neutral state; and a landmark vector calculation unit that calculates the landmark vector of the video frame in the neutral state and the landmark vector of the video frame in the expression state, respectively, by utilizing the positions of the landmarks of the video frames in the neutral state and the landmarks of the video frames in the expression state registered in the landmark registration unit. A facial paralysis grade diagnosis evaluation system comprising: a movement symmetry calculation unit that calculates the direction and amount of movement of landmarks in the image frame of the expression state using the landmark vector of the image frame of the expression state calculated by the landmark vector calculation unit and the landmark vector of the image frame of the expression state, and calculates a movement symmetry score (MSS) using the calculated direction and amount of movement of landmarks in the image frame of the expression state; and a facial symmetry judgment unit that determines facial symmetry using the movement symmetry score calculated by the movement symmetry calculation unit. Claim 12 A facial paralysis grade diagnosis evaluation system according to claim 11, wherein the facial expression state detection unit calculates the landmark vector of the image frame in the non-expression state and the landmark vector of the subsequent image frames by utilizing the location of the landmark of the image frame in the non-expression state and the location of the landmark of the subsequent image frames, respectively, and compares the calculated landmark vector of the image frame in the non-expression state and the landmark vector of the subsequent image frames to detect the subsequent image frame among the subsequent image frames that has the largest total amount of movement of the landmark based on the landmark vector of the image frame in the non-expression state and determines it as the image frame in the expression state. Claim 13 In claim 12, a facial paralysis grade diagnosis evaluation system in which the landmark vector of the image frame in the neutral state and the landmark vector of the subsequent image frame are calculated using the [Mathematical Formula] below. [Mathematical Formula] Here, i 1st landmark vector is the terminal t of i nth landmark coordinates ( point in ) s of i nth landmark coordinates ( Displayed as the value excluding ). Claim 14 In claim 12, a facial paralysis grade diagnosis evaluation system that calculates the displacement amount of each landmark in the subsequent video frames using the [Mathematical Formula] below. [Mathematical Formula] Here, Is k of the th subsequent video frame i Indicates the location of the th landmark, is a video frame in a blank expression state i Indicates the location of the th landmark. Claim 15 In claim 14, a facial paralysis grade diagnosis evaluation system that calculates the total displacement of each landmark of the subsequent video frames using the [Mathematical Formula] below. [Mathematical Formula] Claim 16 A facial paralysis grade diagnosis evaluation system according to claim 15, which detects the subsequent image frame among the subsequent image frames that has the largest total displacement of landmarks based on the landmark vector of the image frame in the neutral expression state using the following [mathematical formula]. [Mathematical formula] Claim 17 In claim 11, the displacement symmetry calculation unit is a facial paralysis grade diagnosis evaluation system that calculates using the following [mathematical formula]. [Mathematical formula] Here, i is the landmark number of the video frame in the facial expression state, and It has the range of, Is i Indicates the left horizontal directionality of the nth landmark, and Is i Indicates the right horizontal directionality of the nth landmark, and Is i Indicates the left vertical directionality of the nth landmark, and Is i Indicates the right vertical direction of the nth landmark, and Is i It represents the sum of the values ​​of the left and right horizontal directions of the i-th landmark, and Is i Represents the sum of the differences in left-right vertical orientation of the nth landmark. Claim 18 In claim 11, the facial symmetry judgment unit is a facial paralysis grade diagnosis evaluation system in which the displacement symmetry score has a value between 0 and 100, and the closer the value is to 100, the higher the symmetry.

Images