Apparatus and method for generating facial movement information using multiple frame data from video

Abstract

The present specification discloses an apparatus and a method for generating face movement information. The apparatus for generating face movement information according to the present specification comprises: a hardware processor; and a memory connected to the processor and configured to store at least one computer program configured to perform a method for generating face movement information. The method for generating face movement information may include: a step of inputting a plurality of frame data included in a video of a subject to be filmed into an artificial intelligence model to obtain a plurality of landmark information for the subject's face in each frame; a step of extracting at least one neutral frame and at least one expression frame using the landmark information corresponding to each frame; and a step of generating information related to face movement using the landmark information corresponding to each other in the at least one neutral frame and the at least one expression frame.

Description

Technology Field

[0001] The present invention relates to a device and method for generating face movement information, and more specifically, to a device and method for generating face movement information using frame data of a video of a subject. Background Technology

[0002] The content described in this section merely provides background information regarding the embodiments described in this specification and does not necessarily constitute prior art.

[0003] Facial paralysis refers to a decrease in facial muscle symmetry caused by facial nerve dysfunction. Symptoms of facial paralysis range from mild weakness to severe paralysis and may include forehead wrinkles, drooping corners of the mouth, dry eyes, excessive tearing, ear pain, difficulty eating, and taste disorders. While approximately two-thirds of patients fully recover within three months, a significant number continue to experience persistent symptoms such as facial asymmetry and involuntary movements (donkinesis). Failure to provide appropriate early treatment carries a risk of complications, including psychological issues such as anxiety and depression. Therefore, accurately diagnosing and determining the stage of progression of facial paralysis is crucial.

[0004] Traditional facial muscle training methods primarily involve patients undergoing mirror feedback training. In mirror feedback training, patients with facial paralysis repeatedly perform facial movements such as raising eyebrows, grimace, closing eyes, showing teeth, and pouting while looking in a mirror. This training process can be monotonous, tedious, and repetitive, which may reduce patient motivation. Furthermore, the severity of facial paralysis varies among patients, and these differences cannot be effectively quantified. Consequently, feedback training makes it difficult to develop personalized training plans and may fail to achieve optimal rehabilitation outcomes. Assessment scales commonly used in clinical practice include the House-Brackmann Facial Nerve Grading System (HBGS), Facial Nerve Grading System 2.0 (FNGS 2.0), and the Nottingham System. However, most of these assessment methods involve the subjective judgment of medical professionals.

[0005] Recently, objective evaluation methods such as attaching optical markers to the face, 3D scanning via optical scanning, and electromyography signal analysis are being researched. However, these methods require additional equipment and a controlled environment, which can cause discomfort to patients. Furthermore, diagnostic equipment is primarily used to monitor the patient's facial movements, and the quantitative assessment of the degree of facial paralysis relies on the subjective judgment of medical professionals. Such subjective assessments have limitations in their universal application to clinical practice.

[0006] Therefore, there is a need for technology capable of generating objective evaluation information regarding the degree of facial paralysis without causing discomfort to the patient. Prior art literature

[0007] Published Patent Application No. 10-2020-0011819, Feb. 4, 2020. The problem to be solved

[0008] The present specification aims to provide an apparatus and method for generating face movement information using frame data of a video.

[0009] This specification is not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0010] A face movement information generating device according to the present specification for solving the above-described problem comprises: a hardware processor; and a memory connected to the processor and configured to store at least one computer program configured to perform a face movement information generating method. The face movement information generating method may include: (a) inputting a plurality of frame data included in a video of a subject to be filmed into an artificial intelligence model and acquiring a plurality of landmark information, which are preset points for the subject's face included in each frame data; (b) using the landmark information corresponding to each frame to extract at least one expressionless frame corresponding to a first criterion and at least one expression frame corresponding to a second criterion among the plurality of frames; and (c) generating information related to face movement using the landmark information corresponding to each other in the at least one expressionless frame and the at least one expression frame.

[0011] According to one embodiment of the present specification, the method for generating face movement information may further include, after step (b), a representative frame extraction step of extracting one representative frame from a plurality of frameless frames when there are a plurality of frameless frames, and extracting one representative frame from a plurality of frameless frames when there are a plurality of frameless frames.

[0012] According to one embodiment of the present specification, step (b) may be a step of calculating the pupil-eyebrow distance, which is the distance between the subject's pupil and the subject's eyebrow in each frame, extracting frames in which the pupil-eyebrow distance is less than or equal to a preset distance value as expressionless frames, and extracting frames in which the pupil-eyebrow distance is greater than or equal to a preset distance value as expression frames.

[0013] At this time, the above step (b) may include calculating the left pupil-eyebrow distance on the left side of the subject's face in each frame and calculating the right pupil-eyebrow distance on the right side of the subject's face.

[0014] At this time, the step may involve calculating the distance between both pupils and eyebrows using the following formula, extracting frames where the distance between both pupils and eyebrows is less than or equal to a preset distance value as expressionless frames, and extracting frames where the distance between both pupils and eyebrows is greater than or equal to a preset distance value as expression frames.

[0015]

[0016] - : Bilateral pupil-eyebrow distance

[0017] - : Left pupil-eyebrow distance

[0018] - : Right pupil-eyebrow distance

[0019] At this time, the representative frame extraction step may be a step of extracting the frame with the shortest pupil-eyebrow distance as the representative neutral frame and extracting the frame with the longest pupil-eyebrow distance as the representative expression frame.

[0020] According to another embodiment of the present specification, step (b) may be a step of extracting frames as expressionless frames in which the area of ​​the eye region formed by a plurality of landmarks corresponding to the position of the subject's eye in each frame is greater than or equal to a preset area value, and extracting frames as expression frames in which the area of ​​the eye region is less than or equal to a preset area value.

[0021] At this time, the representative frame extraction step may be a step of extracting the frame with the largest eye area as the representative neutral frame and extracting the frame with the smallest eye area as the representative expression frame.

[0022] According to another embodiment of the present specification, step (b) may be a step of extracting frames as expressionless frames in which the area of ​​the mouth region formed by a plurality of landmarks corresponding to the position of the subject's mouth in each frame is less than or equal to a preset area value, and extracting frames as expressionless frames in which the area of ​​the mouth region is greater than or equal to a preset area value.

[0023] At this time, the representative frame extraction step may be a step of extracting the frame with the smallest mouth area as the representative expressionless frame and extracting the frame with the largest mouth area as the representative expression frame.

[0024] According to one embodiment of the present specification, the face movement information generation method may further include, after step (a), a first frame landmark transformation step of transforming a frame landmark set, which is a set of landmarks generated in each frame based on a set of reference face landmarks generated in advance.

[0025] At this time, the first frame landmark transformation step may be a step of generating a transformed frame landmark set by performing at least one of scale matching, rotation transformation, and translation based on the set of reference face landmarks.

[0026] At this time, the first frame landmark transformation step may be a step of calculating a plurality of scale factors using coordinate information of landmarks corresponding to each other in the frame landmark set and the reference face landmark set, calculating a plurality of transformation matrices according to each scale factor, and generating the transformed frame landmark set using a first final transformation matrix which is one of the plurality of transformation matrices.

[0027] At this time, the first final transformation matrix can transform the frame landmark set such that the statistical value for the distance between corresponding landmarks in the transformed frame landmark set and the reference face landmark set is minimized.

[0028] According to one embodiment of the present specification, step (c) may be a step of generating movement information in which each non-expression frame landmark generated in the non-expression frame moves to the position of a corresponding expression frame landmark among a plurality of expression frame landmarks generated in the expression frame.

[0029] At this time, the above step (c) may be a step of generating multiple right movement information such that multiple right neutral frame landmarks located to the right of a preset reference axis move to the positions of corresponding right expression frame landmarks, and generating multiple left movement information such that multiple left neutral frame landmarks located to the left of the reference axis move to the positions of corresponding left expression frame landmarks to further generate symmetry information of face movement.

[0030] At this time, the above step (c) may be a step of generating symmetry information of face movement by calculating the sum of displacements in the horizontal direction from the plurality of right movement information and the corresponding plurality of left movement information, and calculating the difference in displacements in the vertical direction.

[0031] At this time, the above step (c) may be a step of generating additional facial symmetry scores calculated by the following formula.

[0032]

[0033] - : Total of displacements in the horizontal direction

[0034] - : The sum of the displacement differences with respect to the vertical direction

[0035] - : Total movement in the horizontal direction

[0036] - : Total movement in the vertical direction

[0037] According to one embodiment of the present specification, the method for generating face movement information may further include, after step (b), a second landmark transformation step of transforming one set based on either one set of non-expression frame landmarks generated from the non-expression frame and the other set of expression frame landmarks generated from the expression frame.

[0038] At this time, the second landmark transformation step may be a step of generating a transformed set of facial frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of facial frame landmarks, or generating a transformed set of facial frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of facial frame landmarks.

[0039] At this time, the second landmark transformation step may be a step of calculating a plurality of scale factors using coordinate information of landmarks corresponding to each other in the set of non-expression frame landmarks and the set of expression frame landmarks, calculating a plurality of transformation matrices according to each scale factor, and generating the transformed expression frame landmark set or the transformed non-expression frame landmark set using a second final transformation matrix which is one of the plurality of transformation matrices.

[0040] At this time, the second final transformation matrix can transform the set of non-expression frame landmarks so that the statistical value for the distance between corresponding landmarks in the transformed set of non-expression frame landmarks and the set of expression frame landmarks is minimized, or transform the set of expression frame landmarks so that the statistical value for the distance between corresponding landmarks in the transformed set of expression frame landmarks and the set of non-expression frame landmarks is minimized.

[0041] Other specific details of the present invention are included in the detailed description and drawings. Effects of the invention

[0042] According to one aspect of the present specification, a facial movement information generating device can generate quantitative information about a patient's facial movements using images of a patient with facial paralysis, without separate additional equipment and a controlled environment.

[0043] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0044] FIG. 1 is a flowchart of a method for generating face movement information according to one embodiment of the present specification. FIG. 2 is a flowchart of a method for generating face movement information according to another embodiment of the present specification. FIG. 3 is a drawing illustrating an example of criteria for extracting a neutral frame and a facial frame according to one embodiment of the present specification. Figure 4 is an example of a graph showing the bilateral pupil-eyebrow distance, the area of ​​the eyeball region, and the area of ​​the mouth region calculated in each frame. Figure 5 illustrates an example of extracting neutral and facial expression frames from a video. FIG. 6 is a flowchart of a method for generating face movement information according to another embodiment of the present specification. Figure 7 illustrates an example of transforming a set of frame landmarks based on a reference face landmark. Figure 8 is a diagram illustrating an example of a set of landmarks generated from a neutral frame and a facial expression frame. Figure 9 is a diagram illustrating an example of the change in corresponding landmarks when changing from a neutral frame landmark set to a facial frame landmark set. Figure 10 is a diagram showing the movement of an arbitrary landmark as a vector when changing from a set of landmarks in a neutral frame to a set of landmarks in a facial frame. FIG. 11 illustrates an example of selecting a plurality of landmarks to generate information related to facial movement according to one embodiment of the present specification. FIG. 12 is a flowchart of a method for generating face movement information according to another embodiment of the present specification. Figure 13 is a diagram illustrating an example of transforming a facial expression frame landmark set based on a neutral expression frame landmark set. Specific details for implementing the invention

[0045] The advantages and features of the invention disclosed herein, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, this specification is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of this specification is complete and to fully inform those skilled in the art (hereinafter referred to as "skilled in the art") of the scope of this specification, and the scope of rights of this specification is defined only by the scope of the claims.

[0046] The terms used herein are for describing the embodiments and are not intended to limit the scope of the claims herein. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0047] Throughout the specification, the same reference numerals refer to the same components, and "and / or" includes each of the mentioned components and all combinations of one or more thereof. Although terms such as "first," "second," etc., are used to describe various components, they are not limited by these terms. These terms are used merely to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the scope of the technical concept of the present invention.

[0048] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by a person skilled in the art to which this specification pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0049] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0050] A face movement information generating device according to one embodiment of the present specification may include a hardware processor and a memory. The memory may be connected to the processor and may store at least one computer program configured to perform a face movement information generating method. The processor executes at least one computer program stored in the memory and may generate information regarding the face movements of a subject from a video of the subject. The subject may be a patient with facial paralysis.

[0051] FIG. 1 is a flowchart of a method for generating face movement information according to one embodiment of the present specification.

[0052] Referring to FIG. 1, in step S10, the processor inputs a plurality of frame data included in a video of a subject to artificial intelligence model, and can obtain a plurality of landmark information, which are preset points for the subject's face included in each frame data.

[0053] As an artificial intelligence model, the Mediapipe Face Mesh model provided by Google ("Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs", arXiv:1907.06724v1) can be used. The Mediapipe Face Mesh model is a model that generates 468 predetermined 3D face landmarks from frame data in which a face is captured, and can generate 3D coordinates for each landmark.

[0054] In this specification, the Mediapipe Face Mesh model was used, but this is merely an example, and various neural network models for extracting landmarks from images of captured faces may be used.

[0055] The above video may include frame data in which the subject displays various facial expressions, including a neutral facial expression. Preferably, the video may include frame data in which the subject starts with a neutral facial expression and displays a specific facial expression. The neutral facial expression and the facial expression with a specific expression will be explained in detail below.

[0056] In step S11, the processor can extract at least one expressionless frame corresponding to a first criterion and at least one expression frame corresponding to a second criterion among the plurality of frames using landmark information corresponding to each frame.

[0057] Multiple landmarks extracted from each frame may be assigned different index numbers. A landmark with a specific index number may correspond to a specific point on the subject's face. For example, a landmark with index number 1 may correspond to the tip of the subject's nose. A landmark with index number 61 may correspond to the left corner of the subject's mouth, and a landmark with index number 291 may correspond to the right corner of the subject's mouth. The above index numbers can be assigned to each landmark when face landmarks are generated in the Mediapipe Face Mesh model. Users can match specific index numbers to different parts of the face.

[0058] The user (medical staff) can set multiple landmark groups to analyze specific facial expressions from landmark information generated from frame data. For example, to analyze a subject's expression of raising their eyebrows, multiple eyebrow landmarks corresponding to the subject's eyebrows and pupil landmarks corresponding to the pupil can be set as a single group. This is just one example, and the user can set various landmark groups depending on the facial expression of the subject to be analyzed.

[0059] The above processor can extract a neutral frame in which the subject is captured in a neutral state and an expression frame in which the subject is captured making a specific expression, according to preset criteria, by using multiple landmark information corresponding to a specific expression.

[0060] In step S12, the processor can generate information related to facial movement using corresponding landmark information in at least one neutral frame and at least one facial frame. The corresponding landmarks may refer to landmarks having the same index number among a plurality of neutral frame landmarks generated in the neutral frame and a plurality of facial frame landmarks generated in the facial frame. The processor can generate facial movement information using the 3D coordinates of the plurality of neutral frame landmarks and the 3D coordinates of the plurality of facial frame landmarks. The processor can generate information related to facial movement using the coordinates of corresponding landmarks in the neutral frame and the facial frame.

[0061] In the above step S11, multiple neutral frames and / or expression frames may be extracted. For example, two neutral frames and two expression frames may be extracted. At this time, the processor may generate information related to facial movement using landmarks generated from the first neutral frame and the first expression frame. Alternatively, the processor may generate information related to facial movement using landmarks generated from the first neutral frame and the second expression frame. Alternatively, the processor may generate information related to facial movement using landmarks generated from the second neutral frame and the first expression frame. Alternatively, the processor may generate information related to facial movement using landmarks generated from the second neutral frame and the second expression frame.

[0062] FIG. 2 is a flowchart of a method for generating face movement information according to another embodiment of the present specification.

[0063] Referring to FIG. 2, steps S20 and S21 are identical to steps S10 to S11, so a repetitive description is omitted.

[0064] When there are multiple expressionless frames and / or expression frames (step S22 Y), in step S23', the processor extracts one representative expressionless frame from the multiple expressionless frames and can extract one representative expression frame from the multiple expression frames.

[0065] For example, when one neutral frame and multiple facial expression frames are extracted, the processor can extract one representative facial expression frame from the multiple facial expression frames.

[0066] As another example, when multiple neutral frames and one facial expression frame are extracted, the processor can extract one representative neutral frame from the multiple neutral frames.

[0067] As another example, when multiple neutral frames and multiple facial expression frames are extracted, the processor can extract one representative neutral frame from the multiple neutral frames and one representative facial expression frame from the multiple facial expression frames.

[0068] In step S24, the processor can generate information related to facial movement using corresponding landmark information from one extracted neutral frame and one extracted facial frame.

[0069] When there is only one neutral frame and one facial frame (step S22 N), in step S24, the processor can generate information related to facial movement using corresponding landmark information from the extracted neutral frame and the extracted facial frame.

[0070] FIG. 3 is a drawing illustrating an example of criteria for extracting a neutral frame and a facial frame according to one embodiment of the present specification.

[0071] Referring to FIG. 3, the processor can extract a neutral frame (hereinafter 'neutral eyebrow frame') and a facial frame (hereinafter 'brow raised facial frame') by using a pupil landmark corresponding to the position of the subject's pupil and a plurality of eyebrow landmarks corresponding to the eyebrows in each frame, such as the eyebrow reference image (1). The processor can extract at least one neutral eyebrow frame and at least one eyebrow raised facial frame from the plurality of frame data in steps S11 and S21.

[0072] The processor can calculate the pupil-eyebrow distance, which is the distance between the subject's pupil and the subject's eyebrow in each frame. The processor can extract frames in which the pupil-eyebrow distance is less than or equal to a preset distance value as expressionless frames, and frames in which the pupil-eyebrow distance is greater than or equal to a preset distance value as expression frames.

[0073] The pupil-eyebrow distance calculated in a frame in which the subject is photographed with their eyebrows raised may be greater than the pupil-eyebrow distance calculated in a frame in which the subject is photographed without their eyebrows raised.

[0074] The processor can extract eyebrow neutral expression frames and eyebrow raised expression frames by calculating the vertical distance between the pupil landmark and the eyebrow landmark with the largest y-axis coordinate among the plurality of eyebrow landmarks in each frame.

[0075] The x-axis and y-axis coordinates at the above pupil landmark Let be, and the x and y coordinates of the eyebrow landmark with the largest y-axis coordinate When so, the above processor uses [Equation 1] to determine the pupil-eyebrow distance can calculate.

[0076]

[0077] More specifically, the plurality of landmarks generated in each frame may include a left pupil landmark (10) corresponding to the subject's left pupil and a right pupil landmark (11) corresponding to the right pupil. Additionally, the plurality of landmarks may include a plurality of left eyebrow landmarks extracted from the subject's left eyebrow area (12) and a plurality of right eyebrow landmarks extracted from the right eyebrow area (13).

[0078] In this specification, the left side refers to the left side of the midline plane (14) that divides the body (face) into left and right sides, and the right side refers to the right side of the midline plane (14).

[0079] At this time, the processor uses [Equation 1] to determine the left pupil-eyebrow distance, which is the distance between the left pupil landmark (10) and the left eyebrow landmark with the largest y-value among the left eyebrow landmarks in each frame. It can calculate the right pupil-eyebrow distance, which is the distance between the right pupil landmark (11) and the right eyebrow landmark with the largest y-value among the right eyebrow landmarks in each frame, using the [Mathematical Formula 1]. can calculate.

[0080] The processor can extract a frame in which at least one of the left pupil-eyebrow distance and the right pupil-eyebrow distance is less than or equal to a preset distance value as a frame with a neutral eyebrow expression. The processor can extract a frame in which at least one of the left pupil-eyebrow distance and the right pupil-eyebrow distance is greater than or equal to a preset distance value as a frame with an eyebrow raised expression.

[0081] In addition, the processor has the bilateral pupil-eyebrow distance calculated by [Equation 2] It can produce.

[0082]

[0083] The processor extracts frames in which the distance between both pupils and eyebrows is less than or equal to a preset distance value as eyebrow neutral frames, and can extract frames in which the distance between both pupils and eyebrows is greater than or equal to a preset distance value as eyebrow raised expression frames.

[0084] Additionally, the processor may extract a frame in which the area of ​​the eye region, formed by a plurality of eye landmarks corresponding to the position of the subject's eye in each frame as an eye region reference image (2), is greater than or equal to a preset area value as a neutral frame (hereinafter 'blank eye frame'), and may extract a frame in which the eye region is less than or equal to a preset area value as an expression frame (hereinafter 'closed eye expression frame'). The processor may extract at least one neutral eye frame and at least one closed eye expression frame from the plurality of frame data in steps S11 and S21.

[0085] More specifically, the processor can calculate the area of ​​a left eye region (20) formed by a plurality of left eye landmarks corresponding to the position of the subject's left eye in each frame. Additionally, the processor can calculate the area of ​​a right eye region (21) formed by a plurality of right eye landmarks corresponding to the position of the subject's right eye in each frame.

[0086] The above processor can calculate the area of ​​the left eye area (20) and the right eye area (21) using the Gaussian area formula known as the Shoelace formula. The Gaussian area formula is a formula used to calculate the area of ​​a polygon based on vertex coordinates. In the Gaussian area formula, the vertex coordinates of a polygon having m (natural number) vertices Given, the area A can be calculated using [Equation 3].

[0087]

[0088] The processor can calculate the area of ​​the left eye region (20) using the x-coordinates and y-coordinates of a plurality of left eye landmarks forming the left eye region (20). The processor can calculate the area of ​​the right eye region (21) using the x-coordinates and y-coordinates of a plurality of right eye landmarks forming the right eye region (21). Subsequently, the processor can calculate the area of ​​both eye regions by adding the area of ​​the left eye region (20) and the area of ​​the right eye region (21).

[0089] The area of ​​the bilateral eye region calculated in the frame in which the subject is captured with their eyes open may be larger than the area of ​​the bilateral eye region calculated in the frame in which the subject is captured with their eyes closed.

[0090] The processor can extract a frame in which the area of ​​both eye regions is greater than or equal to a preset area value as the neutral eye frame. The processor can extract a frame in which the area of ​​both eye regions is less than or equal to a preset area value as the closed eye expression frame.

[0091] Additionally, the processor may extract a frame as a neutral expression frame (hereinafter 'neutral mouth frame') in which the area of ​​the mouth region (30), formed by a plurality of mouth landmarks corresponding to the position of the subject's mouth in each frame as a mouth region reference image (3), is less than or equal to a preset area value, and may extract a frame as an expression frame (hereinafter 'smiling expression frame') in which the eye region is greater than or equal to a preset area value. The processor may extract at least one neutral mouth frame and at least one smiling expression frame from the plurality of frame data in steps S11 and S21.

[0092] The mouth area (30) may refer to an area corresponding to the inside of the mouth when the subject opens their mouth. The area of ​​the mouth area (30) calculated in a frame in which the subject is captured with their mouth wide open and smiling may be larger than the area of ​​the mouth area (30) calculated in a frame in which the subject is captured without smiling.

[0093] The processor can calculate the area of ​​the entrance region (30) using the x-coordinates and y-coordinates of a plurality of entrance landmarks forming the entrance region (30). At this time, the processor can calculate the area of ​​the entrance region (30) using [Equation 3].

[0094] The processor can extract a frame in which the area of ​​the mouth region (30) is greater than or equal to a preset area value as the smiling expression frame. The processor can extract a frame in which the area of ​​the mouth region (30) is less than or equal to a preset area value as the neutral mouth expression frame.

[0095] Figure 4 is an example of a graph showing the bilateral pupil-eyebrow distance, the area of ​​the eyeball region, and the area of ​​the mouth region calculated in each frame.

[0096] Referring to FIG. 4, the bilateral pupil-eyebrow distance graph (4) may represent the bilateral pupil-eyebrow distance calculated in each frame. The bilateral pupil-eyebrow distance values ​​may be normalized. When the bilateral pupil-eyebrow distance calculated in any frame is greater than the value indicated by the yellow dotted line (40), the processor may extract the frame as an eyebrow-raising expression frame. When the bilateral pupil-eyebrow distance calculated in any frame is smaller than the value indicated by the purple dotted line (41), the processor may extract the frame as an eyebrow-neutral expression frame. When the bilateral pupil-eyebrow distance calculated in any frame has a value between the yellow dotted line (40) and the purple dotted line (41), the frame may represent a frame in which a scene is captured in which the subject changes from an eyebrow-raising neutral expression state to an eyebrow-raising expression state, or from an eyebrow-raising expression state to an eyebrow-raising neutral expression state.

[0097] In the bilateral pupil-eyebrow distance graph (4), there may be multiple frames in which the bilateral pupil-eyebrow distance is greater than the value indicated by the yellow dotted line (40). In step S23, the processor may extract the frame with the longest bilateral pupil-eyebrow distance among the multiple eyebrow-raising expression frames as the representative eyebrow-raising expression frame. In the bilateral pupil-eyebrow distance graph (4), the representative eyebrow-raising expression frame may correspond to frame 20, where the red dotted line (42) is located.

[0098] Additionally, in the bilateral pupil-eyebrow distance graph (4), there may be multiple frames in which the bilateral pupil-eyebrow distance is smaller than the value indicated by the purple dotted line (41). In this case, the processor may extract the frame with the smallest bilateral pupil-eyebrow distance among the multiple eyebrow neutral frames as the representative eyebrow neutral frame. In the bilateral pupil-eyebrow distance graph (4), the representative eyebrow neutral frame may correspond to the frame where the green dotted line (43) is located.

[0099] In this specification, the eyebrow-raising expression frame and the eyebrow-neutral expression frame were extracted based on the values ​​indicated by the yellow dotted line (40) and the purple dotted line (41), but this is an example and is not limited to the values.

[0100] The eye area graph (5) may represent the area of ​​both eye areas calculated in each frame. The area values ​​of both eye areas may be normalized. When the area of ​​both eye areas calculated in any frame is smaller than the value indicated by the yellow dotted line (50), the processor may extract the corresponding frame as an eye-closed expression frame. When the area of ​​both eye areas calculated in any frame is larger than the value indicated by the purple dotted line (51), the processor may extract the corresponding frame as an eye-neutral expression frame. When the area of ​​both eye areas calculated in any frame has a value between the yellow dotted line (50) and the purple dotted line (51), the corresponding frame may represent a frame in which a scene is captured of the subject changing from an eye-neutral expression state to an eye-closed expression state, or a scene is captured of the subject changing from an eye-closed expression state to an eye-neutral expression state.

[0101] In the eye area graph (5), there may be multiple frames in which the area of ​​both eye areas is smaller than the value indicated by the yellow dotted line (50). In step S23, the processor may extract the frame with the smallest area of ​​both eye areas among the multiple eye-closing expression frames as the representative eye-closing expression frame. In the eye area graph (5), the representative eye-closing expression frame may correspond to the frame where the green dotted line (52) is located.

[0102] Additionally, in the eye area graph (5), there may be multiple frames in which the area of ​​both eye areas is larger than the value indicated by the purple dotted line (61). In this case, the processor may extract the frame with the largest area of ​​both eye areas among the multiple eye neutral frames as the representative eye neutral frame. In the eye area graph (5), the representative eye neutral frame may correspond to the frame where the red dotted line (53) is located.

[0103] In this specification, the eye-closed expression frame and the eye-blank expression frame were extracted based on the values ​​indicated by the yellow dotted line (50) and the purple dotted line (51), but this is an example and is not limited to the above values.

[0104] The mouth area graph (6) may represent the area of ​​the mouth area (30) calculated for each frame. The area value of the mouth area (30) may be normalized. When the area of ​​the mouth area (30) calculated in any frame is greater than the value indicated by the yellow dotted line (60), the processor may extract the frame as a smiling face frame. When the area of ​​the mouth area (30) calculated in any frame is smaller than the value indicated by the purple dotted line (61), the processor may extract the frame as a neutral mouth frame. When the area of ​​the mouth area (30) calculated in any frame has a value between the yellow dotted line (60) and the purple dotted line (61), the frame may represent a frame in which a scene is captured of the subject changing from a neutral mouth state to a smiling face state, or changing from a smiling face state to a neutral mouth state.

[0105] In the mouth area graph (6), there may be multiple frames in which the area of ​​the mouth area (30) is larger than the value indicated by the yellow dotted line (60). In step S23, the processor may extract the frame with the largest area of ​​the mouth area (30) among the multiple smiling expression frames as the representative smiling expression frame. In the mouth area graph (6), the representative smiling expression frame may correspond to the frame where the red dotted line (62) is located.

[0106] Additionally, in the mouth area graph (6), there may be multiple frames in which the area of ​​the mouth area (30) is smaller than the value indicated by the purple dotted line (61). In this case, the processor may extract the frame with the smallest area of ​​the mouth area (30) among the multiple mouth non-expression frames as the representative mouth non-expression frame. In the mouth area graph (6), the representative mouth non-expression frame may correspond to the frame where the green dotted line (63) is located.

[0107] In this specification, smiling expression frames and neutral mouth frames were extracted based on the values ​​indicated by the yellow dotted line (60) and the purple dotted line (61), but this is an example and is not limited to the values.

[0108] Figure 5 illustrates an example of extracting neutral and facial expression frames from a video.

[0109] Referring to FIG. 5, the processor can extract expression frames and neutral expression frames from a video of a subject using the method described above. The processor can extract a neutral eyebrow frame (7a) and an eyebrow-raising expression frame (7b) from a video of a subject, and can extract a neutral mouth frame (8a) and a smiling expression frame (8b). Additionally, the processor can extract a neutral eye frame and a closed eye expression frame from the video.

[0110] FIG. 6 is a flowchart of a method for generating face movement information according to another embodiment of the present specification.

[0111] Referring to FIG. 6, step S30 is identical to step S10, so a repetitive description is omitted.

[0112] In step S31, the processor can transform the frame landmark set, which is a set of landmarks generated in each frame, based on a pre-generated set of reference face landmarks. When video recording a situation where a subject makes a specific expression while in a neutral state, the size and angle of the subject's face may vary in each frame depending on the subject's movement. For example, if the subject moves their face toward the camera, the subject's face area in that frame may be larger than in other frames. Additionally, if the subject tilts their face to the left, the subject's face area may be rotated to the left in that frame. In such cases, it may not be possible to accurately compare the distance between the subject's pupil landmark and eyebrow landmark, the area of ​​the eyeball region, and the area of ​​the mouth region according to each frame. To prevent this, the processor can transform the frame landmark set based on the set of reference face landmarks.

[0113] Figure 7 illustrates an example of transforming a set of frame landmarks based on a reference face landmark.

[0114] Referring to FIG. 7, the reference face landmark set (70) may refer to a set of landmarks that can be generated from an image in which the face is captured from the front. The reference face landmark set (70) may refer to a set of landmarks that can be generated from an image in which the face is captured without tilting. In this specification, the reference face landmark set (70) is a set of landmarks generated from a standard face model (Canonical Face Model) provided by MediaPipe, which is an example, and various standard face models may be used.

[0115] The processor can generate a transformed frame landmark set by performing at least one of scale matching, rotation transformation, and translation for each frame landmark set based on a reference face landmark set (70).

[0116] As illustrated in FIG. 7, there may be differences in the scale of the reference face landmark set (70) and the frame landmark set (71) for any frame, the angle at which the face shape formed by the landmarks is tilted, etc. The processor can transform the frame landmark set (71) based on the reference face landmark set (70). At this time, the processor adjusts the scale and angle of the face shape of the frame landmark set (71) based on the reference face landmark set (70), but does not accurately match the positions of landmarks having the same index number. As shown in the image (72) in which the reference face landmark set (70) and the frame landmark set (71) are superimposed, the processor can transform the frame landmark set (71) based on the reference face landmark set (70). Through this, the processor can standardize each frame landmark set based on the reference face landmark set. Additionally, the processor can align each frame landmark set into a single common coordinate system.

[0117] Let the first set of landmarks be the source set of landmarks and the second set of landmarks be the target set of landmarks. The array of 2D coordinates of each landmark in the source set of landmarks and the target set of landmarks can be expressed by [Equation 4] and [Equation 5].

[0118]

[0119]

[0120] In [Equation 4] and [Equation 5], n represents the number of landmarks included in the source landmark set and the target landmark set. In this specification, n is 468, which corresponds to one example and may vary depending on the number of face landmarks.

[0121] The above processor can calculate the scale factor using [Equation 6].

[0122]

[0123] -

[0124] In [Equation 6], i represents the index number; since the index numbers generated by the MediaPipe Face Mesh model start from 0, i can have values ​​from 0 to 467. Also, and may mean the origin in the source landmark set and the origin in the target landmark set. In this specification, and Is It may apply to.

[0125] When transforming the frame landmark set (71) based on the reference face landmark set (70), the source landmark set corresponds to the frame landmark set (71), and the target landmark set corresponds to the reference face landmark set (70).

[0126] The above processor can calculate multiple scale factors using coordinate information of corresponding landmarks in the frame landmark set (71) and the reference face landmark set (70). The corresponding landmarks in the frame landmark set (71) and the reference face landmark set (70) may refer to landmarks having the same index number.

[0127] The processor can calculate a scale factor for each landmark in the frame landmark set (71). The processor from Up to n scale factors can be calculated. can mean a scale factor calculated with the coordinates of the landmark corresponding to index number 0 in the reference face landmark set (70) and frame landmark set (71). can mean a scale factor calculated with the coordinates of the landmark corresponding to index number n-1 in the reference face landmark set (70) and frame landmark set (71).

[0128] The above processor can convert the scale of the frame landmark set (71) using the calculated scale factor.

[0129] After transforming the scale of the frame landmark set (71), the processor can perform global registration of the reference face landmark set (70) and the frame landmark set (71). The processor can perform global registration of the two landmark sets using the RANSAC (RANdom SAmple Consensus) algorithm, the FOFH (Fast Point Feature Histograms) algorithm, the SIFT (Scale-Invariant Feature Transform) algorithm, the ICP (Iterative Closest Point) algorithm, etc., which is an example and is not limited by the algorithms.

[0130] The processor can perform global alignment to produce a first transformation matrix that transforms the scaled frame landmark set into the coordinate system of the reference face landmark set (70). Subsequently, the processor can perform Point to Plane Iterative Closest Point (ICP) between the frame landmark set transformed through the first transformation matrix and the reference face landmark set (70) to produce a second transformation matrix that transforms the frame landmark set (71) into the coordinate system of the reference face landmark set (70).

[0131] The processor can produce n quadratic transformation matrices corresponding to n scale factors. The processor The scale of the frame landmark set (71) can be transformed using . Afterwards, the processor A second transformation matrix that uses to transform the transformed frame landmark set into the coordinate system of the reference face landmark set (70). It can calculate. In addition, the above processor The scale of the frame landmark set (71) can be transformed using . Afterwards, the processor A second transformation matrix that uses to transform the transformed frame landmark set into the coordinate system of the reference face landmark set (70). It can produce n quadratic transformation matrices through n iterations. It can produce.

[0132] can mean a quadratic transformation matrix corresponding to the scale factor of the i-th landmark. It transforms the frame landmark set in the homogeneous coordinate system into the coordinate system of the reference face landmark set. It can mean a matrix.

[0133] The processor can generate a transformed frame landmark set using a first final transformation matrix, which is any one of n second transformation matrices. The first final transformation matrix can transform the frame landmark set such that the statistical value for the distance between each landmark and the corresponding reference face landmark in the transformed frame landmark set is minimized.

[0134] The above first final transformation matrix may mean a second transformation matrix that transforms the set of frame landmarks so that the Root Mean Square Error (RMSE) calculated by [Equation 7] is minimized.

[0135]

[0136] In [Equation 7], n represents the number of landmarks, and can represent the Euclidean distance between the landmark with index number i in the source landmark set transformed using a quadratic transformation matrix and the landmark with index number i in the target landmark set.

[0137] The above processor calculates a first final transformation matrix for each frame using the method described above, and can transform each frame landmark set based on the coordinate system of the reference face landmark set.

[0138] In step S32, the processor can extract the neutral frame and the expression frame by calculating the bilateral pupil-eyebrow distance, the area of ​​the bilateral eyeball region, and / or the area of ​​the mouth region from the transformed frame landmark set using the first final transformation matrix.

[0139] In step S33, the processor can generate information related to facial movement using corresponding landmark information in at least one neutral frame and at least one facial frame.

[0140] In the above steps S12, S24, and S33, when the processor generates information related to facial movement using a neutral frame and a facial frame, it can generate movement information in which a plurality of neutral frame landmarks generated in the neutral frame move to the position of a corresponding facial frame landmark among a plurality of facial frame landmarks generated in the facial frame.

[0141] Since the video recording of the subject captures a scene where the subject makes a specific expression while in a neutral state, the subject's change from a neutral state to a specific expression can correspond to multiple neutral frame landmarks moving to the respective positions of expression frame landmarks. Additionally, the subject's change from a neutral state to a specific expression can correspond to a set of neutral frame landmarks changing into a set of expression frame landmarks.

[0142] FIG. 8 is a diagram illustrating an example of a set of landmarks generated from a neutral frame and a facial frame, FIG. 9 is a diagram illustrating an example of a change in corresponding landmarks when changing from a set of landmarks in a neutral frame to a set of landmarks in a facial frame, and FIG. 10 is a diagram showing the movement of an arbitrary landmark as a vector when changing from a set of landmarks in a neutral frame to a set of landmarks in a facial frame.

[0143] Referring to FIGS. 8 to 10, when changing from a set of expressionless frame landmarks (80) generated from an expressionless frame to a set of expressionless frame landmarks (81) generated from an expressionless frame, the processor can generate movement information such that each expressionless frame landmark in the set of expressionless frame landmarks (80) moves to the position of a corresponding expressionless frame landmark in the set of expressionless frame landmarks (81). When changing from a set of expressionless frame landmarks (80) to a set of expressionless frame landmarks (81), the processor can generate movement information such that an expressionless frame landmark with index number i moves to the position of an expressionless frame landmark with index number i.

[0144] In FIG. 9, the processor can represent the movement of each landmark in the form of a cone. The base of the cone corresponds to the location of each non-expression frame landmark, and the vertex of the cone corresponds to the location of each expression frame landmark. That is, the cone can represent the movement of a landmark with index number i to the location of an expression frame landmark with index number i.

[0145] A taller cone height may indicate that the displacement of the corresponding landmark is greater than that of other landmarks. Additionally, the cone may be displayed in red when the landmark displacement is the largest, and in blue when the landmark displacement is the smallest.

[0146] The above processor can generate movement information as a vector in which each neutral frame landmark moves to the position of the corresponding expression frame landmark.

[0147] Additionally, the processor may generate multiple right movement information such that multiple right neutral frame landmarks located to the right of a preset reference axis move to the positions of corresponding right expression frame landmarks, and generate multiple left movement information such that multiple left neutral frame landmarks located to the left of the preset reference axis move to the positions of corresponding left expression frame landmarks, thereby further generating symmetry information of face movement. The preset reference axis may correspond to the midline (14).

[0148] In Fig. 10, It is one of the left expressionless frame landmarks located on the left side of the center plane (14), and It may be any one of the right expressionless frame landmarks located to the right of the center plane (14). At this time, and It can be symmetrical with respect to the midline (14).

[0149] For example, the neutral frame and the expression frame can be a mouth neutral frame and a smiling expression frame. In this case, It may correspond to the landmark corresponding to the left corner of the mouth in the set of blank frame landmarks (80). It may correspond to the landmark corresponding to the left corner of the mouth in the facial expression frame landmark set (81). It may correspond to the landmark corresponding to the right corner of the mouth in the set of neutral frame landmarks (80). It may correspond to the landmark corresponding to the left corner of the mouth in the facial expression frame landmark set (81).

[0150] The coordinates of any blank frame landmark in the blank frame landmark set (80) Let's call it that, and the coordinates of the corresponding facial frame landmark in the facial frame landmark set (81) When so, the vector starting from the above expressionless frame landmark and heading toward the above expression frame landmark can be defined by [Equation 8].

[0151]

[0152] The above processor from A vector pointing toward It produces, from A vector pointing toward It can produce.

[0153] vector may refer to the above leftward movement information, and vector can mean the above right movement information.

[0154] The above processor and Symmetry information regarding the subject's facial movements can be generated using the horizontal and vertical components.

[0155] For example, when a subject makes a smiling face from a neutral state, the left corner of the mouth may move to the upper left, and the right corner of the mouth may move to the upper right. Accordingly, It can be represented as a vector pointing toward the upper left, and It can be represented as a vector pointing toward the upper right.

[0156] When changing from a neutral mouth frame landmark set to a smiling mouth frame landmark set, the landmark corresponding to the left corner of the mouth and the landmark corresponding to the right corner of the mouth can move in opposite directions with respect to the horizontal direction. This is, The horizontal component of class The horizontal component of If the sum is close to 0, it may mean that the subject's face is moving symmetrically.

[0157] When changing from a neutral mouth frame landmark set to a smiling mouth frame landmark set, the landmark corresponding to the left corner of the mouth and the landmark corresponding to the right corner of the mouth can move in the same direction with respect to the vertical direction. This is, The component in the vertical direction of class The component in the vertical direction of If the difference is close to 0, it may mean moving symmetrically.

[0158] The set of expressionless frame landmarks (80) may have a first left expressionless frame landmark to a p (a natural number smaller than n) left expressionless frame landmark on the left side of the median plane (14), and a first right expressionless frame landmark to a p right expressionless frame landmark on the right side.

[0159] In the set of expressionless frame landmarks (80), the qth (natural number from 1 to p) left expressionless frame landmark and the qth right expressionless frame landmark may be symmetrical with respect to the median plane (14).

[0160] The facial expression frame landmark set (81) may have a first left facial expression frame landmark to a p (a natural number smaller than n) left facial expression frame landmark on the left side of the median plane (14), and a first right facial expression frame landmark to a p right facial expression frame landmark on the right side.

[0161] The q-left neutral frame landmark can have the same index number as the q-left expression frame landmark, and the q-right neutral frame landmark can have the same index number as the q-right expression frame landmark.

[0162] When changing from a neutral frame landmark set (80) to a facial frame landmark set (81), the processor can calculate p vectors (first left vector to p-th left vector) calculated from the left side of the midline plane (14) and p vectors (first right vector to p-th right vector) calculated from the right side of the midline plane (14).

[0163] When changing from the neutral frame landmark set (80) to the facial frame landmark set (81), the qth left vector is a vector indicating the movement of the qth left neutral frame landmark to the position of the qth right facial frame landmark, and the qth right vector is a vector indicating the movement of the qth right neutral frame landmark to the position of the qth right facial frame landmark.

[0164] When changing from a neutral frame landmark set (80) to a facial frame landmark set (81), the processor can calculate the sum of displacements in the horizontal direction from a plurality of right movement information (first right vector to the p-th right vector) and corresponding left movement information (first left vector to the p-th left vector). The processor can calculate the sum of the q-th horizontal components by adding the horizontal component of the q-th right vector and the horizontal component of the q-th left vector. Subsequently, the processor can calculate the total sum of displacements in the horizontal direction by adding all the sums of the q-th horizontal components using [Equation 9].

[0165]

[0166] Additionally, when changing from a neutral frame landmark set (80) to a facial frame landmark set (81), the processor can calculate the difference in displacement in the vertical direction from each right movement information (first right vector to the p-th right vector) and the corresponding left movement information (first left vector to the p-th left vector). The processor can calculate the q-th vertical component difference, which is the difference between the vertical component of the q-th left vector and the vertical component of the q-th right vector. Subsequently, the processor can calculate the total sum of the displacement differences in the vertical direction by adding all the q-th vertical component differences using [Equation 10].

[0167]

[0168] When a scene is captured in the video where the subject makes a smiling face while in a blank expression state, the processor above Using the values, symmetry information regarding the horizontal movement of facial muscles when the subject makes a smiling expression can be generated, and Using the values, symmetry information regarding the vertical movement of facial muscles can be generated when the subject makes a smiling expression.

[0169] In addition, the above processor and Using [the above], a translational symmetry score (MSS), which is a score that quantifies the degree of symmetry, can be calculated. The processor can calculate the total amount of horizontal movement in the q-th left vector and the q-th right vector using [Equation 11], and the total amount of vertical movement can be calculated using [Equation 12].

[0170]

[0171]

[0172] Afterwards, the processor can calculate the movement symmetry score using [Equation 13].

[0173]

[0174] The translation symmetry score can have a value from 0 to 100. If the translation symmetry score is close to 100, it may indicate that the symmetry of the subject's facial muscle movements is high. The facial movement information generation method according to the present specification can quantitatively analyze the symmetry of the subject's facial muscle movements by calculating the translation symmetry score.

[0175] In this specification, the change from the neutral frame landmark set (80) to the expression frame landmark set (81) is described as generating symmetry information of facial movement based on landmarks generated in the mouth neutral frame and the smiling expression frame; however, the same method may also be applied when generating symmetry information of facial movement according to the change from the landmark set generated in the eyebrow neutral frame to the landmark set generated in the eyebrow raising expression frame, and the change from the landmark set generated in the eye neutral frame to the landmark set generated in the eye closing expression frame.

[0176] FIG. 11 illustrates an example of selecting a plurality of landmarks to generate information related to facial movement according to one embodiment of the present specification.

[0177] Referring to FIG. 11, the processor can generate symmetry information of facial movement using changes in a plurality of landmarks included in a preset area. In FIG. 11, landmarks may correspond to vertices where each line meets. The preset area may vary depending on a specific facial expression.

[0178] For example, when changing from a set of eyebrow neutral frame landmarks generated from the eyebrow neutral frame to a set of eyebrow upward expression frame landmarks generated from the eyebrow upward expression frame, the processor can generate information related to facial movement and facial symmetry information using movement information of multiple landmarks included in the left eyebrow comparison area (90) and the right eyebrow comparison area (91). The left eyebrow comparison area (90) and the right eyebrow comparison area (91) may be symmetrical with respect to the midline (14).

[0179] As another example, when changing from the set of eye neutral frame landmarks generated from the above eye neutral frame to the set of eye-closed frame landmarks generated from the above eye-closed frame, the processor can generate information related to facial movement and facial symmetry information using movement information of multiple landmarks included in the left eye comparison area (92) and the right eye comparison area (93). The left eye comparison area (92) and the right eye comparison area (93) may be left-right symmetrical with respect to the midline (14).

[0180] As another example, when changing from a set of neutral mouth frame landmarks generated from a neutral mouth frame to a set of smiling face frame landmarks generated from a smiling face frame, the processor can generate information related to facial movement and facial symmetry information using movement information of multiple landmarks included in the left mouth comparison area (94) and the right mouth comparison area (95). The left mouth comparison area (94) and the right mouth comparison area (95) may be symmetrical with respect to the midline (14).

[0181] FIG. 12 is a flowchart of a method for generating face movement information according to another embodiment of the present specification.

[0182] Referring to FIG. 12, step S40 is identical to step S10, so a repetitive description is omitted.

[0183] In step S41, the processor can extract at least one neutral frame and at least one facial frame from a plurality of frames. At this time, the processor can extract a representative neutral frame and a representative facial frame as described above. Additionally, the processor can extract the representative neutral frame and the representative facial frame after transforming each frame landmark set based on a reference face landmark set as described above.

[0184] In step S42, the processor can transform one set of landmarks based on either the set of landmarks generated from the non-expression frame or the set of landmarks generated from the expression frame. The processor can transform the set of landmarks from the expression frame based on the set of landmarks from the non-expression frame, or transform the set of landmarks from the non-expression frame based on the set of landmarks from the expression frame.

[0185] The processor may align the expression frame landmark set based on the coordinate system of the expression frame landmark set, or align the expression frame landmark set based on the expression frame landmark set. The processor may align the expression frame landmark set and the expression frame landmark set in a single common coordinate system.

[0186] According to one embodiment of the present specification, the processor can generate a set of transformed facial frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of non-facial frame landmarks.

[0187] Alternatively, the processor may generate a set of transformed non-expression frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of expression frame landmarks.

[0188] Figure 13 is a diagram illustrating an example of transforming a facial expression frame landmark set based on a neutral expression frame landmark set.

[0189] Referring to FIG. 13, when photographing a subject, the scale of the neutral frame landmark set (100) and the expression frame landmark set (110), and the angle of the face shape formed by the landmarks may differ due to the subject's head movement. As a result, the coordinate systems of the neutral frame landmark set (100) and the expression frame landmark set (110) may be mismatched, making it impossible to generate accurate information related to face movement and / or information regarding face symmetry. To reduce errors caused by the subject's head movement, the processor may align the neutral frame landmark set (100) and the expression frame landmark set (110) in a single common coordinate system.

[0190] The processor can calculate multiple scale factors using coordinate information of landmarks having the same index number in the non-expression frame landmark set (100) and the expression frame landmark set (110). The processor can calculate scale factors using [Equation 6]. The processor can calculate scale factors and convert the scale of the expression frame landmark set (110) to the scale of the non-expression frame landmark set (100).

[0191] At this time, the source landmark set is the facial expression frame landmark set (110), and the target landmark set may be the neutral expression frame landmark set (100).

[0192] After transforming the scale of the facial frame landmark set (110), the processor can globally align the scaled facial frame landmark set (110) with the non-facial frame landmark set (100). Subsequently, the processor can perform point-plane ICP between the aligned facial frame landmark set and the non-facial frame landmark set (100) to produce a second transformation matrix that transforms the facial frame landmark set (110) into the coordinate system of the non-facial frame landmark set (100). The processor can produce multiple second transformation matrices according to multiple scale factors. The processor can transform the facial frame landmark set (110) using a second final transformation matrix, which is one of the multiple second transformation matrices.

[0193] At this time, the second final transformation matrix can transform the expression frame landmark set so that the statistical value for the distance between corresponding landmarks in the expression frame landmark set (100) and the transformed expression frame landmark set is minimized. The processor can transform the expression frame landmark set (110) so that the root mean square deviation according to [Equation 7] is minimized.

[0194] The process of converting the expression frame landmark set (110) based on the expression frame landmark set (100) is the same as the process of converting the frame landmark set (71) based on the reference face landmark set (70), so a detailed explanation is omitted.

[0195] In this specification, it is described that the processor transforms the expression frame landmark set (110) based on the expression frame landmark set (100), but the processor may also transform the expression frame landmark set (100) based on the expression frame landmark set (110). In this case, the source landmark set may be the expression frame landmark set (100), and the target landmark set may be the expression frame landmark set (110). The processor may calculate a second final transformation matrix that transforms the expression frame landmark set (100) based on the expression frame landmark set (110). At this time, the second final transformation matrix may transform the expression frame landmark set (100) such that the root mean square deviation calculated by [Equation 7] is minimized.

[0196] Referring again to FIG. 12, in step S43, the processor may generate information related to facial movement and / or symmetric information of facial movement using the facial expression frame landmark set (110) and the converted facial expression frame landmark set. Alternatively, the processor may generate information related to facial movement and / or symmetric information of facial movement using the facial expression frame landmark set (100) and the converted facial expression frame landmark set. Since the process by which the processor generates information related to facial movement and / or symmetric information of facial movement has been described above, a repetitive description is omitted.

[0197] A method for generating facial movement information may be implemented in the form of a computer program written to perform each step on a computer and recorded on a computer-readable recording medium. The aforementioned computer program may include code encoded in computer languages ​​such as C / C++, C#, JAVA, Python, and machine language, which can be read by the computer's processor (CPU) through the computer's device interface, so that the computer reads the program and executes the methods implemented in the program. Such code may include functional code related to functions that define the necessary functions for executing the methods, and may include control code related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. Furthermore, such code may further include memory reference code regarding where (address) additional information or media necessary for the computer's processor to execute the functions should be referenced in the computer's internal or external memory. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to transmit or receive during communication.

[0198] The above-mentioned storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the above-mentioned storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the above-mentioned program may be stored on various recording media on various servers that the computer can access, or on various recording media on the user's computer. Additionally, the above-mentioned medium may be distributed across networked computer systems, and computer-readable code may be stored in a distributed manner.

[0199] Although embodiments of this specification have been described above with reference to the attached drawings, those skilled in the art to which this specification pertains will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0200] 10: Left pupil landmark 11: Right pupil landmark 12: Left eyebrow area 13: Right eyebrow area 14: Midline 20: Left eye area 21: Right eye area

Claims

Claim 1 A face movement information generating device comprising: a hardware processor; and a memory connected to the processor and configured to store at least one computer program configured to perform a face movement information generating method, wherein the face movement information generating method comprises: (a) inputting a plurality of frame data included in a video of a subject to be filmed into an artificial intelligence model and acquiring a plurality of landmark information, which are preset points for the subject's face included in each frame data; (b) using the landmark information corresponding to each frame to extract at least one expressionless frame corresponding to a first criterion and at least one expression frame corresponding to a second criterion among the plurality of frames. and (c) a step of generating information related to facial movement using corresponding landmark information in at least one non-expression frame and at least one expression frame; wherein step (c) is a step of generating movement information such that each non-expression frame landmark generated in the non-expression frame moves to the position of a corresponding expression frame landmark among a plurality of expression frame landmarks generated in the expression frame, generating a plurality of right movement information such that a plurality of right non-expression frame landmarks located to the right of a preset reference axis move to the position of a corresponding right expression frame landmark, and generating a plurality of left movement information such that a plurality of left non-expression frame landmarks located to the left of the reference axis move to the position of a corresponding left expression frame landmark to generate symmetry information of facial movement. Claim 2 A face movement information generating device according to claim 1, wherein the face movement information generating method further comprises, after step (b), a representative frame extraction step of extracting one representative non-expression frame from a plurality of non-expression frames when there are a plurality of non-expression frames, and extracting one representative expression frame from a plurality of expression frames when there are a plurality of expression frames. Claim 3 A face movement information generating device according to claim 2, wherein step (b) is a step of calculating the pupil-eyebrow distance, which is the distance between the subject's pupil and the subject's eyebrow in each frame, extracting frames in which the pupil-eyebrow distance is less than or equal to a preset distance value as expressionless frames, and extracting frames in which the pupil-eyebrow distance is greater than or equal to a preset distance value as expression frames. Claim 4 A face movement information generating device according to claim 3, wherein step (b) comprises calculating the left pupil-eyebrow distance on the left side of the subject's face in each frame and calculating the right pupil-eyebrow distance on the right side of the subject's face. Claim 5 A face movement information generating device according to claim 4, wherein step (b) is a step of calculating the distance between both pupils and eyebrows using the following formula, extracting frames in which the distance between both pupils and eyebrows is less than or equal to a preset distance value as expressionless frames, and extracting frames in which the distance between both pupils and eyebrows is greater than or equal to a preset distance value as expression frames. - : Bilateral pupil-eyebrow distance Left pupil-eyebrow distance- : Right pupil-eyebrow distance Claim 6 A face movement information generating device according to claim 5, wherein the representative frame extraction step is a step of extracting the frame with the shortest pupil-eyebrow distance as a representative neutral frame and extracting the frame with the longest pupil-eyebrow distance as a representative expression frame. Claim 7 A face movement information generating device according to claim 2, wherein step (b) is a step of extracting frames as expressionless frames in which the area of ​​the eye region formed by a plurality of landmarks corresponding to the position of the subject's eye in each frame is greater than or equal to a preset area value, and extracting frames as expression frames in which the area of ​​the eye region is less than or equal to a preset area value. Claim 8 A face movement information generating device according to claim 7, wherein the representative frame extraction step is a step of extracting the frame with the largest eye area as a representative expressionless frame and extracting the frame with the smallest eye area as a representative expression frame. Claim 9 A face movement information generating device according to claim 2, wherein step (b) is a step of extracting frames as expressionless frames in which the area of ​​the mouth region formed by a plurality of landmarks corresponding to the position of the subject's mouth in each frame is less than or equal to a preset area value, and extracting frames as expression frames in which the area of ​​the mouth region is greater than or equal to a preset area value. Claim 10 A face movement information generating device according to claim 9, wherein the representative frame extraction step is a step of extracting the frame with the smallest mouth area as a representative expressionless frame and extracting the frame with the largest mouth area as a representative expression frame. Claim 11 A face movement information generating device according to claim 1, wherein the face movement information generating method further comprises, after step (a), a first frame landmark conversion step of converting a frame landmark set, which is a set of landmarks generated in each frame based on a set of reference face landmarks generated in advance. Claim 12 A face movement information generating device according to claim 11, wherein the first frame landmark transformation step is a step of generating a transformed frame landmark set by performing at least one of scale matching, rotation transformation, and translation based on the set of reference face landmarks. Claim 13 A face movement information generating device according to claim 12, wherein the first frame landmark transformation step is a step of calculating a plurality of scale factors using coordinate information of landmarks corresponding to each other in the frame landmark set and the reference face landmark set, calculating a plurality of transformation matrices according to each scale factor, and generating the transformed frame landmark set using a first final transformation matrix which is one of the plurality of transformation matrices. Claim 14 A face movement information generating device according to claim 13, wherein the first final transformation matrix transforms the frame landmark set such that the statistical value for the distance between corresponding landmarks in the transformed frame landmark set and the reference face landmark set is minimized. Claim 15 delete Claim 16 delete Claim 17 A face movement information generating device according to claim 1, wherein step (c) is a step of generating symmetry information of face movement by calculating the sum of displacements in the horizontal direction from the plurality of right movement information and the corresponding plurality of left movement information, and calculating the difference in displacements in the vertical direction. Claim 18 A face movement information generating device according to claim 17, wherein step (c) is a step of further generating a face symmetry score calculated by the following formula. - : Total sum of displacements in the horizontal direction : Sum of displacement differences with respect to the vertical direction - : Total displacement in the horizontal direction- : Total movement in the vertical direction Claim 19 A face movement information generating device according to claim 1, wherein the face movement information generating method further comprises, after step (b), a second landmark transformation step of transforming one set based on either one set of non-expression frame landmarks generated from the non-expression frame and the other set of expression frame landmarks generated from the expression frame. Claim 20 A face movement information generating device according to claim 19, wherein the second landmark transformation step is a step of generating a set of transformed facial expression frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of non-expression frame landmarks, or generating a set of transformed non-expression frame landmarks by performing at least one of scale matching, rotation transformation, and translation based on the set of facial expression frame landmarks. Claim 21 A face movement information generating device according to claim 20, wherein the second landmark transformation step is a step of calculating a plurality of scale factors using coordinate information of landmarks corresponding to each other in the set of neutral frame landmarks and the set of expression frame landmarks, calculating a plurality of transformation matrices according to each scale factor, and generating the transformed expression frame landmark set or the transformed neutral frame landmark set using a second final transformation matrix which is one of the plurality of transformation matrices. Claim 22 A face movement information generating device according to claim 21, wherein the second final transformation matrix transforms the set of neutral frame landmarks so that the statistical value for the distance between corresponding landmarks in the transformed set of neutral frame landmarks and the set of facial frame landmarks is minimized, or transforms the set of facial frame landmarks so that the statistical value for the distance between corresponding landmarks in the transformed set of facial frame landmarks and the set of neutral frame landmarks is minimized.

Images