System and method for acquiring human face motion attributes based on unmarked motion capture

Abstract

The invention provides a system and a method for acquiring human face motion attributes based on unmarked motion capture. The method comprises the following steps: S1, selecting an HRNet human face recognition model; s2, inputting the first face two-dimensional image cluster into an HRNet face recognition model, and performing primary training on the HRNet face recognition model, so that the HRNet face recognition model can automatically recognize feature points of a face; s3, collecting a second face two-dimensional image cluster in an actual application environment, inputting the second face two-dimensional image cluster into the HRNet face recognition model after primary training, and performing secondary training on the HRNet face recognition model after primary training, so that the HRNet face recognition model can automatically recognize feature points of a face in the actual application environment; and S4, measuring the facial action of the testee in a three-dimensional space by using the HRNet face recognition model after the secondary training to obtain the position coordinate and the movement track of each feature point of the face of the testee. The systems and methods do not need to paste or wear colored indicia on the patient's face.

Description

Technical Field

[0001] The present invention relates to the intersection of facial motion measurement technology and medical detection technology, and in particular to a system and method for acquiring facial motion attributes based on markerless motion capture, wherein the system and method can dynamically and quantitatively measure facial motion in three-dimensional space. Background Art

[0002] Medically, it is generally believed that the facial nerve controls the facial expression muscles, and the degree of facial paralysis can be intuitively presented by the three-dimensional movement of the patient's face. The first step in the treatment of facial paralysis is to conduct an accurate and effective evaluation of facial nerve function. Visual evaluation of facial nerve function is one of the important methods in clinical practice, which has the advantages of being simple and non-invasive.

[0003] The traditional method of visually evaluating facial nerve function is that the physician (tester) observes the changes in the patient's (testee's) facial expressions with the naked eye (this evaluation method is usually called subjective visual evaluation). This visual evaluation method is highly dependent on the physician's (tester's) subjective feelings and subjective experience, so it is difficult to accurately reflect the actual degree of lesions in patients with facial paralysis.

[0004] In view of the above-mentioned shortcomings of the subjective visual evaluation method, technicians began to try to use objective visual evaluation methods to evaluate the facial nerve function of patients with facial paralysis as early as 2000. The objective visual evaluation method uses a colored marker attached to the lower edge of the patient's upper eyelid, and uses a measuring device to automatically detect the vertical displacement of the upper edge of the colored marker to obtain the patient's eyelid movement. However, this objective visual evaluation method requires sticking or wearing a colored marker on the patient's face, which reduces the patient's comfort and makes the patient's facial expression changes unnatural. In addition, the test device and test process of this objective visual evaluation method are complex and cumbersome, making it difficult to be easily applied in clinical practice.

[0005] In recent years, artificial intelligence technology with deep learning as its core has developed rapidly. Its main advantage is the processing of large amounts of complex data. Artificial Intelligence (AI) was first proposed at the Dartmouth Conference in 1956 and defined as the use of computers to simulate human logical thinking. The so-called artificial neural network refers to a logical structure constructed by using computers to simulate the structure of human neuron connections. The neural networks currently used in image recognition are various deep network models, that is, nonlinear multi-layer network structures, which can extract image features layer by layer, and the extracted features have stronger discriminative power. Artificial intelligence and artificial neural networks provide feasible engineering and technical support for more accurate and convenient dynamic and quantitative measurement of facial movements in three-dimensional space.

[0006] Based on the above problems, how to provide a measuring device and a measuring method based on markerless motion capture (MMC) based on artificial intelligence and artificial neural networks, which can dynamically and quantitatively measure facial movements in three-dimensional space, has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0007] The purpose of the present invention is to provide a system and method for acquiring facial motion attributes based on markerless motion capture, so as to solve the problem that the existing objective visual evaluation method requires sticking or wearing colored markers on the patient's face.

[0008] To solve the above problems, on the one hand, the present invention first provides a system for acquiring facial motion attributes based on markerless motion capture, the system comprising a test seat, a first camera, a second camera, a third camera, and a computer: the test seat is fixed relative to the ground, the three cameras are fixed in front of the test seat in a surrounding manner, the three cameras are located in the same horizontal plane, and the lenses of the three cameras are all facing the test seat; the second camera is set in front of the test seat, the first camera and the third camera are respectively located on the left and right sides of the second camera, and the first camera and the third camera are symmetrically distributed relative to the line connecting the test seat and the second camera; the three cameras can Videos or images are taken from different angles towards the face of the test subject sitting on the test seat; the three cameras are all electrically connected to the computer, the computer can send an opening command or a closing command to the three cameras at the same time, the three cameras can be synchronously opened in response to the opening command from the computer, the three cameras can be synchronously closed in response to the closing command from the computer, and the three cameras can transmit the image data collected by them to the computer in real time; the three cameras are identical; before measurement, the three cameras need to be time-synchronized based on the PTP protocol so that the time of the three cameras is completely synchronized; before measurement, the internal parameters of the three cameras need to be calibrated using the Zhang Zhengyou chessboard calibration method.

[0009] Preferably, the system also includes a first background light and a second background light; the two background lights are arranged above the three cameras, and the two background lights are located in the same horizontal plane; the light emitted by the two background lights can evenly illuminate the face of the test subject sitting on the test seat from different angles.

[0010] On the other hand, the present invention provides a method for acquiring facial motion attributes based on markerless motion capture, the method adopts the system for acquiring facial motion attributes based on markerless motion capture, and the method comprises the following steps:

[0011] S1: A face recognition model based on HRNet is selected from the existing deep learning network models and is called the HRNet face recognition model;

[0012] S2: input the first face two-dimensional image cluster into the HRNet face recognition model, and perform a first-level training on the HRNet face recognition model, so that the HRNet face recognition model can automatically recognize the feature points of the face;

[0013] S3: collecting a second face two-dimensional image cluster in an actual application environment and inputting it into the first-level trained HRNet face recognition model, and performing second-level training on the first-level trained HRNet face recognition model, so that the HRNet face recognition model can automatically identify the feature points of the face in the actual application environment;

[0014] S4: Using the HRNet face recognition model after secondary training, the facial movements of the test subject are measured in three-dimensional space to obtain the facial motion attributes of the test subject, which include the position coordinates and motion trajectory of each feature point;

[0015] The computer of the system is configured to be able to execute the method for acquiring facial motion attributes based on markerless motion capture.

[0016] Preferably, step S2 includes the following sub-steps:

[0017] S2.1: Select several points on the face as feature points; select 18 points on the face as feature points, including the left eyebrow, left eyebrow peak, left eyebrow tail, left upper eyelid midpoint, left lower eyelid midpoint, left nose wing, left corner of mouth, right eyebrow, right eyebrow peak, right eyebrow tail, right upper eyelid midpoint, right lower eyelid midpoint, right nose wing, right corner of mouth, nose tip, nose center, upper lip midpoint, and lower lip midpoint;

[0018] S2.2: Select at least 200,000 two-dimensional face images from the open source dataset as the first-level training basic image set, input the first-level training basic image set into the HRNet face recognition model, and perform first-level basic training on the HRNet face recognition model. The ratio of the number of two-dimensional face images corresponding to the training set, validation set, and test set used in the first-level basic training is 7:2:1;

[0019] S2.3: Collect at least 50,000 2D facial images as the first-level training and tuning image set, and then manually annotate all the feature points of each 2D facial image in the first-level training and tuning image set. Input the first-level training and tuning image set with manually annotated feature points into the HRNet face recognition model, and then use the SBR algorithm to perform first-level tuning training on the HRNet face recognition model that has undergone first-level basic training. The ratio of the number of 2D facial images corresponding to the training set, validation set, and test set used in the first-level tuning training is 7:2:1.

[0020] S2.4: For each 2D face image in the first-level training and tuning image set, the HRNet face recognition model that has undergone first-level tuning and training automatically identifies a unique feature point corresponding to each manually annotated feature point;

[0021] S2.5: For each two-dimensional face image in the first-level training and tuning image set, a two-dimensional coordinate space is constructed on the two-dimensional face image, and the Euclidean distance between each manually annotated feature point on the two-dimensional face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the two-dimensional coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the two-dimensional coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model; the order of steps S2.1 and S2.2 can be interchanged.

[0022] Preferably, step S3 includes the following sub-steps:

[0023] S3.1: for any test subject, collecting a two-dimensional image group of the test subject's facial movements;

[0024] S3.2: Repeat step S3.1 for different test subjects to obtain a two-dimensional image group of facial movements of each test subject, and the two-dimensional image groups of facial movements of all test subjects constitute a secondary training basic image set, and the secondary training basic image set contains at least 60,000 frames of two-dimensional images;

[0025] S3.3: using a random function to sample 10% of the two-dimensional images from the above-mentioned secondary training basic image set as the secondary training sample image set;

[0026] S3.4: for each frame of the two-dimensional image in the secondary training sample image set, all feature points on the two-dimensional image are marked by manual marking;

[0027] S3.5: For each 2D face image in the secondary training sample image set after manually annotating feature points, the HRNet face recognition model automatically identifies a unique feature point corresponding to each manually annotated feature point;

[0028] S3.6: For each two-dimensional face image in the secondary training sampling image set, a two-dimensional coordinate space is constructed on the two-dimensional face image, and the Euclidean distance between each manually annotated feature point on the two-dimensional face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the two-dimensional coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the two-dimensional coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model.

[0029] Preferably, step S4 includes the following sub-steps:

[0030] S4.1: Set up some basic facial expression units;

[0031] S4.2: using the first camera, the second camera and the third camera of the system to synchronously capture a series of two-dimensional images of the facial movements of the same test subject and transmit them to the computer in real time, obtaining a face measurement image set of the test subject currently being measured and inputting it into the HRNet face recognition model;

[0032] S4.3: Input the face measurement image set of the subject being measured into the HRNet face recognition model, and the HRNet face recognition model automatically recognizes all feature points on each two-dimensional face image in the face measurement image set;

[0033] S4.4: For any feature point on the face of the subject being measured, the HRNet face recognition model automatically obtains the position coordinates and motion trajectory of the feature point in the three-dimensional coordinate space based on the measurement image set of the subject being measured;

[0034] S4.5: Repeat step S4.4 until the HRNet face recognition model obtains the position coordinates and motion trajectories of all feature points on the face of the currently measured subject in the three-dimensional coordinate space, and generates a facial movement measurement report of the currently measured subject.

[0035] Preferably, the step S4.4 further comprises the following sub-steps:

[0036] S4.4.1: For any feature point on the face of the subject being measured, a three-dimensional coordinate space is constructed using the position of the feature point on different two-dimensional images obtained by synchronously photographing two or more cameras of the system, and the position coordinates of the feature point in the three-dimensional coordinate space are obtained based on the binocular imaging principle;

[0037] S4.4.2: For any feature point on the face of the subject currently being measured, the motion trajectory of the feature point in the three-dimensional coordinate space is obtained based on the change of the position coordinates of the feature point in the three-dimensional coordinate space over time.

[0038] Further preferably, the step S4.4.1 includes: for any feature point on the face of the test subject currently being measured, the two-dimensional image obtained by taking pictures with the first camera at any time point is the A1 frame image, and the two-dimensional image obtained by taking pictures with the second camera at this time point is the B1 frame image, and a three-dimensional coordinate space is constructed using the position of the feature point on the A1 frame image and the position of the feature point on the B1 frame image, and the position coordinates of the feature point in the three-dimensional coordinate space are obtained based on the binocular imaging principle.

[0039] Preferably, in the step S2.5, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME as a model evaluation indicator; in the step S2.5, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the corresponding feature point automatically recognized by the HRNet face recognition model is zero.

[0040] Preferably, in the step S3.6, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME as the model evaluation index; in the step S3.6, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the corresponding feature point automatically recognized by the HRNet face recognition model is zero.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The system and method for acquiring facial motion attributes based on markerless motion capture provided in the present application selects the HRNet face recognition model from the existing deep learning network model, performs primary training and secondary training on the HRNet face recognition model, so that the HRNet face recognition model can automatically identify the feature points of the face in an actual application environment, and there is no need to stick or wear colored markers on the face of the test subject.

[0043] (2) The system and method for acquiring facial motion attributes based on markerless motion capture provided in the present application can measure the facial movements of the subject in three-dimensional space and obtain the position coordinates of each feature point on the face of the subject, thereby realizing quantitative measurement of facial movements in three-dimensional coordinate space.

[0044] (3) The system and method for acquiring facial motion attributes based on markerless motion capture provided by the present application can use three cameras to shoot videos of the face of the subject from different angles, and the three cameras can send the collected image data to a computer in real time to obtain the motion trajectory of each feature point on the face of the subject, thereby realizing dynamic measurement of facial movements in a three-dimensional coordinate space. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A schematic diagram of the relative positional relationship between a test seat, a first camera, a second camera, and a third camera of a system for acquiring facial motion attributes based on markerless motion capture provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of selecting several points of a face as feature points in a method for acquiring face motion attributes based on markerless motion capture provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of basic facial expression units of a method for acquiring facial motion attributes based on markerless motion capture provided by an embodiment of the present invention;

[0049] Figure 4 The curve of the movement displacement of the feature points changing with time when the facial expression is to raise the eyebrows;

[0050] Figure 5 This is the curve of the movement displacement of the feature points over time when the facial expression is slightly closed eyes:

[0051] Figure 6 The curve of the movement displacement of the feature points changing with time when the facial expression is closing the eyes;

[0052] Figure 7 The curve of the movement displacement of the feature point changing with time when the facial expression is a raised nose;

[0053] Figure 8 The curve of the movement displacement of the feature points changing with time when the facial expression is a closed-mouth smile;

[0054] Fig. 9 is the curve of the motion displacement of the feature points changing with time when the facial expression is grinning;

[0055] Fig.10 The curve of the movement displacement of the feature points changing with time when the facial expression is open mouth;

[0056] Fig.11 is the curve of the movement displacement of the feature points changing with time when the facial expression is whistling;

[0057] Fig.12 This is the curve showing the change of motion displacement of feature points over time when the facial expression is lifting the upper lip. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] This embodiment provides a system for acquiring facial motion attributes based on markerless motion capture, the system comprising a test seat, a first camera, a second camera, a third camera, a first background light, a second background light, and a computer.

[0060] like Figure 1 As shown, the test seat is fixed relative to the ground, the first camera, the second camera and the third camera are fixed in front of the test seat in a surrounding manner, the first camera, the second camera and the third camera are located in the same horizontal plane, and the lenses of the first camera, the second camera and the third camera are all facing the test seat. In this embodiment, for example, the second camera is set in front of the test seat, the first camera and the third camera are respectively located on the left and right sides of the second camera, and the first camera and the third camera are symmetrically distributed relative to the line connecting the test seat and the second camera. The first camera, the second camera and the third camera can shoot videos or images of the face of the testee (healthy person or patient, the same below) sitting on the test seat from different angles.

[0061] The first background light and the second background light are arranged above the three cameras, and the first background light and the second background light are located in the same horizontal plane. The light emitted by the first background light and the second background light can be evenly irradiated to the face of the testee sitting on the test seat from different angles.

[0062] The first camera, the second camera and the third camera are all electrically connected to the computer. The computer can send an opening instruction or a closing instruction to the first camera, the second camera and the third camera at the same time. The first camera, the second camera and the third camera can be synchronously opened in response to the opening instruction from the computer. The first camera, the second camera and the third camera can be synchronously closed in response to the closing instruction from the computer. The first camera, the second camera and the third camera can transmit the image data collected by them to the computer in real time.

[0063] In a preferred implementation of this embodiment, the first camera, the second camera and the third camera are the same. Before measurement, the first camera, the second camera and the third camera need to be time synchronized based on the PTP protocol (Precision Time Protocol, IEEE1588) so that the time of the three cameras is completely synchronized. Before measurement, the internal parameters of the first camera, the second camera and the third camera need to be calibrated using the Zhang Zhengyou chessboard calibration method.

[0064] This embodiment also provides a method for obtaining face motion attributes based on markerless motion capture, the method comprising the following steps:

[0065] S1: A face recognition model based on HRNet (High-Resolution Network) is selected from the existing deep learning network models and is called the HRNet face recognition model;

[0066] S2: input the first face two-dimensional image cluster into the HRNet face recognition model, and perform a first-level training on the HRNet face recognition model, so that the HRNet face recognition model can automatically recognize the feature points of the face;

[0067] S3: collecting a second face two-dimensional image cluster in an actual application environment and inputting it into the first-level trained HRNet face recognition model, and performing second-level training on the first-level trained HRNet face recognition model, so that the HRNet face recognition model can automatically identify the feature points of the face in the actual application environment;

[0068] S4: Using the HRNet face recognition model after secondary training, the facial movements of the test subject are measured in three-dimensional space to obtain the facial motion attributes of the test subject, which include the position coordinates and motion trajectory of each feature point.

[0069] It should be noted that the computer of the above system is configured to be able to execute the above method for obtaining facial motion attributes based on markerless motion capture.

[0070] The above step S2 includes the following sub-steps:

[0071] S2.1: Select several points on the face as feature points;

[0072] In this embodiment, for example, Figure 2 As shown, select the left eyebrow of the face ( Figure 2 Point I shown), left eyebrow peak ( Figure 2 Point b shown), left eyebrow tail ( Figure 2 m point shown), the midpoint of the left upper eyelid ( Figure 2 Point c shown), the midpoint of the left lower eyelid ( Figure 2 Point d shown), left nose wing ( Figure 2 G-spot shown), left corner of mouth ( Figure 2 h point shown), right eyebrow ( Figure 2 Point L shown), right eyebrow peak ( Figure 2 Point B as shown), right eyebrow tail ( Figure 2 M point shown), the midpoint of the right upper eyelid ( Figure 2 Point C shown), the midpoint of the right lower eyelid ( Figure 2 Point D shown), right nose wing ( Figure 2 G point shown), right corner of mouth ( Figure 2 H point shown), nose tip ( Figure 2 shown|point), nose( Figure 2 K point shown), the middle point of the upper lip ( Figure 2 II point), and the midpoint of the lower lip ( Figure 2 A total of 18 points (point III shown) are used as feature points; those skilled in the art will understand that the selection of facial feature points is not limited to the above-mentioned feature points, and other feature points can be selected as needed, for example, moles and dimples on the face of the test person can be selected as feature points, and the number of feature points can be less than 18 or more than 18.

[0073] S2.2: Select at least 200,000 (a total of 218,595 images are actually selected in this embodiment) two-dimensional face images from open source datasets (the open source datasets used in this embodiment are COFW, WFLW, 300W and / or 300VW open source datasets) as a first-level training basic image set, input the first-level training basic image set into the HRNet face recognition model, and perform first-level basic training on the HRNet face recognition model. The ratio of the number of two-dimensional face images corresponding to the training set, verification set, and test set used in the first-level basic training is 7:2:1;

[0074] S2.3: Collect at least 50,000 two-dimensional facial images as the first-level training and tuning image set, and then manually annotate all the feature points of each two-dimensional facial image in the first-level training and tuning image set (the feature points here refer to the feature points selected in step S2.1), input the first-level training and tuning image set with manually annotated feature points into the HRNet face recognition model, and then use the SBR (Supervision-by-Registration) algorithm to perform first-level tuning training on the HRNet face recognition model that has undergone first-level basic training (here refers to the first-level basic training performed in S2.2). The ratio of the number of two-dimensional facial images corresponding to the training set, validation set, and test set used in the first-level tuning training is 7:2:1;

[0075] S2.4: For each two-dimensional face image in the first-level training and tuning image set, the HRNet face recognition model that has been trained and tuned at the first level automatically identifies a unique feature point corresponding to each manually annotated feature point (i.e., the manually annotated feature points correspond one-to-one to the feature points automatically identified by the HRNet face recognition model);

[0076] S2.5: For each two-dimensional face image in the first-level training and tuning image set, a two-dimensional coordinate space is constructed on the two-dimensional face image, and the Euclidean distance between each manually annotated feature point on the two-dimensional face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the two-dimensional coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the two-dimensional coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model.

[0077] It should be noted that in the above step S2.5, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME (Normalized Mean Error, NME) as the model evaluation indicator.

[0078] It should be noted that in the above step S2.5, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the feature point automatically recognized by the corresponding HRNet face recognition model is zero, that is, the position of the manually marked feature point and the feature point automatically recognized by the corresponding HRNet face recognition model completely coincides.

[0079] It should be noted that the order of the above step S2.1 and the above step S2.2 can be interchanged.

[0080] The above step S3 includes the following sub-steps:

[0081] S3.1: for any test subject (such as a healthy person), collecting a two-dimensional image group of the test subject's facial movements;

[0082] In this embodiment, for example, for any one subject, the first camera, the second camera and the third camera of the system are used to synchronously shoot a series of two-dimensional images of the subject's facial movements (i.e., shoot a video with a shooting frame rate of, for example, 60 frames per second) to obtain a two-dimensional image group of the subject's facial movements; in this step, the first camera, the second camera and the third camera of the system are used to synchronously shoot a series of two-dimensional images of the subject's facial movements, including shooting a series of two-dimensional images of the subject's facial movements under different lighting conditions, and also including shooting a series of two-dimensional images of the subject's facial movements when the subject's facial expressions change differently;

[0083] S3.2: Repeat the above step S3.1 for different test subjects to obtain a two-dimensional image group of facial movements of each test subject, and the two-dimensional image groups of facial movements of all test subjects constitute a secondary training basic image set, and the secondary training basic image set includes at least 60,000 frames of two-dimensional images (for example, a total of 64,011 frames of images in this embodiment);

[0084] S3.3: using a random function to sample 10% of the two-dimensional images from the above-mentioned secondary training basic image set as the secondary training sample image set;

[0085] S3.4: for each frame of the two-dimensional image in the secondary training sample image set, all feature points on the two-dimensional image are marked by manual marking;

[0086] S3.5: For each 2D face image in the secondary training sample image set after manually annotating feature points, the HRNet face recognition model automatically identifies a unique feature point corresponding to each manually annotated feature point (i.e., the manually annotated feature points correspond one-to-one to the feature points automatically identified by the HRNet face recognition model);

[0087] S3.6: For each two-dimensional face image in the secondary training sampling image set, a two-dimensional coordinate space is constructed on the two-dimensional face image, and the Euclidean distance between each manually annotated feature point on the two-dimensional face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the two-dimensional coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the two-dimensional coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model.

[0088] It should be noted that in the above step S3.6, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME (Normalized Mean Error, NME) as the model evaluation indicator.

[0089] It should be noted that in the above step S3.6, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the feature point automatically recognized by the corresponding HRNet face recognition model is zero, that is, the position of the manually marked feature point and the feature point automatically recognized by the corresponding HRNet face recognition model completely coincides.

[0090] The above step S4 includes the following sub-steps:

[0091] S4.1: Set several basic facial expression units based on the autonomy and bilateral symmetry of different parts of the face;

[0092] In this embodiment, Figure 3 As shown, the following ten basic facial expression units are set according to the needs:

[0093] (1) Static state: No facial expression is performed and the facial muscles are in a relaxed state. Figure 3 A);

[0094] (2) Raise eyebrows with force: Raise eyebrows as a whole upward, using maximum force ( Figure 3 B);

[0095] (3) Gently close your eyes: Close your eyes gently and slowly. Figure 3 C);

[0096] (4) Close your eyes forcefully: Close your eyes quickly and forcefully. Figure 3 D)

[0097] (5) Nose lift: wrinkle the upper part of the nose and both nostrils upwards. Figure 3 E);

[0098] (6) Closed-mouth smile: Keep your lips tightly closed and the corners of your mouth naturally rise to the sides. Figure 3 F);

[0099] (7) Grinning: The corners of the mouth are pointed upward and outward to the greatest extent. Figure 3 G);

[0100] (8) Open your mouth wide: open your upper and lower lips as wide as possible ( Figure 3 H);

[0101] (9) Whistling: Open your lips slightly, puff up your cheeks and blow air forward. Figure 3 I);

[0102] (10) Lifting the upper lip: The upper lip is lifted upward, often accompanied by a nose-lifting action. Figure 3 J);

[0103] The basic facial expression unit "static" (i.e., the above-mentioned (1)th basic facial expression unit) is used as a facial reference expression unit for reference of the remaining nine basic facial expression units (i.e., the above-mentioned (2)th to (10)th basic facial expression units); those skilled in the art will appreciate that the basic facial expression units are not limited to the above-mentioned 10 types, and other facial expressions can be selected as basic facial expression units as needed, and the number of basic facial expression units can be more than 10 or less than 10;

[0104] S4.2: Using the first camera, the second camera and the third camera of the above system to synchronously shoot a series of two-dimensional images of the facial movements of the same test subject (i.e., shoot a video, the frame rate of which is, for example, 60 frames per second) and transmit it to the computer in real time, the test subject currently being measured needs to continuously complete the remaining nine basic facial expression units (i.e., the above-mentioned (2) to (10) basic facial expression units) except for "static", obtain the face measurement image set of the test subject currently being measured and input it into the HRNet face recognition model;

[0105] S4.3: Input the face measurement image set of the subject being measured into the HRNet face recognition model, and the HRNet face recognition model automatically recognizes all feature points on each two-dimensional face image in the face measurement image set;

[0106] S4.4: For any feature point on the face of the subject being measured, the HRNet face recognition model automatically obtains the position coordinates and motion trajectory of the feature point in the three-dimensional coordinate space based on the measurement image set of the subject being measured;

[0107] S4.5: Repeat the above step S4.4 until the HRNet face recognition model obtains the position coordinates and motion trajectories of all feature points on the face of the currently measured subject in the three-dimensional coordinate space, and generates a facial movement measurement report of the currently measured subject.

[0108] The above step S4.4 further includes the following sub-steps:

[0109] S4.4.1: For any feature point on the face of the subject being measured, the position of the feature point on different two-dimensional images captured synchronously (i.e., simultaneously) by two or more cameras of the above system is used to construct a three-dimensional coordinate space, and the position coordinates of the feature point in the three-dimensional coordinate space are obtained based on the binocular imaging principle;

[0110] Specifically, for any feature point on the face of the subject being measured, such as the feature point of the nose tip, the two-dimensional image captured by the first camera at any time point (for example, time point t1) is the A1 frame image, and the two-dimensional image captured by the second camera at this time point is the B1 frame image (that is, the A1 frame image and the B1 frame image are respectively captured by the first camera and the second camera at the time point t1). The position of the feature point on the A1 frame image, i.e., the nose tip, and the position of the feature point on the B1 frame image, i.e., the nose tip, are used to construct a three-dimensional coordinate space, and the position of the feature point, i.e., the nose tip, in the three-dimensional coordinate space is obtained based on the binocular imaging principle. Similarly, for the feature point of the nose tip, the two-dimensional image captured by the first camera at the next time point (for example, time point t2, which is the next time point after time point t1) is the A2-th frame image, and the two-dimensional image captured by the second camera at this time point is the B2-th frame image (that is, the A2-th frame image and the B2-th frame image are respectively captured by the first camera and the second camera at the time point t2). Using the position of the feature point, i.e., the nose tip, on the A2-th frame image and the position of the feature point, i.e., the nose tip, on the B2-th frame image, the position coordinates of the feature point, i.e., the nose tip, in the three-dimensional coordinate space are obtained based on the binocular imaging principle;

[0111] S4.4.2: For any feature point on the face of the subject currently being measured, the motion trajectory of the feature point in the three-dimensional coordinate space is obtained based on the change of the position coordinates of the feature point in the three-dimensional coordinate space over time.

[0112] Table 1 Measurement results of the subjects' facial expressions in static conditions

[0113]

[0114] Specifically, the measurement results of the test subject's facial expression in a static state are shown in Table 1, which lists the measurement results of the distances between various feature points on the test subject's face.

[0115] Specifically, the measurement results of the test subjects' facial expressions in dynamic situations are shown in Tables 2 and Figures 4 to 12 . Figure 4 The patient's facial expression is shown as the characteristic point brow peak ( Figure 2 The curve of the movement displacement of point B / b) versus time; Figure 5The patient's facial expression is slightly closed eyes. The characteristic point is the midpoint of the upper eyelid ( Figure 2 The curve of the movement displacement of point C / c) versus time; Figure 6 The figure shows the characteristic point of the upper eyelid midpoint ( Figure 2 The curve of the movement displacement of point C / c) versus time; Figure 7 The figure shows the characteristic point nose wing ( Figure 2 The curve of the movement displacement of the G / g point) versus time; Figure 8 The figure shows the characteristic points of the corner of the mouth ( Figure 2 The curve of the movement displacement of point H / h) versus time; Fig. 9 The patient's facial expression is grinning, and the characteristic points of the mouth corners ( Figure 2 The curve of the movement displacement of point H / h) versus time; Fig.10 The figure shows the feature point of the upper lip midpoint when the patient's facial expression is open mouth ( Figure 2 II point) and the midpoint of the lower lip ( Figure 2 The curve of the movement displacement of point III) over time; Fig.11 The patient's facial expression is shown as the characteristic point of the mouth corner in the case of whistling ( Figure 2 The curve of the movement displacement of point H / h) versus time; Fig.12 The patient's facial expression is the upper lip lift, and the characteristic point is the midpoint of the upper eyelid ( Figure 2 The curve of the movement displacement of point C / c) changing with time.

[0116] Table 2 Measurement results of feature point motion displacement when the facial expressions of the subjects are dynamic

[0117]

[0118] It should be noted that when the test subject is, for example, a patient with facial paralysis, the measurement results of the method and system for obtaining facial movement attributes based on markerless motion capture provided in this embodiment can objectively reflect the facial function status of facial paralysis, so the measurement results can be used as a reference for relevant doctors.

[0119] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

[0120] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0121] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0122] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system for acquiring facial motion attributes based on markerless motion capture, characterized in that: The system includes a test seat, a first camera, a second camera, a third camera, and a computer; The test seat is fixed relative to the ground, and three cameras are fixed in front of the test seat in a surrounding manner. The three cameras are located in the same horizontal plane, and the lenses of the three cameras are all facing the test seat; the second camera is set in front of the test seat, and the first camera and the third camera are respectively located on the left and right sides of the second camera, and the first camera and the third camera are symmetrically distributed relative to the line connecting the test seat and the second camera; the three cameras can shoot videos or images of the face of the testee sitting on the test seat from different angles; The three cameras are all electrically connected to the computer, and the computer can send an opening instruction or a closing instruction to the three cameras at the same time. The three cameras can be synchronously opened in response to the opening instruction from the computer, and the three cameras can be synchronously closed in response to the closing instruction from the computer. The three cameras can transmit the image data collected by them to the computer in real time; The three cameras are identical; before measurement, the three cameras need to be time synchronized based on the PTP protocol so that the time of the three cameras is completely synchronized; before measurement, the internal parameters of the three cameras need to be calibrated using Zhang Zhengyou's chessboard calibration method.

2. The system for acquiring facial motion attributes based on markerless motion capture according to claim 1, characterized in that: The system also includes a first background light and a second background light; the two background lights are arranged above the three cameras and are located in the same horizontal plane; the light emitted by the two background lights can evenly illuminate the face of the testee sitting on the test seat from different angles.

3. A method for acquiring facial motion attributes based on markerless motion capture, the method adopting the system for acquiring facial motion attributes based on markerless motion capture according to claim 1 or 2, characterized in that: The method comprises the following steps: S1: A face recognition model based on HRNet is selected from the existing deep learning network models and is called the HRNet face recognition model; S2: input the first face two-dimensional image cluster into the HRNet face recognition model, and perform a first-level training on the HRNet face recognition model, so that the HRNet face recognition model can automatically recognize the feature points of the face; S3: collecting a second face two-dimensional image cluster in an actual application environment and inputting it into the first-level trained HRNet face recognition model, and performing second-level training on the first-level trained HRNet face recognition model, so that the HRNet face recognition model can automatically identify the feature points of the face in the actual application environment; S4: Using the HRNet face recognition model after secondary training, the facial movements of the test subject are measured in three-dimensional space to obtain the facial motion attributes of the test subject, which include the position coordinates and motion trajectory of each feature point; The computer of the system is configured to be able to execute the method for acquiring facial motion attributes based on markerless motion capture.

4. The method for acquiring facial motion attributes based on markerless motion capture according to claim 3, characterized in that: The step S2 includes the following sub-steps: S2.1: Select several points on the face as feature points; select 18 points on the face as feature points, including the left eyebrow, left eyebrow peak, left eyebrow tail, left upper eyelid midpoint, left lower eyelid midpoint, left nose wing, left corner of mouth, right eyebrow, right eyebrow peak, right eyebrow tail, right upper eyelid midpoint, right lower eyelid midpoint, right nose wing, right corner of mouth, nose tip, nose center, upper lip midpoint, and lower lip midpoint; S2.2: Select at least 200,000 two-dimensional face images from the open source dataset as the first-level training basic image set, input the first-level training basic image set into the HRNet face recognition model, and perform first-level basic training on the HRNet face recognition model. The ratio of the number of two-dimensional face images corresponding to the training set, validation set, and test set used in the first-level basic training is 7:2:1; S2.3: Collect at least 50,000 2D facial images as the first-level training and tuning image set, and then manually annotate all the feature points of each 2D facial image in the first-level training and tuning image set. Input the first-level training and tuning image set with manually annotated feature points into the HRNet face recognition model, and then use the SBR algorithm to perform first-level tuning training on the HRNet face recognition model that has undergone first-level basic training. The ratio of the number of 2D facial images corresponding to the training set, validation set, and test set used in the first-level tuning training is 7:2:

1. S2.4: For each 2D face image in the first-level training and tuning image set, the HRNet face recognition model that has undergone first-level tuning and training automatically identifies a unique feature point corresponding to each manually annotated feature point; S2.5: For each 2D face image in the first-level training and tuning image set, a 2D coordinate space is constructed on the 2D face image, and the Euclidean distance between each manually annotated feature point on the 2D face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the 2D coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the 2D coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model; The order of step S2.1 and step S2.2 can be interchanged.

5. The method for acquiring facial motion attributes based on markerless motion capture according to claim 3, characterized in that: The step S3 includes the following sub-steps: S3.1: for any test subject, collecting a two-dimensional image group of the test subject's facial movements; S3.2: Repeat step S3.1 for different test subjects to obtain a two-dimensional image group of facial movements of each test subject, and the two-dimensional image groups of facial movements of all test subjects constitute a secondary training basic image set, and the secondary training basic image set contains at least 60,000 frames of two-dimensional images; S3.3: using a random function to sample 10% of the two-dimensional images from the above-mentioned secondary training basic image set as the secondary training sample image set; S3.4: for each frame of the two-dimensional image in the secondary training sample image set, all feature points on the two-dimensional image are marked by manual marking; S3.5: For each 2D face image in the secondary training sample image set after manually annotating feature points, the HRNet face recognition model automatically identifies a unique feature point corresponding to each manually annotated feature point; S3.6: For each two-dimensional face image in the secondary training sampling image set, a two-dimensional coordinate space is constructed on the two-dimensional face image, and the Euclidean distance between each manually annotated feature point on the two-dimensional face image and the corresponding feature point automatically recognized by the HRNet face recognition model is calculated in the two-dimensional coordinate space, so as to obtain the accuracy of the HRNet face recognition model. The accuracy of the HRNet face recognition model is determined according to the degree of overlap between each manually annotated feature point in the two-dimensional coordinate space and the corresponding feature point automatically recognized by the HRNet face recognition model.

6. The method for acquiring facial motion attributes based on markerless motion capture according to claim 3, characterized in that: The step S4 includes the following sub-steps: S4.1: Set up some basic facial expression units; S4.2: using the first camera, the second camera and the third camera of the system to synchronously capture a series of two-dimensional images of the facial movements of the same test subject and transmit them to the computer in real time, obtaining a face measurement image set of the test subject currently being measured and inputting it into the HRNet face recognition model; S4.3: Input the face measurement image set of the subject being measured into the HRNet face recognition model, and the HRNet face recognition model automatically recognizes all feature points on each two-dimensional face image in the face measurement image set; S4.4: For any feature point on the face of the subject being measured, the HRNet face recognition model automatically obtains the position coordinates and motion trajectory of the feature point in the three-dimensional coordinate space based on the measurement image set of the subject being measured; S4.5: Repeat step S4.4 until the HRNet face recognition model obtains the position coordinates and motion trajectories of all feature points on the face of the currently measured subject in the three-dimensional coordinate space, and generates a facial movement measurement report of the currently measured subject.

7. The method for acquiring facial motion attributes based on markerless motion capture according to claim 6, characterized in that: The step S4.4 further comprises the following sub-steps: S4.4.1: For any feature point on the face of the subject being measured, a three-dimensional coordinate space is constructed using the position of the feature point on different two-dimensional images obtained by synchronously photographing two or more cameras of the system, and the position coordinates of the feature point in the three-dimensional coordinate space are obtained based on the binocular imaging principle; S4.4.2: For any feature point on the face of the subject currently being measured, the motion trajectory of the feature point in the three-dimensional coordinate space is obtained based on the change of the position coordinates of the feature point in the three-dimensional coordinate space over time.

8. The method for acquiring facial motion attributes based on markerless motion capture according to claim 7, characterized in that: The step S4.4.1 includes: for any feature point on the face of the subject currently being measured, the two-dimensional image obtained by taking pictures with the first camera at any time point is the A1 frame image, and the two-dimensional image obtained by taking pictures with the second camera at the same time point is the B1 frame image, and a three-dimensional coordinate space is constructed using the position of the feature point on the A1 frame image and the position of the feature point on the B1 frame image, and the position coordinates of the feature point in the three-dimensional coordinate space are obtained based on the binocular imaging principle.

9. The method for acquiring facial motion attributes based on markerless motion capture according to claim 4, characterized in that: In the step S2.5, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME as a model evaluation index; In step S2.5, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the corresponding feature point automatically recognized by the HRNet face recognition model is zero.

10. The method for acquiring facial motion attributes based on markerless motion capture according to claim 5, characterized in that: In the step S3.6, the two-dimensional coordinates of the two-dimensional face image are normalized using the distance between the two pupils on the two-dimensional face image, and the HRNet face recognition model is evaluated using NME as a model evaluation index; In step S3.6, for any feature point on the two-dimensional face image, the optimal situation is that the Euclidean distance between the manually marked feature point and the corresponding feature point automatically recognized by the HRNet face recognition model is zero.

Images