Real-time gaze estimation method and system based on facial feature calibration

Abstract

The application provides a real-time gaze estimation method and system based on facial feature calibration, comprising the following steps: S1, determining the position of the eyeball center of a user through a facial feature calibration method; S2, determining the position of the pupil center of the user through a facial key point detection algorithm; and converting the position of the eyeball center of the user and the position of the pupil center of the user to a camera coordinate system, and calculating the line direction as the gaze direction of the user. The calculation process of the gaze estimation method is light and efficient, the requirement for computing power is low, the accuracy of the gaze estimation is high, and the real-time performance is good.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a real-time gaze estimation method and system based on facial feature calibration. Background Technology

[0002] Human eye gaze contains a wealth of information, including intent, cognition, behavior, and attention. Gaze estimation technology has wide applications in cognitive science, healthcare, driver assistance, virtual reality, human-computer interaction, and sports.

[0003] Existing gaze estimation techniques mainly fall into three categories:

[0004] 1) Regression-based 2D gaze point estimation. This method typically uses multiple infrared light sources to generate light spots on the corneal surface. The coordinates of these light spots, captured by an infrared camera, and the pupil center coordinates are input into a regression function, directly mapping them to the 2D gaze point coordinates on the screen observed by the user. This type of method usually requires a multi-point calibration process to determine the unknown parameters in the regression function. This method has several drawbacks, such as being susceptible to head movements, requiring the user's head to be fixed, and the calibration process being cumbersome and resulting in a poor user experience; it also has certain hardware requirements, typically requiring multiple infrared light sources to ensure accuracy; it is only applicable to scenarios where screen observation is required, and the 2D gaze point contains limited information, losing 3D information, which limits the practical application of this method.

[0005] 2) Gaze estimation based on 3D eye models. This method typically uses multiple light sources or cameras to reconstruct a 3D model of the user's eyes and determines the user's gaze direction based on the reconstructed model. Because different users have different eye shapes, this type of method usually requires a calibration process to determine the parameters of the eye model. These methods generally have high accuracy and good robustness to user head movements; however, they require sophisticated hardware, which limits application scenarios and increases application costs.

[0006] 3) Appearance-based gaze estimation. This typically uses a single RGB camera to capture user facial images, extracts features from the images, and then uses a function or model to map the features to the user's gaze direction. This type of method uses an RGB camera, has low hardware requirements, and is therefore relatively flexible in application scenarios. However, it requires very effective feature extraction methods and mapping functions; otherwise, it is difficult to achieve good results. Therefore, in recent years, many gaze estimation models based on deep neural networks have emerged, capable of performing feature extraction and mapping simultaneously, achieving end-to-end gaze estimation. However, to ensure the accuracy and robustness of the model, these methods usually require designing complex network models and training with large amounts of data, which increases their application cost and computational consumption, and also limits real-time performance. Furthermore, since these methods directly calculate the gaze direction from RGB images, many factors such as image quality, background environment, user differences, and camera parameters can limit their generalization performance in practical applications. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this invention is to provide a real-time gaze estimation method and system based on facial feature calibration.

[0008] A real-time gaze estimation method based on facial feature calibration provided by the present invention includes:

[0009] Step S1: Determine the location of the center of the user's eyeball using facial feature mapping methods;

[0010] Step S2: The facial image to be detected uses a facial key point detection algorithm to determine the position of the user's pupil center; the position of the user's eyeball center and the position of the user's pupil center are transformed into the camera coordinate system, and the direction of the line connecting them is calculated as the user's gaze direction.

[0011] Preferably, step S1 employs:

[0012] Step S1.1: Acquire calibration images;

[0013] Step S1.2: Obtain the coordinates of each key point in the facial key point set in the facial coordinate system based on the calibration image set; calculate the coordinates of each key point in the facial key point set in the pixel coordinate system based on each facial image in the calibration image set; calculate the camera extrinsic parameters corresponding to each facial image in the calibration image set based on the coordinates of the key points in the facial key point set in the pixel coordinate system and the facial coordinate system, respectively, combined with the camera intrinsic parameters.

[0014] Step S1.3: Calculate the coordinates of the pupil center in the pixel coordinate system based on each facial image in the calibration image set using facial key point detection technology, and calculate the linear equation of the line of sight in the facial coordinate system based on camera intrinsic and extrinsic parameters.

[0015] Step S1.4: Obtain all line equations of the gaze based on all facial images in the calibration image set, and calculate the coordinates of the user's eye center by finding the intersection of all line equations of the gaze.

[0016] Preferably, step S1.1 involves the user keeping their gaze fixed on the center of the camera lens and freely turning and moving their head, while continuously capturing RGB images of the user's face using the camera, and using the captured RGB images as a calibration image set.

[0017] Preferably, step S1.2 employs the following:

[0018] Step S1.2.1: Reconstruct the user's facial 3D model using facial 3D modeling technology based on the user's facial images in the calibration image set, and obtain the coordinates of each key point in the facial key point set in the facial coordinate system based on the user's facial 3D model;

[0019] Step S1.2.2: Calculate the coordinates of each key point in the pixel coordinate system of the facial key point set based on each facial image in the calibration image set using facial key point detection technology.

[0020] Preferably, step S2 employs:

[0021] Step S2.1: For the face image to be detected, transform the coordinates of the two pupil centers from the pixel coordinate system to the camera coordinate system according to the camera intrinsic parameters;

[0022] Step S2.2: For the face image to be detected, calculate the coordinates of each key point in the pixel coordinate system of the facial key point set, and combine the camera intrinsic parameters and the coordinates of each key point in the facial coordinate system to calculate the camera extrinsic parameters corresponding to the face image to be detected.

[0023] Step S2.3: Based on the camera extrinsic parameters, transform the coordinates of the user's two eyeball centers from the facial coordinate system to the camera coordinate system;

[0024] Step S2.4: For the left and right eyes, calculate the unit vector pointing from the center of the eyeball to the center of the pupil, and calculate the average value of the current vector for the left and right eyes. Use the current average value as the three-dimensional vector of the user's gaze direction in the camera coordinate system. The current three-dimensional vector of the gaze direction is the gaze estimation result.

[0025] A real-time gaze estimation system based on facial feature calibration provided by the present invention includes:

[0026] Module M1: Determines the location of the center of the user's eyeball using facial feature calibration methods;

[0027] Module M2: The facial image to be detected uses a facial key point detection algorithm to determine the position of the user's pupil center; the position of the user's eyeball center and the position of the user's pupil center are transformed into the camera coordinate system, and the direction of the line connecting them is calculated as the user's gaze direction.

[0028] Preferably, module M1 adopts:

[0029] Module M1.1: Acquires calibration images;

[0030] Module M1.2: Based on the calibration image set, obtain the coordinates of each key point in the facial key point set in the facial coordinate system; based on each facial image in the calibration image set, calculate the coordinates of each key point in the facial key point set in the pixel coordinate system; based on the coordinates of the key points in the facial key point set in the pixel coordinate system and the facial coordinate system respectively, and combined with the camera intrinsic parameters, calculate the camera extrinsic parameters corresponding to each facial image in the calibration image set.

[0031] Module M1.3: Utilizes facial key point detection technology to calculate the coordinates of the pupil center in the pixel coordinate system based on each facial image in the calibration image set, and calculates the linear equation of the line of sight in the facial coordinate system based on camera intrinsic and extrinsic parameters;

[0032] Module M1.4: Based on all facial images in the calibration image set, obtain the equations of all line lines of sight, and calculate the coordinates of the user's eye center by the intersection of all line lines of sight.

[0033] Preferably, module M1.1 employs the following method: the user keeps their gaze fixed on the center of the camera lens and freely rotates and moves their head, using the camera to continuously capture RGB images of the user's face, and uses the acquired RGB images as a calibration image set.

[0034] Preferably, module M1.2 adopts:

[0035] Module M1.2.1: Reconstructs the user's facial 3D model based on the user's facial images in the calibration image set using facial 3D modeling technology, and obtains the coordinates of each key point in the facial key point set in the facial coordinate system based on the user's facial 3D model;

[0036] Module M1.2.2: Utilizes facial key point detection technology to calculate the coordinates of each key point in the pixel coordinate system based on each facial image in the calibration image set.

[0037] Preferably, the module M2 adopts:

[0038] Module M2.1: For the facial image to be detected, the coordinates of the two pupil centers are transformed from the pixel coordinate system to the camera coordinate system based on the camera intrinsic parameters;

[0039] Module M2.2: For the face image to be detected, calculate the coordinates of each key point in the facial key point set in the pixel coordinate system, and combine the camera intrinsic parameters and the coordinates of each key point in the facial key point set in the facial coordinate system to calculate the camera extrinsic parameters corresponding to the face image to be detected.

[0040] Module M2.3: Transforms the coordinates of the user's two eyeball centers from the facial coordinate system to the camera coordinate system based on camera extrinsic parameters;

[0041] Module M2.4: For the left and right eyes, calculate the unit vector from the center of the eyeball to the center of the pupil, and calculate the average value of the current vector for the left and right eyes. Use the current average value as the three-dimensional vector of the user's gaze direction in the camera coordinate system. The current three-dimensional vector of the gaze direction is the gaze estimation result.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. By adopting the method of the present invention, the real-time and accurate calculation of the gaze direction of a person in a real scene is realized using a single monocular RGB camera, without the need for other professional equipment or instruments, such as infrared light sources, depth cameras, eye-tracking sensors, etc., and can be easily implemented in various application scenarios.

[0044] 2. In the facial feature calibration method proposed in this invention, the calibration image acquisition process is simple and quick, requiring no professional personnel to guide the process; the calibration calculation is short, and the calibration results are accurate and reliable.

[0045] 3. The gaze estimation method proposed in this invention has a lightweight and efficient computation process, low computational requirements, high accuracy in gaze estimation, and good real-time performance.

[0046] 4. The proposed method is plug-and-play, requiring no complex deep network models or training on large-scale datasets, resulting in lower application costs and computational consumption.

[0047] 5. The gaze estimation results are robust to changes in head pose, so there is no need to fix the user's head, which greatly improves the ease of use and user experience compared to existing methods. Attached Figure Description

[0048] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0049] Figure 1 This is a diagram of the overall plan.

[0050] Figure 2 Flowchart for facial feature mapping.

[0051] Figure 3A geometric diagram of the calculation for the center of the eyeball.

[0052] Figure 4 Flowchart for real-time gaze estimation.

[0053] Figure 5 The image shows the experimental results.

[0054] Figure 6 This is a visualization of all lines of sight calculated from the calibration image set during the experiment.

[0055] Figure 7 This is a statistical graph showing the gaze estimation results for different numbers of calibration images in the experiment. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0057] This invention provides a real-time gaze estimation method and system based on facial feature calibration. By combining the geometric model of the human eye and the optical model of a camera, a simple and accurate facial feature calibration method is designed. Based on this method, a real-time gaze estimation method is implemented, which can realize the real-time and accurate calculation of the gaze direction of a person. The image acquisition device used in the above facial feature calibration and real-time gaze estimation method is only a monocular RGB camera, without the need for other professional hardware such as infrared light sources, depth cameras, eye-tracking sensors, etc.

[0058] Example 1

[0059] This invention provides a real-time gaze estimation method based on facial feature calibration, such as... Figure 2 , Figure 4 As shown, it includes:

[0060] Step S1: Determine the location of the center of the user's eyeball using facial feature mapping methods;

[0061] Step S2: The facial image to be detected uses a facial key point detection algorithm to determine the position of the user's pupil center; the position of the user's eyeball center and the position of the user's pupil center are transformed into the camera coordinate system, and the direction of the line connecting them is calculated as the user's gaze direction.

[0062] like Figure 2As shown, facial feature calibration includes: The purpose of facial feature calibration is to determine the coordinates (fixed values) of the user's two eyeball centers and multiple facial key points in the facial coordinate system, and to reconstruct the user's three-dimensional facial model defined in the facial coordinate system. The key aspect is determining the coordinates of the user's two eyeball centers.

[0063] Among them, the Face Coordinate System (FCS) is a three-dimensional coordinate system that is fixed to the face of the person and is a right-handed coordinate system with the direction of the face facing the positive z-axis.

[0064] Specifically, step S1 employs the following:

[0065] Step S1.1: Acquire calibration images; The user keeps their gaze focused on the center of the camera lens and freely turns and moves their head. The camera continuously captures RGB images of the user's face during this process, which will serve as the calibration image set for subsequent steps. It is important to note that the user should try to keep their face intact and clear in the captured images, as this will help improve calibration accuracy.

[0066] Step S1.2: Reconstruct the user's 3D facial model using facial 3D modeling technology based on the user's facial images in the calibration image set. Obtain the coordinates of each keypoint in the facial keypoint set in the facial coordinate system based on the user's 3D facial model. Calculate the coordinates of each keypoint in the facial keypoint set in the pixel coordinate system based on each facial image in the calibration image set using facial keypoint detection technology. Calculate the camera extrinsic parameters based on the coordinates of the keypoints in the pixel coordinate system and the facial coordinate system, combined with the camera intrinsic parameters.

[0067] More specifically, the facial 3D modeling uses 3DMM technology to reconstruct a 3D model of the user's face based on the user's facial images in the calibration image set. The 3D facial model is defined in the aforementioned facial coordinate system and consists of multiple vertices distributed across the entire face. It should be noted that the 3D facial model is unique to each user and reflects the shape information of the user's face, but does not include information such as expression, texture, or pose.

[0068] More specifically, the camera extrinsic parameters are defined as follows: a set of facial key points is defined, including but not limited to points distributed across the entire face such as the corners of the eyes, the tip of the nose, and the corners of the mouth, which can reflect the facial pose, thereby ensuring the efficiency and robustness of subsequent calculations. Using existing facial key point detection techniques, for each facial image in the calibration image set, the coordinates of each key point in the set of facial key points in the pixel coordinate system are calculated. Based on the facial 3D model, the coordinates of each key point in the set of facial key points in the facial coordinate system are obtained. The camera's intrinsic parameters, or camera intrinsic parameters for short, are one of the characteristics of the camera and can be obtained through pre-calibration or by consulting the data of the camera used. Camera intrinsic parameters can be used to transform the coordinates of points between the pixel coordinate system and the camera coordinate system. The camera's extrinsic parameters, or camera extrinsic parameters for short, reflect the pose of the camera relative to the object being photographed when capturing an image. Camera extrinsic parameters can be used to transform the coordinates of points between the camera coordinate system and the face coordinate system. Based on the coordinates of the key points in the set of facial key points in the pixel coordinate system and the face coordinate system, respectively, combined with the camera intrinsic parameters, the camera extrinsic parameters can be calculated. This problem is called the Perspective-n-Points (PnP) problem, which can be solved iteratively using the Levenberg-Marquardt method.

[0069] The pixel coordinate system (PCS) is a two-dimensional coordinate system that is fixed to the image. The origin is located at the top left corner of the image, the positive x-axis is horizontal to the right, and the positive y-axis is horizontal downward. One pixel is one unit length.

[0070] The camera coordinate system (CCS) is a three-dimensional coordinate system that is fixed to the camera. Its origin is located at the center of the camera lens, and it is a right-handed coordinate system with the positive z-axis pointing outward from the lens.

[0071] Step S1.3: For each facial image in the calibration image set, calculate the coordinates of the user's pupil center in the pixel coordinate system based on facial key point detection technology, and calculate the linear equation of the line of sight in the facial coordinate system based on camera intrinsic and extrinsic parameters.

[0072] More specifically, this step considers only one of the user's eyes; the content is identical for the other eye. Because the user maintains a gaze at the camera lens center during the acquisition of the calibration image set, the user's eye center, pupil center, and camera lens center lie on the same straight line in both the camera coordinate system and the facial coordinate system. This straight line represents the user's line of sight (i.e., the line of gaze direction). Figure 3As shown. According to the optical model of a pinhole camera, the projection point of the gaze on the image coincides with the center of the pupil. For each facial image in the calibration image set, the coordinates of the user's pupil center in the pixel coordinate system can be calculated using the facial key point detection technology. As mentioned above, these coordinates are the coordinates of the projection point of the gaze on the image. Based on these coordinates, combined with the camera's intrinsic and extrinsic parameters, the linear equation of the gaze in the facial coordinate system (hereinafter referred to as the gaze equation) can be calculated. Note that the above calculation process is performed separately on each image in the calibration image set.

[0073] Step S1.4: Obtain all line equations of the gaze based on all facial images in the calibration image set, and calculate the coordinates of the user's eye center by finding the intersection of all line equations of the gaze.

[0074] More specifically, such as Figure 3 As shown, this step only considers one of the user's eyes. Because the user's gaze direction changes continuously during the acquisition of the calibration image set, the calculated gaze equation in the facial coordinate system is different for each image. Since gazes in any direction pass through the center of the eyeball, and the position of the eyeball center in the facial coordinate system remains fixed, the gaze equations calculated from different calibration images in the calibration image set intersect at a point in the facial coordinate system at the center of the eyeball. By calculating the intersection point of all gaze equations in the facial coordinate system, the coordinates of the user's eyeball center can be obtained. This calculation process can be transformed into a least-squares optimization problem and solved.

[0075] Specifically, step S2 involves: determining the position of the user's pupil center using a facial key point detection algorithm on the face image to be detected; converting the position of the user's eyeball center and the position of the user's pupil center to the camera coordinate system, and calculating the direction of the line connecting them as the user's gaze direction.

[0076] More specifically, after facial feature calibration, the camera can be used to perform real-time gaze estimation of the user. The input to the gaze estimation process is an RGB image of the user's face captured by the camera, and the output is a three-dimensional vector of the user's gaze direction in the camera coordinate system. Known information includes camera intrinsic parameters, the user's three-dimensional facial model, the coordinates of key points in the user's facial keypoint set in the facial coordinate system, and the coordinates of the centers of the user's two eyeballs in the facial coordinate system. For the input image, the facial keypoint detection technology is used to calculate the coordinates of the key points in the facial keypoint set and the centers of the two pupils in the pixel coordinate system. Combined with the coordinates of the key points in the user's facial keypoint set in the facial coordinate system, the camera extrinsic parameters are calculated. Based on the camera intrinsic parameters, the coordinates of the two pupil centers are transformed from the pixel coordinate system to the camera coordinate system. Based on the camera extrinsic parameters, the coordinates of the user's two eyeball centers are transformed from the facial coordinate system to the camera coordinate system. For the left and right eyes, a unit vector pointing from the center of the eyeball to the center of the pupil is calculated, and the average value of this vector is calculated for the left and right eyes, which is used as the three-dimensional vector of the user's gaze direction in the camera coordinate system, i.e., the gaze estimation result.

[0077] More specifically, step S1.2 employs:

[0078] The 3DMM technology is an existing technology. 3DMM stands for 3D Morphable Model, a type of model that decouples facial shape, texture, expression, and pose information. It uses multiple parameters corresponding to this information to control an initial 3D facial model, generating a 3D facial model that meets the requirements. Through projection, the 3D facial model can be projected onto its corresponding facial image. 3DMM technology can reconstruct a 3D facial model from a single facial image: given a facial image, the parameters can be adjusted iteratively to make the projected facial image from the parameter-controlled 3D model approximate the known facial image; the resulting 3D facial model is the reconstructed 3D facial model. In step S1.2, the 3D facial model is reconstructed from multiple user facial images in the calibration image set. Different facial images of the same user should have the same shape information, but their texture, expression, and pose information may differ. Therefore, for each facial image in the calibration image set, the aforementioned 3DMM technology is used to perform facial 3D modeling. The resulting multiple sets of shape parameters are averaged, and the resulting texture, expression, and pose parameters are not used. The facial 3D model generated using the averaged shape parameters is then used as the user's facial 3D model. It should be noted that reconstructing a facial 3D model from multiple facial images using 3DMM technology is not limited to the parameter averaging method described above; this method is the solution selected in this invention.

[0079] The facial landmark detection technology uses the existing MediaPipe model, which can detect the location of facial landmarks on RGB facial images, covering the set of facial landmarks, and has good accuracy and real-time performance. It should be noted that using the MediaPipe model is one method for implementing facial landmark detection, and it is the solution chosen in this invention.

[0080] More specifically, step S1.2 employs the transformation relationship between the three coordinate systems, which can be explained by the following formula.

[0081] Let L j =[u j v j ] T Let j = 1, ..., m represent the coordinates of the facial keypoint set in the pixel coordinate system, with a total of m facial keypoints. Let Let represent the coordinates of the facial keypoint set in the facial coordinate system. This represents the coordinates of the facial keypoint set in the camera coordinate system. K is the camera intrinsic parameter, a 3×3 matrix. T = [R t] is the camera extrinsic parameter, a 3×4 matrix, where R and t are the rotation matrix and translation vector, respectively. Then we have...

[0082]

[0083] in, L represents j The homogeneous coordinate form, i.e. express The homogeneous coordinate form, i.e.

[0084] The Levenberg-Marquardt method is one approach to solving PnP problems and is the solution chosen in this invention.

[0085] More specifically, the calculation principle of the line-of-sight equation in step S1.3 can be explained by the following formula.

[0086] Let L p，i =[u p，i v p，i ] T This represents the coordinates of the pupil center in the pixel coordinate system for the i-th facial image in the calibration image set, obtained using the facial keypoint detection technique. (Two distinct real numbers can be chosen.) Let z represent two different z-axis coordinates in the camera coordinate system. According to the above formula (1), the two different coordinates in the face coordinate system can be calculated. It satisfies the following relationship,

[0087]

[0088]

[0089] in, L represents p，i homogeneous coordinate form, express homogeneous coordinate form, express The homogeneous coordinate form of T. i This indicates that for the i-th facial image in the calibration image set, the camera extrinsic parameters are obtained using the method for solving camera extrinsic parameters described in step three. According to formulas (2) and (3), the pixel coordinates projected onto the image by the two coordinates are both L. p，i That is, the center of the pupil. Based on the optical principles of a pinhole camera, in the facial coordinate system, after... The straight line between the two points represents the line of sight corresponding to the i-th facial image. The equation of this line of sight in the facial coordinate system can be expressed as:

[0090]

[0091] in, This represents the coordinates of any point on the line.

[0092] It should be noted that this calculation process is one method for calculating the line-of-sight equation, and it is the solution selected in this invention.

[0093] More specifically, step S1.4 uses the calculation principle of the eyeball center coordinates, which can be explained by the following formula.

[0094] Since the center of the eyeball is fixed relative to the three-dimensional facial model, the coordinates of the center of the eyeball in the facial coordinate system are also unique. When i takes different values, that is, for different facial images, the line of sight represented by formula (4) is different, but they all pass through the center of the eyeball. Therefore, the coordinates of the intersection point of the straight line equation calculated by formula (4) for all facial images are the coordinates of the center of the eyeball in the facial coordinate system. Since the number of constraint equations is greater than the number of unknowns, this calculation process can be transformed into solving a least squares optimization problem. Finally, the coordinates of the center of the eyeball can be calculated according to the following formula (5).

[0095]

[0096] Where, vector I represents the identity matrix.

[0097] It should be noted that this calculation process is one method for calculating the coordinates of the eyeball center, and it is the solution selected in this invention.

[0098] In step S2, the calculation principle of the pupil center coordinates, eyeball center coordinates, and gaze direction unit vector in the camera coordinate system can be explained by the following formula.

[0099] make The vertex coordinates of the 3D facial model (defined in the facial coordinate system) are represented by n, where n is the number of vertices. According to formula (1) and the camera extrinsic parameter T, the 3D facial model can be transformed to the camera coordinate system, where the vertex coordinates are... The following formula is satisfied.

[0100]

[0101] in, for The homogeneous coordinate form.

[0102] Based on the coordinates of the eyeball center in the facial coordinate system Its coordinates in the camera coordinate system can be calculated. The following formula is satisfied.

[0103]

[0104] in, for The homogeneous coordinate form.

[0105] For the input image of the real-time gaze estimation process described in step S2, let L pupil This represents the coordinates of the pupil center in the pixel coordinate system obtained using the facial key point detection technology. According to formula (1) and the camera intrinsic parameter K, the 3D facial model can be transformed to the pixel coordinate system, with vertex coordinates L... k The following formula is satisfied.

[0106]

[0107] in, For L k The homogeneous coordinate form. In k = 1, ..., n, such that L k With L pupil The coordinates of the vertex corresponding to the nearest k value That is, the coordinates of the pupil center in the camera coordinate system.

[0108] The unit vector pointing from the center of the eyeball to the center of the pupil can be represented as:

[0109]

[0110] The three-dimensional vector of the gaze direction is the aforementioned vector v. gaze The average value for both eyes.

[0111] The present invention also provides a real-time gaze estimation system based on facial feature calibration. The real-time gaze estimation system based on facial feature calibration can be implemented by executing the process steps of the real-time gaze estimation method based on facial feature calibration. That is, those skilled in the art can understand the real-time gaze estimation method based on facial feature calibration as a preferred embodiment of the real-time gaze estimation system based on facial feature calibration.

[0112] To verify the effectiveness of the gaze estimation method proposed in this invention, a human gaze estimation dataset, called the IMR-CaliGaze dataset, was collected. This dataset contains approximately 1600 facial RGB images from 8 subjects and corresponding 3D gaze direction annotations, as well as approximately 200 calibration images for each subject acquired according to the calibration image acquisition process. The camera intrinsics of the RGB camera used in the dataset were obtained through the camera calibration process of the OpenCV library. The proposed method was compared with four existing depth model-based gaze estimation methods: GazeNet, Full-Face, Dilated-Net, and GazeTR. These four methods also use facial images captured by an RGB camera as input to obtain gaze estimation results. Angle deviation, i.e., the angle between the estimated result vector and the labeled vector, was used as a metric for the gaze estimation effect.

[0113] For the four deep learning model methods, firstly, a leave-one-subject-out testing method is used on the IMR-CaliGaze dataset. Data from one subject is used for testing, while data from the remaining subjects is used for training. The test results for each subject are recorded, referred to as the Single-dataset test results. Subsequently, the four deep learning models are trained using an existing gaze estimation dataset, MPIIFaceGaze, and then tested on the IMR-CaliGaze dataset to obtain the test results for each subject, referred to as the Cross-dataset test results. For the method proposed in this invention, due to its plug-and-play nature and the absence of a training process, testing is directly performed on the data of each subject in the IMR-CaliGaze dataset. The above test results are recorded in... Figure 5As can be seen from the results, the gaze estimation effect of the method of the present invention is the best, with both the mean and standard deviation of the angle deviation being smaller than those of existing methods. Furthermore, on an i7-11800H CPU, the method of the present invention can achieve a processing speed of 96 frames per second for a 1280*720 input image, demonstrating good real-time performance.

[0114] like Figure 6 As shown, for the calibration image set of a subject in the IMR-CaliGaze dataset, the line of sight equation in the facial coordinate system is calculated; the line of sight calculated for each calibration image is visualized together with the reconstructed facial 3D model, and it can be seen that for the left and right eyes, the line of sight intersects well at the center point of the eyeball.

[0115] For each subject in the IMR-CaliGaze dataset, eye center calibration was performed on randomly selected images from the calibration image set in varying quantities, and the calibration results were used for fixation estimation. For each subject at each quantity level, 10 sets of calibration images were randomly selected for testing, and the mean and standard deviation of the angle deviation of the fixation estimation results were calculated. Figure 7 .

[0116] As can be seen, the gaze estimation performance tends to stabilize after more than 50 calibration images, and reaches its best performance with 150 or more images. Even with only 5 calibration images, the gaze estimation accuracy of the method of this invention is comparable to... Figure 5 The existing methods in this area are at the same level. According to the test, based on the characteristic of the calibration image acquisition process that can continuously acquire images, all subjects were able to complete the acquisition of more than 200 calibration images within 20 seconds, which is also due to the advantages of the simple and fast acquisition process.

[0117] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0118] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A real-time gaze estimation method based on facial feature calibration, characterized in that, include: Step S1: Determine the location of the center of the user's eyeball using facial feature mapping methods; Step S2: The facial image to be detected uses a facial key point detection algorithm to determine the position of the user's pupil center; the position of the user's eyeball center and the position of the user's pupil center are transformed into the camera coordinate system, and the direction of the line connecting them is calculated as the user's gaze direction; Step S1 adopts the following: Step S1.1: Acquire calibration images; The step S1.1 involves the user keeping their gaze focused on the center of the camera lens and freely turning and moving their head. The camera continuously captures RGB images of the user's face, and the collected RGB images are used as a calibration image set. Step S1.2: Based on the calibration image set, obtain the coordinates of each key point in the facial key point set in the facial coordinate system, and calculate the coordinates of each key point in the facial key point set in the pixel coordinate system based on each facial image in the calibration image set. Based on the coordinates of the key points in the facial key point set in the pixel coordinate system and the facial coordinate system, the camera extrinsic parameters corresponding to each facial image in the calibration image set are calculated by combining the camera intrinsic parameters. Step S1.3: Calculate the coordinates of the pupil center in the pixel coordinate system based on each facial image in the calibration image set using facial key point detection, and calculate the linear equation of the line of sight in the facial coordinate system based on camera intrinsic and extrinsic parameters. Step S1.4: Based on all facial images in the calibration image set, obtain all line equations of gaze, calculate the intersection of all line equations of gaze to obtain the coordinates of the user's eye center. The process of calculating the intersection of all line equations of gaze is transformed into solving a least squares optimization problem. The image acquisition device consists of only a single monocular RGB camera; Step S2 employs the following: Step S2.1: For the face image to be detected, transform the coordinates of the two pupil centers from the pixel coordinate system to the camera coordinate system according to the camera intrinsic parameters; Step S2.2: For the face image to be detected, calculate the coordinates of each key point in the pixel coordinate system of the facial key point set, and combine the camera intrinsic parameters and the coordinates of each key point in the facial coordinate system to calculate the camera extrinsic parameters corresponding to the face image to be detected. Step S2.3: Based on the camera extrinsic parameters, transform the coordinates of the user's two eyeball centers from the facial coordinate system to the camera coordinate system; Step S2.4: For the left and right eyes, calculate the unit vector pointing from the center of the eyeball to the center of the pupil, and calculate the average value of the current vector for the left and right eyes. Use the current average value as the three-dimensional vector of the user's gaze direction in the camera coordinate system. The current three-dimensional vector of the gaze direction is the gaze estimation result.

2. The real-time gaze estimation method based on facial feature calibration according to claim 1, characterized in that, Step S1.2 adopts the following: Step S1.2.1: Reconstruct the user's facial 3D model based on the user's facial images in the calibration image set using facial 3D modeling, and obtain the coordinates of each key point in the facial key point set in the facial coordinate system based on the user's facial 3D model; Step S1.2.2: Calculate the coordinates of each key point in the pixel coordinate system of the facial key point set based on each facial image in the calibration image set using facial key point detection.

3. A real-time gaze estimation system based on facial feature calibration, characterized in that, include: Module M1: Determines the location of the center of the user's eyeball using facial feature calibration methods; Module M2: The facial image to be detected uses a facial key point detection algorithm to determine the position of the user's pupil center; the position of the user's eyeball center and the position of the user's pupil center are transformed into the camera coordinate system, and the direction of the line connecting them is calculated as the user's gaze direction; The module M1 adopts: Module M1.1: Acquires calibration images; The module M1.1 employs the following method: the user keeps their gaze fixed on the center of the camera lens and freely rotates and moves their head, using the camera to continuously capture RGB images of the user's face, and uses the captured RGB images as a calibration image set. Module M1.2: Based on the calibration image set, obtain the coordinates of each key point in the facial key point set in the facial coordinate system, and calculate the coordinates of each key point in the facial key point set in the pixel coordinate system based on each facial image in the calibration image set. Based on the coordinates of the key points in the facial key point set in the pixel coordinate system and the facial coordinate system, the camera extrinsic parameters corresponding to each facial image in the calibration image set are calculated by combining the camera intrinsic parameters. Module M1.3: Utilizes facial key point detection to calculate the coordinates of the pupil center in the pixel coordinate system based on each facial image in the calibration image set, and calculates the linear equation of the line of sight in the facial coordinate system based on camera intrinsic and extrinsic parameters; Module M1.4: Based on all facial images in the calibration image set, obtain all line equations of the gaze lines, calculate the intersection of all line equations of the gaze lines to obtain the coordinates of the user's eye center. The process of calculating the intersection of all line equations of the gaze lines is transformed into solving a least squares optimization problem. The image acquisition device consists of only a single monocular RGB camera; The module M2 adopts: Module M2.1: For the facial image to be detected, the coordinates of the two pupil centers are transformed from the pixel coordinate system to the camera coordinate system based on the camera intrinsic parameters; Module M2.2: For the face image to be detected, calculate the coordinates of each key point in the facial key point set in the pixel coordinate system, and combine the camera intrinsic parameters and the coordinates of each key point in the facial key point set in the facial coordinate system to calculate the camera extrinsic parameters corresponding to the face image to be detected. Module M2.3: Transforms the coordinates of the user's two eyeball centers from the facial coordinate system to the camera coordinate system based on camera extrinsic parameters; Module M2.4: For the left and right eyes, calculate the unit vector from the center of the eyeball to the center of the pupil, and calculate the average value of the current vector for the left and right eyes. Use the current average value as the three-dimensional vector of the user's gaze direction in the camera coordinate system. The current three-dimensional vector of the gaze direction is the gaze estimation result.

4. The real-time gaze estimation system based on facial feature calibration according to claim 3, characterized in that, The module M1.2 adopts: Module M1.2.1: Using facial 3D modeling, the user's facial 3D model is reconstructed from the user's facial images in the calibration image set, and the coordinates of each key point in the facial key point set in the facial coordinate system are obtained from the user's facial 3D model. Module M1.2.2: Utilizes facial keypoint detection to calculate the coordinates of each keypoint in the pixel coordinate system based on each facial image in the calibration image set.

Images