A gaze estimation ground truth data recording apparatus and method

Abstract

The application particularly relates to the field of gaze estimation true value data recording technology, and discloses a gaze estimation true value data recording method which adopts a gaze estimation true value data recording device, the device comprising: a head fixing device, an Aruco code pasted on the head fixing device, a calibration plate with a gaze point, a tof camera and a second camera. The gaze estimation true value data recording method adopts the 3D distance from the center point of the Aruco code on the head fixing device worn by the true value data recorder to the left and right pupil points, the center point coordinate value obtained by detecting the Aruco code at the initial moment, the initial moment angle value of the Aruco code in the pitch and yaw directions, and the 3D coordinates of the left and right pupils of the recorder at the initial moment, and combines the 3D coordinates of the head posture of the recorder on the Aruco code detected by the second camera during the true value data recording process and the yaw and pitch direction angle values, so as to calculate the 3D coordinates of the left and right pupils of the recorder in the camera coordinate system of the second camera in real time, and then the true value gaze angles of the recorder in the pitch and yaw directions during the gaze process can be solved in real time.

Description

Technical Field

[0001] This invention relates to the field of gaze estimation technology, and in particular to a gaze estimation truth data recording device and method. Background Technology

[0002] Gaze estimation, also known as eye tracking, has been widely applied in areas such as distraction monitoring in automotive safety and driver assistance systems, human-computer interaction, psychological research, ophthalmic diagnosis, VR technology, and gaming. Currently, an increasing number of researchers and engineers in both academic and practical engineering fields are focusing on gaze estimation algorithm design. However, the development of algorithms and models is based on data, and gaze estimation algorithm design often uses image data as ground truth. Factors such as image resolution, size, richness, recording environment diversity, angular range, and recording errors significantly influence and determine the performance of the gaze estimation algorithms or models ultimately designed by researchers and engineers.

[0003] Based on the aforementioned objective factors, researchers have a strong and clear demand for high-quality ground truth data. However, open-source gaze estimation ground truth data cannot meet the specific needs of researchers. Taking the MPIIGaze, EYEDIAP, Columbia, and ETH-XGaze open-source datasets well-known to those skilled in the art as examples, almost all of these open-source datasets are RGB images. Although some datasets contain information such as head pose, facial landmarks, and 3D coordinates of gaze points, these datasets are all 2D planar images. Landmarks on the face, such as the corners of the eyes, only have pixel coordinates and no actual 3D spatial coordinates. The size and resolution of the images in these open-source datasets are fixed, and the gaze angle range of the recorder may not meet the needs of algorithm development. Taking the MPIIFaceGaze open-source dataset as an example, this dataset obtains the angle values ​​of head pose based on a simulated head model. Considering that each model has its own errors, the magnitude of the angle error of the head pose provided by the open-source dataset is difficult to guarantee. These errors will inevitably have a negative impact on the development of gaze estimation algorithms. Therefore, before developing and designing gaze estimation algorithms, researchers mostly need to record gaze ground truth data specific to their application scenarios. Summary of the Invention

[0004] To address the deficiencies in the existing technology, the present invention provides a gaze estimation truth data recording device and a gaze estimation truth data recording method.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A gaze estimation truth data recording device, comprising:

[0007] A head-mounted device, worn by the truth data recorder, is used to fix the Aruco code to the recorder's head and move with the recorder's head.

[0008] Aruco codes, affixed to the head restraint, are used to determine the angle and position of the person recording the head posture.

[0009] A calibration plate, fixed in front of the recorder, provides the necessary gaze point for the recorder to focus on.

[0010] A tof camera, fixed on a calibration plate, is used to calibrate the average distance in 3D space from the center point (102) of the Aruco code to the pupil of the recorder when the recorder gazes at any point on the calibration plate, the coordinates of the center point of the Aruco code at the initial time (i.e. the Tth calibration time), and the angle values ​​pitch (0) and yaw (0) output in the pitch and yaw directions, as well as the 3D coordinates of the left and right pupils of the recorder;

[0011] The second camera module, fixed on the calibration plate, is used to capture and photograph the Aruco code and the facial image of the recorder when they gaze at the gaze point on the calibration plate.

[0012] This invention also provides a method for recording gaze estimation truth data, which includes the following steps:

[0013] S201: Use OpenCV to calibrate the TOF camera and the second camera, and obtain their intrinsic parameter matrix and distortion coefficients;

[0014] S202: When the recorder wears a head fixation device and stares at any point on the calibration board, the average distances D1 and D2 between the pupils of the left and right eyes and the center point of the Aruco code in 3D space are recorded using a TOF camera to calibrate the true data. At the same time, the coordinates of the center point of the Aruco code at the initial moment are obtained, and the angle values ​​of the Aruco code in the pitch and yaw directions at the initial moment, pitch(0) and yaw(0), and the 3D coordinates of the recorder's left and right pupils are recorded as calibration results.

[0015] S203: Given the fixed position of the second camera on the calibration plate, obtain the position of each gaze point on the calibration plate in the camera coordinate system of the second camera;

[0016] S204: The data recorder continuously wears a head fixation device and stares at the gaze point on the calibration plate in any head posture. The second camera captures the gaze image of the recorder and performs distortion correction on the image.

[0017] S205: Detect the rotation angle of the Aruco code in the pitch and yaw directions relative to the initial moment and the coordinate value of the center point of the Aruco code in the gaze image of the recorder. Combine D1 and D2 to calculate the 3D coordinates of the left and right pupils when the recorder gazes at the gaze point on the calibration plate.

[0018] S206: Calculate the gaze angles of the left and right eyes of the recorder as ground truth data based on the current coordinates of the left and right pupils and the 3D coordinates of the gaze point.

[0019] Furthermore: Before each recorder uses this invention to record gaze estimation true value data, they need to use a TOF camera to calibrate the average distances D1 and D2 in 3D space from the pupils of the recorder's left and right eyes to the center point of the Aruco code on the head fixation device, the coordinates of the center point of the Aruco code at the initial moment, the deflection angles of the Aruco code in the pitch and yaw directions at the initial moment (pitch(0) and yaw(0)), and the 3D coordinates of the recorder's left and right pupils in the camera coordinate system of the TOF camera at the initial moment. The steps are as follows:

[0020] S301: Recorder sets the number of times the calibration process will be recorded (T);

[0021] S302: The truth data recorder wears a head-fixing device, adjusts his / her head posture, and stares at any point on the calibration plate;

[0022] S303: The tof camera captures Aruco code images of the recorder's face and head fixation device, and uses the tof camera intrinsic parameter matrix and distortion coefficients calibrated by OpenCV to perform distortion correction on the entire image;

[0023] S304: Detect the Aruco code image and obtain the 3D coordinates of its center point;

[0024] S305: The pupil point detection model trained independently in this application is used to detect the facial image of the recorded subject. The 3D coordinate values ​​of the left and right pupils are calculated based on the pixel coordinates of the detected pupils. The calculation method is as follows:

[0025]

[0026]

[0027] Among them, X pupil and Y pupil It is the coordinate value of the pupil point in the TOF camera coordinate system, z pupil It is the pupil point depth value on the depth map captured by the ToF camera, u pupil and v pupil Here are the pixel coordinates of the pupil point, w and h are the width and height of the image captured by the TOF camera, and f is the pixel coordinates of the pupil point.x and f y It is the internal parameter of the ToF camera;

[0028] S306: Calculate the 3D distances d and e in the TOF camera coordinate system based on the 3D coordinates of the Aruco code center point and the left and right pupils. The calculation method is as follows:

[0029]

[0030]

[0031] Among them, X pupil_left and Y pupil_left The 3D coordinates of the left eye pupil point were calculated according to the method in S305, X. pupil_right and Y pupil_right It is the 3D coordinate of the right pupil point, z pupil_left and z pupil_right It is the depth value of the left and right pupil points, X aruco Y aruco Z aruco These are the 3D coordinates of the Aruco code center point in the TOF camera coordinate system obtained from S304;

[0032] S307: Determine whether the number of images captured during the tof camera calibration process is equal to the preset T value. If yes, proceed to S308; otherwise, return to S302, and the recorder repeats the calibration process.

[0033] S308: The mean values ​​of d and e in T calibrations are used as the average distances D1 and D2 in 3D space from the left and right pupil points of the recorder to the center point of the Aruco code. The calculation method is as follows:

[0034]

[0035]

[0036] S309: Output the coordinates of the center point obtained by the Aruco code detection during the Tth calibration, the deflection angles of the Aruco code in the pitch and yaw directions pitch(0) and yaw(0), and the 3D coordinates of the left and right pupils of the recorder as the calibration results at the initial moment.

[0037] Furthermore, the TOF camera is only used for the calibration stage before recording gaze ground truth data. The ground truth data is recorded only using the second camera, and the two cameras are not used simultaneously. Although the second camera is a 2D camera, the gaze estimation ground truth data is collected according to the method used in this invention, and the pupil landmarks of the recorder in the final data are 3D coordinates. Therefore, the 2D image collected by the second camera can be given certain depth information.

[0038] Furthermore: when the true data recorder measures the fixed position of the second camera on the calibration board as (camera X camera Y camera z When this happens, the coordinates of the gaze point in the calibration plate coordinate system can be converted into coordinates in the camera coordinate system of the second camera using the following calculation method.

[0039]

[0040] Among them, (P) x P y P z ) is the 3D coordinate value of a gaze point in the camera coordinate system of the second camera, s is the side length of the square on the calibration plate, num is the number of the gaze point arranged from left to right on the calibration plate, and col is the number of squares in each row on the calibration plate.

[0041] Further, in step S202, the Tth tof camera calibration is considered as the initial moment before the true data recording. Based on the calibration results at the initial moment, the initial angles δ and σ between the recorder's left and right pupils and the center point of the Aruco code in the pitch and yaw directions are calculated. The calculation method is as follows:

[0042]

[0043]

[0044]

[0045]

[0046] Among them, (X) a Y a Z a ) is the coordinate value of the center point of the Aruco code in the tof camera coordinate system obtained during the T-th calibration detection, (L x L y L z ) and (R x R y R z ) are the coordinates of the left and right pupil points of the recorder in the tof camera coordinate system at the T-th calibration time, respectively. left and δ right These are the initial angles σ and σ between the left and right pupil points of the recorder and the center point of the Aruco code in the yaw direction at the initial moment. left and σ rightThese are the initial angles between the left and right pupil points and the center point of the Aruco code in the pitch direction at the initial moment.

[0047] Furthermore: when the recorder wearing the head-fixing device begins to gaze at the gaze point on the calibration plate, if the head pose angles of the recorder detected on the Aruco code in the yaw and pitch directions are α and β, and the coordinates of the center point of the Aruco code in the camera coordinate system of the second camera are (X... c Y c Z c At this point, the coordinates of the left and right pupils of the recorder in the second camera coordinate system can be calculated using the following method:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] Among them, (X) left ,Y left Z left ) and (X right ,Y right Z right D1 and D2 are the coordinates of the left and right pupils of the recorder in the second camera coordinate system, respectively. D1 and D2 are the average distances in 3D space from the left and right pupils of the recorder to the center point of the Aruco code, respectively, obtained from the TOF camera calibration process. pitch(0) and yaw(0) are the deflection angles of the Aruco code detected in the pitch and yaw directions at the initial moment. δ left and δ right These are the initial angles σ and σ between the left and right pupil points of the recorder and the center point of the Aruco code in the yaw direction at the initial moment. left and σ right These are the initial angles between the left and right pupil points and the center point of the Aruco code in the pitch direction at the initial moment.

[0055] Furthermore: when the coordinates (X, Y, Z) of the left and right pupils in the camera coordinate system of the second camera are obtained... left ,Y left Z left ), (X right ,Y right Zright ) and the 3D coordinates (P) of the recorder's gaze point in the camera coordinate system of the second camera. x ,P y ,P z After that, the method for calculating the gaze truth angles θ and γ of the left and right eyes of the truth data recorder in the yaw and pitch directions is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] in ， θ left and θ right These are the gaze angles of the left and right eyes of the recorder in the yaw direction, respectively. left and γ right These are the true angles of the recorder's left and right eyes in the pitch direction.

[0061] This invention uses Aruco codes to detect the head pose angle of the recorder in ground truth data. After multiple experiments, it was found that the deviation of Aruco codes is within 3° in the range of 0° to 50° of the recorder's head pose angle. Therefore, it avoids the large angle error in the gaze estimation ground truth data caused by the use of simulated head models or head pose detection models in open source datasets.

[0062] This invention specifically relates to the field of gaze estimation ground-truth data recording technology, and proposes a method for recording gaze estimation ground-truth data. The method employs a gaze estimation ground-truth data recording device, which includes: a head fixation device, Aruco codes pasted on the head fixation device, a calibration plate marked with gaze points, a TOF camera, and a second camera. The gaze estimation true data recording method of the present invention uses a TOF camera to calibrate the average 3D distance from the center point of the Aruco code on the head fixation device worn by the recorder to the left and right pupils, the coordinate values ​​of the center point detected by the Aruco code at the initial moment, the deflection angle values ​​detected by the Aruco code in the pitch and yaw directions, and the 3D coordinates of the recorder's left and right pupils at the initial moment. Combined with the angle values ​​of the recorder's head posture in the yaw and pitch directions detected by the Aruco code captured by the second camera during the true data recording process, as well as the coordinate values ​​of the center point of the Aruco code, the 3D coordinates of the recorder's left and right pupils in the camera coordinate system of the second camera can be calculated in real time. Then, the true gaze angle in the pitch and yaw directions of the recorder can be solved in real time.

[0063] Compared with existing technologies, the advantages of this invention are: the method used is simple and effective, and the device used is easy to operate. For the recorder, there is no need to purchase high-cost precision instruments or build a harsh experimental environment; high-precision true data can be obtained with only a small economic cost, significantly saving the cost of recording true data and improving the efficiency of true data recording. The selection of the second camera used in this invention is determined by the recorder according to their true data recording needs. Therefore, this invention can meet different recording needs, giving the recorder a great deal of autonomy in data recording and has very high practical application value. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the gaze estimation truth data recording device of the present invention. In the figure:

[0065] 101: Head fixation device; 102: Aruco code; 103: Calibration plate; 104: TOF camera; 105: Second camera; 106: Truth data recorder; The dashed line indicates the line of sight of the truth data recorder when gazing at the point on the calibration plate.

[0066] Figure 2 This is the overall system flowchart of the present invention;

[0067] Figure 3This is a flowchart illustrating the calibration results using a TOF camera to determine the average 3D distance from the center point of the Aruco code to the left and right pupils of the recorder, the coordinates of the center point detected on the Aruco code at the initial moment, the deflection angles in the pitch and yaw directions, and the 3D coordinates of the left and right pupils of the recorder at the initial moment. Detailed Implementation

[0068] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0069] Example 1:

[0070] As attached Figure 1 The schematic diagram shown is of the gaze estimation truth data recording device of the present invention. The device includes: a head fixation device 101, worn by the truth data recorder 106, for fixing an Aruco code 102 to the head of the truth data recorder 106 and moving with the recorder's head; the Aruco code 102, affixed to the head fixation device 101, for determining the angle and position of the recorder's head posture; a calibration plate 103, fixed in front of the recorder, providing the gaze point required for the recorder's gaze; and a TOF camera 104, fixed to the calibration plate. On calibration plate 103, the average distance in 3D space from the center point 102 of the Aruco code to the pupil of the recorder 106 when the recorder 106 gazes at any gaze point on calibration plate 103, the coordinates of the center point of the Aruco code 102 at the initial moment, the angle values ​​output in the pitch and yaw directions, and the 3D coordinates of the left and right pupils of the recorder at the initial moment are used; the second camera module 105 is fixed on calibration plate 103 and is used to acquire and photograph the Aruco code 102 and the recorder's facial image when the recorder gazes at the gaze point on the calibration plate.

[0071] The specific principle and characteristics of the gaze estimation true value data recording device described above are as follows:

[0072] (1) Head fixing device 101: connected to the head of the recorder 106 and fixed to the head of the recorder 106, with Aruco code 102 pasted on it, so that the Aruco code 102 can rotate synchronously with the head of the recorder 106 during the recording of the true data.

[0073] (2) Aruco code 102: It is pasted on the head fixation device 101. Because it moves synchronously with the head of the recorder 106 under the action of the head fixation device 101, the rotation angles of the yaw and pitch directions detected on the Aruco code 102 can be regarded as the angles of the current head posture of the recorder 106. Therefore, it is convenient to detect the head posture of the recorder 106 in real time.

[0074] (3) Calibration plate 103: Fixed in front of the recorder 106, used to provide the gaze points required for recording true data. Before the recorder uses this invention to record gaze estimation true data, the gaze points on the calibration plate need to be numbered. The numbering rule is determined by the recorder and can be from left to right or from top to bottom. The distance between the calibration plate 103 and the recorder 106 is set by the recorder 106 according to the true data recording requirements. The calibration plate 103 is marked with multiple squares by intersecting horizontal and vertical straight lines, and the intersection of the horizontal and vertical lines is the gaze point. The calibration plate is full of squares, and the side length of each square is determined by the angle step size of the gaze estimation true data to be recorded. If the angle step size of the two consecutive gazes recorded by the recorder is smaller, the side length of the square on the calibration plate is smaller, and vice versa. The size of the calibration plate and the side length of the squares are determined by the recorder according to the specific recording task. Specifically, the side length S of the square grid on the calibration plate can be calculated as follows:

[0075] s=L*tanλ

[0076] Where L is the distance between the calibration board and the recorder, and λ is the step size of the gaze angle defined by the recorder;

[0077] (4) TOF camera 104: Fixed on calibration plate 103, position is not limited, used to calibrate the average distance in 3D space from the center point 102 of the Aruco code to the pupil of the recorder 106 when the recorder 106 gazes at any gaze point on calibration plate 103, the coordinates of the center point of the Aruco code at the initial moment, the angle values ​​output in the pitch and yaw directions, and the 3D coordinates of the left and right pupils of the recorder at the initial moment, etc. The calibration results are positively correlated with the accuracy of TOF camera 104. The higher the accuracy of TOF camera 104, the more accurate the calibration results. In this embodiment, a TOF camera with a resolution of 300,000 pixels and an accuracy of 1% (1m) is used as an example. When using this invention to record gaze estimation true value data, a TOF camera with higher accuracy can obtain more accurate true value data.

[0078] (5) Second camera 105: Fixed on calibration plate 103, used to acquire and capture Aruco code 102 and facial image when the recorder 106 gazes at the gaze point on the calibration plate. Before recording ground truth data using this invention, the recorder 106 needs to measure the fixed position of the second camera 105 on calibration plate 103 and the height of the lens of the second camera 105 relative to calibration plate 103 to determine the coordinate value of the second camera 105 in the calibration plate coordinate system, so as to facilitate the conversion of the coordinate value of the gaze point in the calibration plate coordinate system into the coordinate value in the camera coordinate system of the second camera 105. At the same time, the optimal fixed position of the second camera 105 is the center of the calibration plate. The gaze image of the recorder acquired by the second camera 105 is the ground truth image data, and the gaze angle of the recorder calculated on the ground truth image is the ground truth angle data. The selection of the second camera 105 used in this invention is determined by the recorder based on the recording task. Commonly used 2D cameras of various resolutions, such as RGB and IR, are all suitable for this invention. If the recorder needs to record a 2-megapixel IR gaze ground truth image, then the second camera 105 will be a 2-megapixel IR camera.

[0079] Example 2:

[0080] A method for recording gaze estimation true value data, comprising the following steps:

[0081] S201: Please refer to Figure 2 Before recording gaze estimation true data using this invention, the recorder needs to use OpenCV to calibrate the TOF camera and the second camera to obtain their intrinsic parameter matrix and distortion coefficients.

[0082] S202: Please refer to Figure 2 and Figure 3 Before officially starting the recording of truth data, each participant in the truth data recording process needs to follow the instructions. Figure 3 The TOF camera calibration process shown measures the average 3D distances D1 and D2 from the center point of the Aruco code on the head-mounted device to its left and right pupils, the initial coordinates of the Aruco code center point, the angle values ​​output in the pitch and yaw directions, and the initial 3D coordinates of the recorder's left and right pupils. This calibration process requires the use of the pre-trained pupil landmark detection model provided by this invention. The calculation method for D1 and D2 is as follows:

[0083]

[0084]

[0085] Where T is the number of calibrations, d i It is the 3D distance from the left pupil of the i-th timer recorder to the center point of the Aruco code, e jIt is the 3D distance from the right pupil of the recorder to the center point of the Aruco code during the j-th calibration;

[0086] S203: Given the fixed position of the second camera on the calibration plate (camera) X camera Y camera Z This allows us to obtain the positions of each gaze point on the calibration plate within the camera coordinate system of the second camera. The position of the gaze point in the camera coordinate system of the second camera (P...) x P y P z Calculated as follows:

[0087]

[0088] Where S is the side length of the square on the calibration plate, num is the number of the gaze point arranged from left to right on the calibration plate, and col is the number of squares in each row on the calibration plate.

[0089] S204: The recorder of the true data wears a head fixation device and stares at the gaze point on the calibration plate in any head posture. The second camera captures the recorder's gaze image and performs distortion correction on the image.

[0090] S205: Detect the rotation angles of the Aruco code in the pitch and yaw directions relative to the initial moment and the coordinates of the Aruco code center point in the image. Combine this with D1 and D2 to calculate the 3D coordinates of the left and right pupil points in the camera coordinate system of the second camera when the recorder is gazing at the gaze point on the calibration board. The calculation method is as follows:

[0091] First, the T-th tof camera calibration is considered as the initial moment before the true data recording. Based on the calibration results at the initial moment, the initial angles δ and σ between the recorder's left and right pupils and the center point of the Aruco code in the pitch and yaw directions are calculated. The calculation method is as follows:

[0092]

[0093]

[0094]

[0095]

[0096] Among them, (X) a ,Y a Z a ) is the coordinate value of the center point of the Aruco code in the tof camera coordinate system obtained during the T-th calibration detection, (L x ,L y,L z ) and (R x ,R y ,R z ) are the coordinates of the left and right pupil points of the recorder in the tof camera coordinate system at the T-th calibration time, respectively. left and δ right These are the initial angles σ and σ between the left and right pupil points of the recorder and the center point of the Aruco code in the yaw direction at the initial moment. left and σ right These are the initial angles between the left and right pupil points and the center point of the Aruco code in the pitch direction at the initial moment.

[0097] Then, by combining the above calculation results with the current rotation angle of the Aruco code, the 3D coordinates of the left and right pupils of the recorder in the camera coordinate system of the second camera can be calculated in real time while the recorder is gazing.

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] Where α and β are the head posture rotation angles of the recorder in the yaw and pitch directions, detected by the Aruco code in the currently captured recorder's gaze image, (X left ,Y left Z left ) and (X right ,Y right Z right D1 and D2 are the coordinates of the left and right pupils of the recorder in the second camera coordinate system, respectively. D1 and D2 are the average distances in 3D space from the left and right pupils of the recorder to the center point of the Aruco code, respectively, obtained from the TOF camera calibration process. pitch(0) and yaw(0) are the deflection angles of the Aruco code detected in the pitch and yaw directions at the initial moment. δ left and δ right These are the initial angles σ and σ between the left and right pupil points of the recorder and the center point of the Aruco code in the yaw direction at the initial moment. left and σ right These are the initial angles between the left and right pupil points and the center point of the Aruco code at the initial moment in the pitch direction, respectively.c ,Y c Z c () is the coordinate of the center point of the Aruco code in the camera coordinate system of the second camera;

[0105] S206: Based on pupil point (X) left ,Y left Z left ), (X right ,Y right Z right ) and the 3D coordinates of the gaze point (P) x ,P y ,P z The left and right eye gaze angles are calculated as ground truth data. The detailed calculation method is as follows:

[0106]

[0107]

[0108]

[0109]

[0110] Where, θ left and θ right These are the gaze angles of the left and right eyes of the recorder in the yaw direction, respectively. left and γ right These are the true angles of the recorder's left and right eyes in the pitch direction.

[0111] The pupil landmark detection model used in this invention is trained using commonly used RGB and IR images with resolutions ranging from 300,000 pixels to 5 million pixels, ensuring effective pupil landmark detection for specific recording data. In this embodiment, the pupil landmark detection model is trained using the open-source convolutional neural network model YOLOv5 with the aforementioned RGB and IR images of various resolutions. The training set of this model consists of tens of thousands of facial images. Through extensive data annotation and training on these facial images, the pupil landmark detection model of this invention achieves good detection performance and accuracy, and can be widely applied to images of different resolutions captured by various common 2D cameras.

[0112] Example 3:

[0113] This embodiment is a specific implementation method to illustrate the above-mentioned method for recording true gaze estimation data:

[0114] The calibration board used for this ground truth data recording has 18 rows and 11 columns, with 170 gaze points distributed across each square, each with a side length of 200 mm. The fixed position of the second camera on the calibration board is known (1686, 1036, 41), and the distance between the recorder and the calibration board is between 600 mm and 900 mm. The intrinsic parameters and distortion coefficients of the TOF camera and the second camera are calibrated using OpenCV's camera calibration tools.

[0115] (1) Participants in recording gaze estimation true value data wore head-fixation devices and gazed at the gaze point on the calibration board in any head posture, according to... Figure 3 The TOF camera calibration procedure shown measures the average 3D distance from the center point of the Aruco code to the left and right pupils, the initial coordinates of the Aruco code center point, the angles of deflection in the pitch and yaw directions, and the initial 3D coordinates of the left and right pupils of the recorder. Assuming 15 calibration attempts are performed, 15 calibration images are acquired, and the above calibration results are calculated and recorded.

[0116] (2) After calibration, the average 3D distance from the recorder's left pupil to the center point of the Aruco code was measured to be 255.81 mm, and the average 3D distance from the right pupil to the center point of the Aruco code was measured to be 255.73 mm. The initial deflection angles of the Aruco code in the pitch and yaw directions were 5.18° and 1.87°, respectively. The initial angles between the recorder's left and right pupils and the center point of the Aruco code in the pitch and yaw directions were:

[0117] Left eye: pitch angle is 77.98°, yaw angle is -28.19°.

[0118] Right eye: pitch angle is 77.89°, yaw angle is 37.95°.

[0119] (3) The recorder wears a head fixation device and stares continuously at the gaze point on the calibration plate. The second camera takes pictures of each gaze moment of the recorder to obtain the true image of the recorder staring at each gaze point.

[0120] The following are the ground truth angle values ​​calculated from the gaze estimation ground truth images captured when the recorder gazes at gaze points 49, 61, 74, 99, and 129:

[0121]

[0122] It should be noted that the above-described specific embodiments are merely exemplary and are intended to enable those skilled in the art to better understand this patent. They should not be construed as limiting the scope of protection of this patent. Any equivalent changes or modifications made in accordance with the spirit disclosed in this patent shall fall within the scope of protection of this patent.

Claims

1. A gaze estimation truth data recording device, characterized in that, include: A head-fixing device (101), worn by the truth data recorder (106), is used to fix the Aruco code (102) to the head of the truth data recorder (106) and move with the head of the recorder (106); Aruco code (102), affixed to the head fixation device (101), is used to determine the angle and position of the recorder's (106) head posture; A calibration plate (103) is fixed in front of the recorder (106) to provide the gaze point required for the recorder (106) to gaze; A tof camera (104) is fixed on a calibration plate (103) and is used to calibrate the average distance in 3D space from the center point (102) of the Aruco code to the pupil of the recorder (106) when the recorder (106) gazes at any point on the calibration plate (103), the coordinates of the center point of the Aruco code at the initial moment, and the angle values ​​pitch (0) and yaw (0) output in the pitch and yaw directions, as well as the 3D coordinates of the left and right pupils of the recorder. The second camera (105), fixed on the calibration plate (103), is used to acquire and capture Aruco code (102) and facial images of the recorder (106) as he gazes at the gaze point on the calibration plate; The head fixing device (101) is connected to the head of the truth data recorder (106), fixed on the head of the recorder (106), and Aruco code (102) is pasted on it. During the recording of truth data, the Aruco code (102) and the head of the recorder (106) rotate synchronously. The Aruco code (102): The head of the recorder (106) moves synchronously with the head of the recorder (106) under the action of the head fixation device (101). Therefore, the rotation angles of the yaw and pitch directions detected on the Aruco code (102) are considered as the angles of the current head posture of the recorder (106) in order to detect the head posture and the 3D coordinates of the left and right pupils of the recorder (106) in real time. The calibration board (103): The distance between the calibration board (103) and the recorder (106) is set by the recorder (106) according to the requirements of recording true data. The calibration board (103) is marked with multiple squares by intersecting horizontal and vertical straight lines. The intersection of the horizontal and vertical straight lines is the gaze point. The side length of the square is determined by the requirements of recording true data. The smaller the gaze angle step of the recorder (106), the smaller the side length of the square, and vice versa. The side length of the square on the calibration board (103) It can be calculated as follows: ； in, It is the distance between the calibration board and the recorder. It is the step length of the gaze angle defined by the recorder; The calibration results of the TOF camera (104) are positively correlated with the accuracy of the TOF camera. The second camera (105): The fixed position of the second camera (105) is the center of the calibration plate (103). The gaze image of the recorder acquired by the second camera (105) is the true image data, and the gaze angle of the recorder calculated on the true image is the true angle data.

2. A method for recording true value data for gaze estimation, characterized in that, The specific steps of using the gaze estimation truth data recording device as described in claim 1 are as follows: S201: Use OpenCV to calibrate the TOF camera and the second camera, and obtain their intrinsic parameter matrix and distortion coefficients; S202: The average distance in 3D space from the pupils of the left and right eyes to the center point of the Aruco code when the recorder wearing a head-mounted device gazes at any point on the calibration board using a ToF camera to calibrate true data. , At the same time, the coordinates of the center point of the Aruco code at the initial moment are obtained, the angle values ​​of the Aruco code in the pitch and yaw directions at the initial moment are recorded as pitch(0) and yaw(0), and the 3D coordinate calibration results of the left and right pupils of the recorder are obtained. S203: Given the fixed position of the second camera on the calibration plate, obtain the position of each gaze point on the calibration plate in the camera coordinate system of the second camera; S204: The data recorder continuously wears a head fixation device and stares at the gaze point on the calibration plate in any head posture. The second camera captures the gaze image of the recorder and performs distortion correction on the image. S205: Detect the rotation angles of the Aruco code in the pitch and yaw directions relative to the initial moment and the coordinates of the Aruco code center point in the image viewed by the data recorder, combined with... , Calculate the 3D coordinates of the left and right pupils when the recorder is looking at the gaze point on the calibration board; S206: Calculate the gaze angles of the left and right eyes of the recorder as true data based on the current coordinates of the left and right pupils and the 3D coordinates of the gaze point; The TOF camera is only used for the calibration process before recording the gaze truth data. The truth data is recorded only using the second camera, and the two cameras are not used simultaneously.

3. The gaze estimation true value data recording method according to claim 2, characterized in that: In step S202, a TOF camera is used to calibrate the average 3D spatial distance from the pupils of the recorder's left and right eyes to the center point of the Aruco code on the head-mounted device. and The initial coordinates of the Aruco code center point, the initial angle values ​​of pitch and yaw directions, pitch(0) and yaw(0), and the 3D coordinate calibration results of the recorder's left and right pupils at the initial time are determined by the following steps: S301: Recorder sets the number of times T is collected during the calibration process; S302: The recorder of the truth data wears a head fixation device, adjusts his head posture, and stares at any point on the calibration plate; S303: The tof camera captures Aruco code images of the recorder's face and head fixation device, and uses the tof camera intrinsic parameter matrix and distortion coefficients calibrated by OpenCV to perform distortion correction on the entire image; S304: Detect the Aruco code image and obtain the 3D coordinates of its center point; S305: Use the pupil point detection model trained independently by this application to detect the face image of the recorded person, and calculate its 3D coordinate value based on the pixel coordinates of the left and right pupils obtained from the detection; S306: Calculate the 3D distance in the TOF camera coordinate system based on the 3D coordinates of the Aruco code center point and the left and right pupils. , ; S307: Determine whether the number of images captured during the tof camera calibration process is equal to the preset T value. If yes, proceed to S308; otherwise, return to S302, and the recorder repeats the calibration process. S308: Statistical calibration of T times and The mean of the data is used as the average distance in 3D space from the left and right pupils of the recorder to the center of the Aruco code. , The calculation method is as follows: ； in, To calibrate the number of times, It is the first The 3D distance from the left pupil of the recorder to the center point of the Aruco code was calibrated. It is the first The 3D distance from the right pupil of the recorder to the center point of the Aruco code during the calibration process; S309: Output the coordinates of the center point obtained by the Aruco code detection at the Tth calibration time, the deflection angles of the Aruco code in the pitch and yaw directions pitch(0) and yaw(0), and the 3D coordinate information of the left and right pupils of the recorder as the calibration results at the initial time.

4. The gaze estimation true value data recording method according to claim 3, characterized in that: In step S305, the 3D coordinate values ​​of the left and right pupils of the recorder are calculated as follows: ； in, and These are the coordinates of the pupil point in the TOF camera coordinate system. It is the pupil depth value extracted from the depth map captured by the ToF camera. and These are the pixel coordinates of the pupil point. and These are the width and height of the image captured by the ToF camera. and It's an internal parameter of the ToF camera.

5. The gaze estimation true value data recording method according to claim 3, characterized in that: In step S306, 3D distance , The calculation method is as follows: ； in, and The 3D coordinates of the left eye pupil were calculated according to the method in S305. and These are the 3D coordinates of the right pupil point. and These are the depth values ​​of the left and right pupil points. , , It is the 3D coordinate value of the center point of the Aruco code obtained in S304 in the tof camera coordinate system.

6. The method for recording gaze estimation truth data according to claim 2 or 3, characterized in that: When the true data recorder measures the fixed position of the second camera on the calibration board as ( , , When this happens, the coordinates of the gaze point in the calibration plate coordinate system are converted to coordinates in the camera coordinate system of the second camera using the following calculation method: ； in,( , , ) represents the 3D coordinates of a gaze point in the camera coordinate system of the second camera. It is the side length of the square on the calibration board. It is the number of the gaze point arranged from left to right on the calibration plate. It is the number of squares in each row of the calibration board.

7. The method for recording gaze estimation true value data according to claim 2 or 3, characterized in that: In step S202, the T-th tof camera calibration is considered as the initial moment before the true data recording. Based on the calibration results at the initial moment, the initial angles between the recorder's left and right pupils and the center point of the Aruco code in the pitch and yaw directions are calculated. , The calculation method is as follows: ； in,( , , ) is the coordinate value of the center point of the Aruco code in the tof camera coordinate system obtained during the T-th calibration. , , )and( , , These are the coordinates of the left and right pupil points of the recorder in the tof camera coordinate system during the T-th calibration. and These are the initial angles in the yaw direction between the left and right pupil points of the recorder and the center point of the Aruco code at the initial moment. and These are the initial angles between the left and right pupil points and the center point of the Aruco code in the pitch direction at the initial moment.

8. The method for recording gaze estimation true value data according to claim 2 or 3, characterized in that: When the recorder wearing the head restraint device begins to gaze at the gaze point on the calibration board, if the head posture angles of the recorder detected by the Aruco code in the yaw and pitch directions are... and The coordinates of the center point of the Aruco code in the camera coordinate system of the second camera are ( , , At this time, the coordinates of the left and right pupils of the recorder in the second camera coordinate system are calculated as follows: ； in,( , , )and( , , These are the coordinates of the left and right pupils of the recorder in the second camera coordinate system. and These are the average distances in 3D space from the left and right pupils of the recorder to the center point of the Aruco code, obtained during the TOF camera calibration process. pitch(0) and yaw(0) are the deflection angles detected by the Aruco code in the pitch and yaw directions at the initial moment. and These are the initial angles in the yaw direction between the left and right pupil points of the recorder and the center point of the Aruco code at the initial moment. and These are the initial angles between the left and right pupil points and the center point of the Aruco code in the pitch direction at the initial moment.

9. The method for recording true gaze estimation data according to claim 8, characterized in that: When the coordinates of the left and right pupils in the camera coordinate system of the second camera are obtained ( , , ), ( , , ) and the 3D coordinates of the recorder's gaze point in the camera coordinate system of the second camera ( , , After that, the gaze angles of the left and right eyes of the truth data recorder in the yaw and pitch directions. and The calculation method is as follows: ； in, and These are the gaze angles of the left and right eyes of the recorder in the yaw direction, respectively, representing the ground truth values. and These are the true angles of the recorder's left and right eyes in the pitch direction.

Images