Human-computer interaction sight line estimation system fusing eye movement and head posture

Abstract

The invention relates to the technical field of human-computer interaction, in particular to a human-computer interaction sight line estimation system fusing eye movement and head posture, which comprises the following steps: acquiring an eye image sequence and head posture sensor data, detecting image coordinates of canthus points and pupil center points, and analyzing to obtain a head rotation angle. Performing three-dimensional reconstruction by using the eye corner point image coordinates and the internal reference of the camera to obtain eye corner point three-dimensional coordinates under the head coordinate system and fitting the initial position of the eyeball center. And converting to a camera and a screen coordinate system according to the attitude data to obtain an eyeball center estimation position under the screen coordinate system. Combining the pupil center point image coordinate, the camera internal reference and the position to calculate a sight direction vector, and intersecting with a screen plane to obtain a drop point coordinate; three-dimensional modeling of a head coordinate system and conversion of multiple coordinate systems along with postures are achieved, and head movement and individual anatomical differences are adapted. According to the invention, the spatial precision and dynamic adaptability of sight line estimation are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction gaze estimation system that integrates eye tracking and head posture. Background Technology

[0002] In human-computer interaction gaze estimation applications, existing technical solutions mainly infer the gaze direction through the two-dimensional features of eye images. Typically, the image coordinates of the pupil center point are directly extracted and combined with a preset fixed mapping model within the camera to map it onto the screen coordinate system to obtain the gaze point, thereby meeting the need to perceive the user's gaze position in the interaction.

[0003] Existing solutions have several drawbacks: they fail to integrate head posture information, assuming the eye center is fixed in the camera or screen coordinate system, ignoring changes in the relative spatial position of the eye caused by head translation and rotation, making it easy for the calculated gaze point to deviate from the actual gaze position when the head moves; they lack 3D spatial modeling, relying solely on the 2D pupil position to calculate the gaze, and fail to reconstruct the initial position of the eye center in the head coordinate system using features such as the corner of the eye, making it difficult to adapt to the differences in the anatomical structure of the eyeball within the head of different individuals, resulting in estimation accuracy being constrained by individual differences.

[0004] After collecting user eye image sequences to extract the coordinate sets of the corner of the eye and the center of the pupil, and analyzing the head posture sensor data to obtain the head rotation angle, it is necessary to use the corner of the eye coordinates and the camera intrinsic parameters to reconstruct the three-dimensional coordinates of the corner of the eye in the head coordinate system to fit the initial position of the eyeball center in the head coordinate system. Then, based on the posture data, this position is transformed to the camera and screen coordinate system. The gaze direction vector is calculated by combining the pupil coordinates and the camera intrinsic parameters and the intersection point with the screen to solve the problem of gaze estimation deviation caused by head movement and individual differences. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a human-computer interaction gaze estimation system that integrates eye tracking and head posture.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a human-computer interaction gaze estimation system that integrates eye tracking and head posture, comprising: The data acquisition and analysis module acquires user eye image sequences and head posture sensor data, performs feature point detection on the eye image sequences, extracts the image coordinate set of the corner of the eye and the center of the pupil, and analyzes the head posture sensor data to obtain posture data representing the head rotation angle. The three-dimensional coordinate calculation module establishes the transformation relationship between the head coordinate system, the camera coordinate system, and the screen coordinate system. Using the image coordinate set of the corner of the eye point and the camera intrinsic parameters, it calculates the three-dimensional spatial coordinates of the corner of the eye point in the head coordinate system through a three-dimensional reconstruction method. The center positioning module fits the initial position of the eyeball center in the head coordinate system based on the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system. According to the posture data, the initial position of the eyeball center is transformed from the head coordinate system to the camera coordinate system, and then transformed to the screen coordinate system according to the relative position relationship between the camera and the screen, so as to obtain the estimated position of the eyeball center in the screen coordinate system. The gaze point estimation module uses the image coordinate set of the pupil center point, camera intrinsic parameters, and the estimated position of the eyeball center in the screen coordinate system to calculate the gaze direction vector. The intersection of the gaze direction vector with the screen plane is then used to obtain the coordinates of the gaze point on the screen.

[0007] As a further aspect of the present invention, the step of performing feature point detection on the eye image sequence to extract the image coordinate set of the corner of the eye and the center of the pupil specifically includes: A face detection algorithm is used to locate each frame of the eye image sequence, and the region of interest containing the eyes is extracted. For each frame of the region of interest image, the image coordinates of the inner canthus of the left eye, the outer canthus of the left eye, the inner canthus of the right eye, and the outer canthus of the right eye are located using the gray-scale integral projection method or the shape model method. For each frame of the region of interest image, edge detection and ellipse fitting methods are used to locate the image coordinates of the center of the left pupil and the center of the right pupil, respectively. The image coordinates of the inner corner of the left eye, the outer corner of the left eye, the inner corner of the right eye, the outer corner of the right eye, the center of the left pupil, and the center of the right pupil are stored in chronological order to form the image coordinate set of the corner of the eye and the center of the pupil.

[0008] As a further aspect of the present invention, the step of using the image coordinate set of the corner of the eye point and camera intrinsic parameters to calculate the three-dimensional spatial coordinates of the corner of the eye point in the head coordinate system through a three-dimensional reconstruction method specifically includes: Based on the camera pinhole model, establish the equation relating the image coordinates of the corner of the eye to its projection coordinates in the three-dimensional space of the camera coordinate system; Select at least two frames of eye images at different times and obtain the pose data at the corresponding times; use the pose data to calculate the relative rotation and translation transformation of the head coordinate system between the two frames; Substitute the image coordinates of the same eye corner point in two frames, the camera intrinsic parameters, and the relative rotation and translation transformations into the projection relationship equation to construct an overdetermined system of equations. Solve the overdetermined system of equations to obtain the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system.

[0009] As a further aspect of the present invention, the step of fitting the initial position of the eyeball center in the head coordinate system based on the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system specifically includes: Based on prior knowledge of human anatomy, a relative spatial position constraint model is established between the center of the eyeball and multiple corner points of the eye. Substitute the calculated three-dimensional spatial coordinates of the inner and outer corners of the left eye in the head coordinate system into the relative spatial position constraint model, and solve the three-dimensional spatial coordinates of the center of the left eyeball in the head coordinate system by the least squares method. Substitute the calculated three-dimensional spatial coordinates of the inner and outer corners of the right eye in the head coordinate system into the relative spatial position constraint model, and solve the three-dimensional spatial coordinates of the center of the right eyeball in the head coordinate system by the least squares method. The three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball together constitute the initial position of the eyeball center in the head coordinate system.

[0010] As a further aspect of the present invention, the step of transforming the initial position of the eyeball center from the head coordinate system to the camera coordinate system based on the posture data specifically includes: Based on the pitch angle, yaw angle, and roll angle contained in the attitude data, construct the rotation matrix from the head coordinate system to the camera coordinate system; Based on the rotation matrix and the preset translation vector of the head coordinate system origin relative to the camera coordinate system, construct a homogeneous transformation matrix; Using the homogeneous transformation matrix, the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the head coordinate system are transformed respectively to obtain the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the camera coordinate system.

[0011] As a further aspect of the present invention, the step of transforming the relative position of the camera and the screen to the screen coordinate system to obtain the estimated position of the eyeball center in the screen coordinate system specifically includes: The rotation and translation relationship between the camera coordinate system and the screen coordinate system is obtained through pre-calibration; Using the aforementioned rotation and translation relationship, the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the camera coordinate system are transformed to the screen coordinate system to obtain the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the screen coordinate system. The average of the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the screen coordinate system is used as the reference position of the user's head center in the screen coordinate system. The positions of the center of the left eyeball and the center of the right eyeball are then finely adjusted in combination with the interpupillary distance to finally obtain the estimated position of the eyeball center in the screen coordinate system for line of sight calculation.

[0012] As a further aspect of the present invention, the step of calculating the gaze direction vector using the image coordinate set of the pupil center point, camera intrinsic parameters, and the estimated position of the eyeball center in the screen coordinate system specifically includes: Based on the image coordinates of the pupil center point and the camera intrinsic parameters, the direction of the projection ray of the pupil center point in the camera coordinate system is obtained by back projection. Based on the estimated position of the eyeball center in the screen coordinate system and the relative position of the camera and the screen, the three-dimensional coordinates of the eyeball center in the camera coordinate system are deduced. In the camera coordinate system, a spatial straight line is formed by connecting the three-dimensional coordinates of the eyeball center with a point on the projection ray; According to the principle of corneal reflection, the direction of the spatial straight line, after being corrected by the refraction of the corneal spherical surface, is the true direction of vision. By solving the geometric constraints that allow the spatial straight line to point to the estimated position of the eyeball center in the screen coordinate system after passing through a specific refractive model of the corneal sphere, the expression of the gaze direction vector in the camera coordinate system is calculated.

[0013] As a further aspect of the present invention, the step of solving the geometric constraints that cause the spatial straight line to point to the estimated position of the eyeball center in the screen coordinate system after passing through a specific refractive model of the corneal sphere, and calculating the expression of the gaze direction vector in the camera coordinate system, specifically includes: A corneal spherical refractive model is established with the corneal surface as an ideal sphere. The corneal spherical refractive model includes corneal radius of curvature and refractive index parameters. In the camera coordinate system, the spatial straight line connecting the three-dimensional coordinates of the eyeball center and a point on the projection ray is extended to the spherical surface of the specific refractive model of the cornea. According to Snell's law of refraction, the normal vector of the spatial line at the incident point on the specific refractive model sphere of the cornea is calculated, and the exit direction of the light after refraction by the cornea is calculated based on the refractive index parameter. Construct geometric constraints that require the backward extension of the outgoing light rays after refraction through the cornea to pass through the three-dimensional coordinates of the camera coordinate system corresponding to the estimated position of the eyeball center in the screen coordinate system. A system of nonlinear equations is established based on the geometric constraints, where the unknowns of the nonlinear equations are the specific points on the projection ray of the spatial line. The nonlinear equations are solved using an iterative optimization algorithm to obtain the direction of the spatial line that satisfies the geometric constraints. The direction vector of the spatial line obtained by the solution is normalized to obtain the expression of the line of sight direction vector in the camera coordinate system.

[0014] As a further aspect of the present invention, the step of intersecting the gaze direction vector with the screen plane to obtain the coordinates of the gaze point on the screen specifically includes: Obtain the plane equation of the screen plane in the camera coordinate system; Solve the equation of the line containing the calculated line direction vector and the equation of the screen plane simultaneously. The coordinates of the intersection point between the line and the plane are calculated, and these coordinates are the physical intersection point between the line of sight optical axis and the screen. The coordinates of the physical intersection point are transformed into the screen pixel coordinate system with the top left corner of the screen as the origin to obtain the final coordinates of the landing point of the line of sight on the screen.

[0015] As a further aspect of the present invention, after acquiring the user's eye image sequence and head posture sensor data, the method further includes a data synchronization and timestamp alignment step: A timestamp of the image acquisition time is appended to each frame of the eye image sequence; A timestamp of the sensor acquisition time is appended to each data packet of the head posture sensor data; A time alignment buffer is established. Based on the timestamps of the image acquisition time and the sensor acquisition time, an interpolation method is used to match the head pose data that is closest in time to each frame of eye image, ensuring that each set of data in subsequent processing is time-aligned.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By utilizing the image coordinate set of the corner of the eye and camera intrinsic parameters, a 3D spatial coordinate system of the corner of the eye in the head coordinate system is calculated using a 3D reconstruction method. Based on this, the initial position of the eyeball center in the head coordinate system is obtained, so that the eyeball center positioning no longer relies on the fixed 2D pupil position calculation, but is established on the 3D spatial point corresponding to the individual head structure. This method can obtain the eyeball center position that is closer to the real anatomical structure when there are individual differences in the head, so that the subsequent gaze direction calculation has a spatial reference that matches the individual morphology and reduces the calculation offset caused by different eyeball positions within the head.

[0017] Based on the head rotation angle and posture data parsed from the head posture sensor, the initial position of the eyeball center in the head coordinate system is transformed to the camera coordinate system, and then further transformed to the screen coordinate system according to the relative positional relationship between the camera and the screen. This allows the eyeball center position to be adjusted in real time across different spatial reference systems as the head posture changes. During head movement, the position of the eyeball center in the screen coordinate system changes accordingly, reflecting the actual spatial relationship. The coordinate reference used to calculate the gaze direction remains consistent with the head orientation, thus ensuring that the gaze point calculation more closely matches the user's actual gaze position during head movement, reducing gaze estimation errors caused by neglecting changes in head posture. Attached Figure Description

[0018] Figure 1 This is a timing diagram of the human-computer interaction gaze estimation system that integrates eye tracking and head posture as described in this invention; Figure 2 A flowchart for calculating the three-dimensional coordinates of the corner of the eye; Figure 3 Curves showing the change in confidence weights for the left and right eyes under different fixation scenarios; Figure 4 This is the original pixel distribution map. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] Please see Figure 1 This invention provides a human-computer interaction gaze estimation system that integrates eye tracking and head posture data. By fusing eye image features and head posture sensor data, it achieves high-precision estimation of gaze direction and screen placement. The overall implementation scheme includes: a data acquisition and analysis module, a three-dimensional coordinate calculation module, a center positioning module, and a gaze placement estimation module.

[0022] The data acquisition and analysis module is responsible for synchronously acquiring the user's eye image sequence and head posture sensor data, analyzing the images to obtain the image coordinates of the corner of the eye and the center of the pupil, and analyzing the sensor data to obtain the head posture angle.

[0023] The 3D coordinate calculation module uses multi-frame image information and head motion information to calculate the 3D coordinates of the corner of the eye in the head coordinate system through 3D reconstruction technology.

[0024] The center localization module fits the three-dimensional coordinates of the corner of the eye to the center position of the eyeball in the head coordinate system based on fixed facial anatomical features. Then, it combines real-time head posture data to transform the position to the camera coordinate system and the screen coordinate system in turn to obtain the estimated position of the eyeball center in the screen coordinate system.

[0025] The gaze point estimation module uses the eye center in the screen coordinate system as a reference, combines the pupil center coordinates extracted from the image, calculates the gaze direction vector through a geometric optics model, and finally intersects the vector with the screen plane to obtain the specific coordinates of the gaze point on the screen.

[0026] See Figure 2 In one embodiment of the present invention, the system continuously captures the user's face using a camera to generate a sequence of eye images. Simultaneously, an inertial measurement unit (IMU) mounted on the user's head continuously outputs head posture sensor data. To process the eye image sequence, a pre-trained face detector is used to process each frame of the image, locating the face region and further cropping a rectangular region containing both eyes as the region of interest (ROI) image based on a standard facial keypoint model. For each frame of the ROI image, an active shape model or grayscale integral projection method is used to accurately locate the pixel coordinates of the inner and outer corners of the left and right eyes in the image. For pupil center localization, the ROI image of each frame is grayscaled and contrast-enhanced. The Canny operator is used for edge detection, and the detected pupil edge contour is fitted with an ellipse using the least squares method. The center coordinates of the fitted ellipse are used as the image coordinates of the left or right pupil center in that frame. All the image coordinates of the corners of the eyes and the pupil center are stored sequentially according to time frames, forming an image coordinate set.

[0027] To achieve temporal consistency of multimodal data, the system performs data synchronization and timestamp alignment steps. When the camera captures each frame, the image acquisition card or software driver appends a precise timestamp to the image acquisition moment. The head pose sensor also appends a timestamp to each data packet it outputs. The system establishes a first-in, first-out (FIFO) time alignment buffer, sending the timestamped image data and sensor data into the buffer. When the processing module needs data, it searches for the closest data packet in the buffer's sensor data sequence based on the timestamp of the current eye image frame to be processed. If the timestamps do not perfectly match, a linear interpolation method is used to calculate head pose data perfectly aligned with the image acquisition moment, based on the data from the preceding and following sensor data packets and their timestamps. This process ensures that the eye feature coordinates used in each subsequent calculation step are strictly synchronized with the head pose data.

[0028] In practical implementation, the system is deployed in an example scenario where the user interacts with the display. The camera captures a video stream of the user's face at a rate of 30 frames per second, generating a sequence of eye images. Simultaneously, the head-mounted inertial measurement unit outputs head posture sensor data at a frequency of 100 times per second. To demonstrate the necessity of data synchronization, the inconsistency between the acquisition time of the eye image sequence and the acquisition time of the head posture sensor data when not synchronized is shown. This inconsistency will cause subsequent modules to use mismatched data for calculation. In practical implementation, a face detection algorithm is used to process each frame of the eye image sequence. The face detection algorithm loads a pre-trained classifier model, scans the face region in the image, determines the coordinates of the face bounding box, and, based on the relative position of the two eye regions in the standard facial key point model, extracts a rectangular region with a fixed aspect ratio from the face bounding box to form a region of interest image containing both eyes. For each frame of the region of interest (ROI) image, the active shape model (AMM) method is used to locate the corner of the eye. The AMM initializes the position of the average shape in the ROI image, iteratively searches for the best matching position for each feature point, and adjusts the shape parameters to minimize the difference between the image grayscale contour and the model. Finally, it outputs the image coordinates of the inner and outer corners of the left and right eyes and stores them in the image coordinate set. The pupil center is located using edge detection and ellipse fitting methods. Gaussian filtering is applied to the ROI image for noise reduction, and the Canny edge detection operator is used to extract the edge pixels of the pupil region. The set of edge pixels is input into the least squares ellipse fitting algorithm to calculate the ellipse parameters. The center coordinates output by the ellipse fitting algorithm are the image coordinates of the center of the left or right pupil and are stored in the image coordinate set.

[0029] In some embodiments, the eye corner point localization employs a gray-scale integral projection method. This method calculates vertical and horizontal gray-scale integral projection curves for the region of interest image. The upper and lower boundaries of the eye are located by finding the troughs of the vertical integral projection curve, and the horizontal position of the eye corner point is located by finding the peaks of the horizontal integral projection curve. Combined with prior geometric relationships, the image coordinates of the left inner and outer eye corners, the right inner and outer eye corners are determined. In the pupil center localization, the ellipse fitting method robustly estimates the ellipse model when noise exists at the edge points. The random sampling consensus algorithm randomly selects the minimum set of points from the edge point set to fit the ellipse, calculates the distances from all edge points to the ellipse model, and iteratively selects the ellipse model with the most interior points as the final result.

[0030] The data synchronization and timestamp alignment steps address the timing drift caused by the independent operation of the acquisition devices in the example scenario. The camera driver adds an image acquisition timestamp to each frame of the eye image, and the head pose sensor firmware adds a sensor acquisition timestamp to each data packet. The system initializes a time alignment buffer, which stores the timestamped head pose sensor data packets in a circular queue structure. When the processing module needs to process a frame of the eye image, it reads the image acquisition timestamp of that frame and searches for two head pose sensor data packets in the time alignment buffer. These two head pose sensor data packets must satisfy the condition that one's sensor acquisition timestamp is not greater than the image acquisition timestamp and the other is not less than the image acquisition timestamp. A linear interpolation method is used to calculate the head pose sensor data aligned with the image acquisition timestamp. The linear interpolation formula is: in: Represents the timestamp of the image acquisition time; The timestamp representing the sensor acquisition time of the first head pose sensor data packet; The timestamp representing the sensor acquisition time of the second head posture sensor data packet; The attitude data representing the first head attitude sensor data packet; The attitude data representing the second head attitude sensor data packet; This represents the aligned pose data output after interpolation. Through this interpolation, each frame of the eye image sequence obtains a temporally synchronized head pose sensor data. Compared to the method of directly using the nearest neighbor head pose sensor data packet, the linear interpolation method reduces the temporal alignment error introduced by discrete sampling.

[0031] It is understood that the capacity of the time alignment buffer is set according to the output frequency of the head pose sensor data. The buffer stores the head pose sensor data packets within the most recent second, ensuring that two head pose sensor data packets are provided for interpolation of each frame of eye image. Optionally, when the timestamp of the image acquisition time exceeds the timestamp range of the head pose sensor data packets in the buffer, the system discards the frame of eye image and records the loss event, avoiding the introduction of uncertainty using extrapolated data. In some embodiments, the data synchronization step is implemented at the operating system kernel level, using a high-precision performance counter to timestamp the data, reducing user space scheduling latency and improving timestamp alignment accuracy. It is understood that the data synchronization and timestamp alignment steps are the foundation for fusing eye movement and head pose. By comparing the temporal correlation between the eye image sequence before and after synchronization and the head pose sensor data, the time offset before synchronization may reach tens of milliseconds. After synchronization, the time offset is eliminated, and the feature point image coordinate set of the eye image sequence is aligned with the head pose sensor data at the frame level, providing consistent input for the 3D coordinate calculation module.

[0032] In one embodiment of the invention, the 3D coordinate calculation module handles example scenarios where the user's head undergoes translation and rotation in a natural state. Head posture sensor data indicates that the head coordinate system has undergone measurable rotation and translation changes relative to the camera coordinate system. To demonstrate the necessity of 3D reconstruction, and in contrast to situations where depth information of the corner of the eye cannot be obtained using only a single frame image, the 3D coordinate calculation module must utilize multiple frames of images and corresponding head motion information. The module establishes a projection relationship based on the camera pinhole model, which projects 3D spatial points onto a 2D image plane. For the 3D spatial coordinates of the corner of the eye in the head coordinate system, its coordinates in the image satisfy the projection relationship equation. The system selects at least two frames of eye images at different times from the data stream after data synchronization and timestamp alignment steps, where the head posture has significant differences. This data synchronization and timestamp alignment step ensures that the selected eye images are precisely aligned in time with the head posture sensor data at the corresponding time by adding a timestamp to each frame of the eye image sequence and each data packet of the head posture sensor data, and by using a time alignment buffer and interpolation method. The system simultaneously acquires aligned head pose sensor data for corresponding moments. This data represents the rotation angle and translation of the head coordinate system relative to the camera coordinate system at different times. Using this data, the relative rotation matrix and translation vector of the head coordinate system from the first frame to the second frame can be calculated. Substituting the image coordinates obtained from feature point detection in two frames of eye images for the same corner of the eye (e.g., the inner corner of the left eye), the known camera intrinsic parameter matrix, and the calculated relative rotation matrix and translation vector, into the projection relation equation, a system of overdetermined equations is constructed with the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system as unknowns. Solving this system of overdetermined equations yields the three-dimensional coordinates of the corner of the eye.

[0033] After obtaining the three-dimensional coordinates of the corner points of the eyes, the center localization module begins to fit the initial position of the eyeball center. Based on prior knowledge of human anatomy, the eyeball center has a relatively stable spatial position relative to the inner and outer corner points of the same side eye. This relationship is modeled as a relative spatial position constraint model, which includes the constraint that the eyeball center is located near the perpendicular bisector of the line connecting the corner points and at a certain proportional distance from the line. The calculated three-dimensional spatial coordinates of the left inner and outer corner points in the head coordinate system are substituted into this relative spatial position constraint model to form a set of geometric constraint equations. Due to measurement and calculation errors, the least squares method is used to solve this set of equations to obtain the three-dimensional spatial coordinates of the left eyeball center in the head coordinate system in the sense of minimizing the error. Similarly, the three-dimensional coordinates of the right inner and outer corner points are substituted into the model, and the least squares method is used to obtain the three-dimensional spatial coordinates of the right eyeball center in the head coordinate system. The coordinates of the left and right eyeball centers together constitute the initial position of the eyeball center in the head coordinate system.

[0034] In practical implementation, the 3D coordinate calculation module handles example scenarios where the user's head undergoes translation and rotation in a natural state. Head posture sensor data indicates measurable rotation and translation changes in the head coordinate system relative to the camera coordinate system. To demonstrate the necessity of 3D reconstruction, and contrasting the situation where depth information of the corner of the eye cannot be obtained using only a single frame image, the 3D coordinate calculation module must utilize multiple frames of images and corresponding head motion information. The module establishes a projection relationship based on the camera pinhole model, which projects 3D spatial points onto a 2D image plane. For the 3D spatial coordinates of the corner of the eye in the head coordinate system, its coordinates in the image satisfy the projection relationship equation. In the synchronized data stream, at least two frames of eye images at different times with significant differences in head posture are selected, and the corresponding head posture sensor data is obtained. This head posture sensor data represents the rotation angle and translation amount of the head coordinate system relative to the camera coordinate system at different times. Using this data, the relative rotation matrix and translation vector of the head coordinate system from the first frame to the second frame can be calculated. The image coordinates of the same corner of the eye, such as the inner corner of the left eye, obtained by feature point detection in two frames of eye images, the known camera intrinsic parameter matrix, and the calculated relative rotation matrix and translation vector are all substituted into the projection relationship equation to construct an overdetermined system of equations with the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system as unknowns. The three-dimensional coordinates of the corner of the eye can be obtained by solving the overdetermined system of equations.

[0035] In some embodiments, two or more eye images are selected to construct an overdetermined system of equations to further increase constraints and reduce noise. For each selected eye image, the transformation from the head coordinate system to the head coordinate system at that frame time is calculated based on the corresponding head pose sensor data. The image coordinates and transformation relationship of the corner of the eye in that frame are substituted into the projection relationship to form more equations, constituting an overdetermined system of equations. Singular value decomposition (SVD) is used to solve the overdetermined system of equations. SVD directly solves the linear least squares problem, obtaining the least squares solution of the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system. Compared with the method of directly using two images to construct just-enough constraint equations, the overdetermined system of equations constructed using multiple images has better numerical stability. The formula for constructing the projection relationship equations is expressed as: in: This represents the column vector of three-dimensional spatial coordinates of the corner of the eye in the head coordinate system. This represents the rotation matrix from the head coordinate system to the camera coordinate system at frame j; This represents the translation vector from the head coordinate system to the camera coordinate system at frame j. Represents the camera intrinsic parameter matrix; This represents the image coordinates of the corner of the eye in the j-th frame of the eye image; This represents a non-zero scaling factor. By eliminating the scaling factor... The formula can be transformed into a formula about The linear equations, combined with multiple frames of data, form an overdetermined system of equations. Solve in the form of .

[0036] After obtaining the three-dimensional spatial coordinates of the left inner and outer canthi of the left eye, the right inner and outer canthi of the right eye in the head coordinate system, the center localization module fits the initial position of the eyeball center based on prior knowledge of human anatomy. The relative spatial position constraint model describes the eyeball center as being located near the midline of the line connecting the canthi on the same side, and the distance from the eyeball center to the line connecting the canthi conforms to the range of anatomical statistical values. Substituting the calculated three-dimensional spatial coordinates of the left inner and outer canthi in the head coordinate system into the relative spatial position constraint model, the model expresses the geometric relationship as a set of linear equations. Due to measurement and reconstruction errors, this set of linear equations may not have an exact solution. The least squares method is used to solve this set of equations to find a three-dimensional spatial coordinate of the left eyeball center in the head coordinate system that minimizes the sum of squared errors from this coordinate to the geometric constraints defined by the model. It can be understood that the specific parameters of the relative spatial position constraint model are obtained through statistical analysis of a large amount of sample data and loaded during system initialization. Optionally, for calculating the center of the right eyeball, the same relative spatial position constraint model and least squares solution process are used. The three-dimensional spatial coordinates of the inner and outer canthi points of the right eyeball in the head coordinate system are input to obtain the three-dimensional spatial coordinates of the right eyeball center in the head coordinate system. The three-dimensional spatial coordinates of the left and right eyeball centers together constitute the initial position of the eyeball center in the head coordinate system. Compared to methods that directly use the midpoint of the canthus as an approximation of the eyeball center without fitting, the least squares fitting based on model constraints can more accurately reflect the true anatomical position of the eyeball center.

[0037] It is understandable that in example scenarios of natural head movement, the 3D coordinates of the corner of the eye obtained through 3D reconstruction may fluctuate due to image noise. The least squares fitting process has a certain smoothing effect on the noise of the input coordinates. In some embodiments, the relative spatial position constraint model not only includes the corner of the eye but also introduces other stable facial feature points as constraints to further improve the robustness of the initial position estimation of the eyeball center. Optionally, the least squares method is solved using the normal equation method or QR decomposition method to ensure computational efficiency on the embedded system. The 3D spatial coordinates of the left and right eyeball centers are stored in memory for use in subsequent coordinate transformation steps of the center localization module.

[0038] In one embodiment of the invention, the center localization module handles example scenarios where the user's head rotates and translates in free space. The head posture sensor outputs real-time posture data including pitch, yaw, and roll angles. To demonstrate the necessity of coordinate transformation, it is shown that directly using the initial position of the eyeball center in the head coordinate system for gaze calculation would produce errors, as the head coordinate system moves with the head. Gaze calculation requires the eyeball center position in a fixed reference frame. The center localization module receives output data from the preceding 3D reconstruction and fitting steps, namely the 3D spatial coordinates of the left and right eyeball centers in the head coordinate system. Based on the pitch, yaw, and roll angles contained in the posture data, the center localization module constructs a 3x3 rotation matrix according to the ZYX Euler angle rotation order. This rotation matrix defines the rotational transformation from the head coordinate system to the camera coordinate system. Combined with a translation vector pre-measured during the initial system calibration phase, representing the 3D coordinates of the head coordinate system origin in the camera coordinate system, the rotation matrix and translation vector are combined into a 4x4 homogeneous transformation matrix. The homogeneous transformation matrix is ​​used to transform the 3D spatial coordinates of the left eyeball center in the head coordinate system, which are already input from the center positioning module. This transformation is achieved by representing the 3D spatial coordinates of the left eyeball center in the head coordinate system as homogeneous coordinates and performing matrix multiplication with the homogeneous transformation matrix. The result is the 3D spatial coordinates of the left eyeball center in the camera coordinate system. The same matrix multiplication operation is performed on the input 3D spatial coordinates of the right eyeball center in the head coordinate system to obtain the 3D spatial coordinates of the right eyeball center in the camera coordinate system. This transformation process links the eyeball center position from the moving head coordinate system to the stationary camera coordinate system.

[0039] In some embodiments, posture data represents head rotation in quaternion form. The center localization module converts the quaternion into an equivalent rotation matrix to construct a homogeneous transformation matrix. Quaternion representation avoids the gimbal lock problem that may exist with Euler angles. Optionally, the translation vector is obtained through a one-time calibration while the user wears the device. The calibration process requires the user to gaze at a known point on the screen, and an optimization algorithm is used to solve for the position of the origin of the head coordinate system relative to the camera coordinate system. The transformation from the camera coordinate system to the screen coordinate system is based on a pre-completed calibration process. The calibration process determines the rotation and translation relationship between the camera coordinate system and the screen coordinate system. This rotation and translation relationship includes a 3x3 rotation matrix and a 3x1 translation vector, forming another homogeneous transformation matrix. Using this homogeneous transformation matrix from the camera coordinate system to the screen coordinate system, the three-dimensional spatial coordinates of the left and right eyeball centers in the camera coordinate system are transformed. The transformation result is the three-dimensional spatial coordinates of the left and right eyeball centers in the screen coordinate system. In the screen coordinate system, the average of the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball is calculated. The average point is used as the reference position of the user's head center in the screen coordinate system. Compared with directly using the coordinates of a single eyeball without averaging, using the average can reduce the impact of unilateral data noise.

[0040] It is understandable that due to individual anatomical differences, simple averaging may not accurately reflect the true positions of both eyes in the screen coordinate system. Therefore, it is necessary to fine-tune the positions of the centers of the left and right eyeballs in conjunction with the interpupillary distance (IPD). The fine-tuning process calculates the distance between the current centers of the left and right eyeballs in the screen coordinate system based on the individual IPD parameters measured during the user calibration phase. This calculated distance is compared with the calibrated individual IPD parameters, and the positions of the centers of the left and right eyeballs are adjusted by reverse translation along the line connecting the two eyes, so that the adjusted distance between the centers of the eyeballs equals the calibrated individual IPD parameters. Optionally, the fine-tuning process uses a scaling method, with the head center reference position as the scaling center, to scale the position coordinates of the centers of the left and right eyeballs, so that the distance between the two points equals the calibrated individual IPD parameters. In some embodiments, the IPD parameters are calculated from eye images when the user gazes at two known horizontal points on the screen. The three-dimensional spatial coordinates of the fine-tuned centers of the left and right eyeballs in the screen coordinate system are output as the final estimated position of the eyeball centers in the screen coordinate system. Compared to the average position without fine-tuning, the estimated position after fine-tuning better reflects the user's individual physiological characteristics, providing a more accurate reference point for subsequent gaze calculations. The complete coordinate transformation formula from the head coordinate system to the screen coordinate system is expressed as: in: This represents the homogeneous vector of the three-dimensional spatial coordinates of the eyeball center in the head coordinate system. Represented by pitch angle Yaw angle Roll angle And the homogeneous transformation matrix from the head coordinate system to the camera coordinate system constructed by the translation vector; This represents the homogeneous transformation matrix from the camera coordinate system to the screen coordinate system; It represents the homogeneous vector of the three-dimensional spatial coordinates of the eyeball center in the screen coordinate system.

[0041] In one embodiment of the present invention, the gaze point estimation module first processes the image coordinates of the pupil center. Based on the camera pinhole model and known camera intrinsic parameters, it calculates a three-dimensional ray direction in the camera coordinate system that originates from the camera's optical center and passes through the image point by back-projecting the two-dimensional coordinates of the pupil center detected in the image. This ray points to the three-dimensional spatial direction where the pupil center may be located. The module receives the estimated position of the eyeball center in the screen coordinate system and, combined with the known transformation relationship between the camera coordinate system and the screen coordinate system, deduces the three-dimensional coordinates of the eyeball center in the camera coordinate system.

[0042] In the camera coordinate system, a spatial straight line can be defined by connecting the three-dimensional coordinates of the eye's center with any point on the aforementioned reverse projection ray. Because light is refracted by the cornea before entering the camera, the direction of this straight line is not directly equal to the actual line of sight. The system establishes a specific refractive model of the corneal spherical surface, simplifying the corneal surface into an ideal sphere. This model includes two physiological parameters: the corneal radius of curvature and the refractive index of the corneal medium. In the camera coordinate system, the spatial straight line connecting the eye's center with a point on the projection ray is extended towards the corneal model sphere. Assuming this line intersects the corneal sphere at a point, according to Snell's law of refraction, the normal vector of the sphere is calculated at the point of incidence, and the direction of light after refraction from air into the corneal medium is calculated based on the corneal refractive index parameter.

[0043] A key geometric constraint is established: the backward extension of the outgoing ray after corneal refraction must pass through the three-dimensional coordinates of the camera coordinate system corresponding to the estimated position of the eye center in the screen coordinate system, as determined in the above embodiment. Based on this constraint, a system of nonlinear equations can be established, with the specific selected point on the projection ray of the spatial line as the unknown variable. An iterative optimization algorithm, such as the Gauss-Newton method, is used to solve this system of nonlinear equations to find the point on the projection ray that satisfies the geometric constraint. After successful solution, connecting this point with the vector direction determined by the eye center in the camera coordinate system yields the expression for the true gaze direction vector in the camera coordinate system, corrected for corneal refraction. This vector is then normalized to obtain the unit gaze direction vector. In practical implementation, the gaze point estimation module handles example scenarios where the user gazes at different points on the screen. The coordinates of the pupil center in the eye image captured by the camera shift, but the accurate gaze direction cannot be obtained by simply projecting the pupil center image coordinates directly backward because light refracts on the corneal surface. The gaze point estimation module uses the set of image coordinates of the pupil center point, camera intrinsic parameters, and the estimated position of the eye center in the screen coordinate system for calculation. The module performs backward projection on the two-dimensional coordinates of the pupil center detected in the image based on the camera pinhole model. The backward projection process uses the inverse of the camera intrinsic parameter matrix to transform the pixel coordinates to the camera normalized plane, resulting in a direction vector of a three-dimensional ray that originates from the camera optical center and passes through the point of the normalized plane in the camera coordinate system. This projection ray points to multiple possible positions of the pupil center in three-dimensional space. The module receives the estimated position of the eye center in the screen coordinate system. This estimated position is a three-dimensional point. Combining the known transformation relationship between the camera coordinate system and the screen coordinate system, the estimated position of the eye center in the screen coordinate system is transformed to the camera coordinate system through inverse transformation to obtain the three-dimensional coordinates of the eye center in the camera coordinate system.

[0044] In the camera coordinate system, a spatial straight line can be defined by connecting the three-dimensional coordinates of the eye center with any point on the projection ray. Since light rays need to be refracted by the cornea before entering the camera lens, the direction of this spatial straight line is not equal to the actual line of sight. The system establishes a corneal spherical refraction model to correct the refraction effect. This model simplifies the anterior surface of the cornea to an ideal sphere and includes two key physiological parameters: the corneal radius of curvature and the corneal refractive index. These parameters are input as known constants into the system. In practice, the spatial straight line connecting the eye center with a point on the projection ray is extended to the spherical surface of the corneal spherical refraction model. Assuming this spatial straight line intersects the corneal spherical surface at an incident point, the normal vector direction at that point is calculated according to the spherical equation of the corneal spherical refraction model. The refraction direction of the incident light ray is then calculated according to Snell's law of refraction. The formula for Snell's law of refraction is: in: Represents the refractive index of air; Represents the refractive index of the corneal medium as defined in a specific refractive model of the corneal spherical surface; This represents the angle between the incident ray and the normal vector at the point of incidence; This represents the angle between the refracted ray and the normal vector. Refraction calculations occur at the interface where the refracted ray enters the corneal medium from the air.

[0045] The system establishes a key geometric constraint: the backward extension of the outgoing ray refracted by a specific refractive model of the corneal sphere must precisely pass through the 3D coordinate point in the camera coordinate system corresponding to the estimated position of the eye's center in the screen coordinate system. This constraint is based on the optical model of the eye, where the line of sight is the outgoing direction of the ray connecting the eye's center and the pupil's center after refraction through the cornea. A system of nonlinear equations is established based on this geometric constraint. The unknowns in this system are the coordinate parameters of specific points on the projection ray. These coordinates determine the position of the incident point on the corneal sphere, thus affecting the direction of the refracted ray. An iterative optimization algorithm is used to solve the nonlinear equations. Starting from an initial point on the projection ray (e.g., the intersection of the projection ray and a plane at an assumed distance), the algorithm calculates the direction of the refracted ray at the current point and the distance error between the backward extension of the refracted ray and the target eye's center. The coordinates of points on the projection ray are iteratively updated using gradient descent or similar methods until the distance error is less than a set threshold. The direction of the spatial straight line obtained by solving, that is, the vector direction connecting the point on the projection ray and the center of the eyeball in the camera coordinate system, is considered to be the true line-of-sight vector after corneal refraction correction. This vector is normalized to obtain the expression of the unit line-of-sight vector in the camera coordinate system.

[0046] In some embodiments, the iterative optimization algorithm employs the Gauss-Newton method or the Levenberg-Marquardt algorithm, which can efficiently solve nonlinear least squares problems, transforming geometric constraints into an objective function that minimizes distance error. Optionally, parameters of the corneal spherical refractive model, such as the corneal radius of curvature and corneal refractive index, can use average physiological values ​​or be estimated and optimized through a user-specific calibration process. Table 1 lists typical values ​​of commonly used physiological parameters in the corneal spherical refractive model.

[0047] Table 1: Typical Physiological Parameters of a Specific Refractive Model of the Cornea It is understandable that constructing and solving the nonlinear geometric constraints involving corneal refraction is crucial for the accuracy of gaze estimation. Compared to the simple pupil vector method that ignores refraction, this method can significantly compensate for the systematic errors in gaze estimation caused by corneal refraction. In some embodiments, to simplify the calculation, a gaze direction lookup table can be pre-established under different eye positions and pupil image coordinates, and the gaze direction vector can be obtained by interpolation, avoiding real-time solving of nonlinear equations. Optionally, the calculation process processes left and right eye data simultaneously, obtaining left-eye and right-eye gaze direction vectors respectively, which can then be fused or the result from the side with higher confidence can be selected. It is understood that the normalization of the gaze direction vector simplifies the subsequent calculation of intersection with the screen plane, as the normalized unit vector directly represents the direction of the gaze in the camera coordinate system.

[0048] See Figure 3 In the analysis of left and right eye gaze direction vector fusion, the changes in confidence scores for both eyes under different gaze scenarios are the core basis for achieving high-precision gaze fusion. The figure uses "gaze scenario" as the horizontal axis, encompassing five typical scenarios: forward gaze, upper left gaze, lower right gaze, head-down gaze, and head-up gaze; and "confidence weight" as the vertical axis, quantifying the confidence performance of the left and right eyes under different scenarios. Data trends show that in upper left gaze and head-up gaze scenarios, the left eye has a higher confidence weight (peaking at 0.7 and 0.65 respectively), reflecting better gaze capture stability in upper left and head-up views. In the head-down gaze scenario, the right eye's confidence weight significantly increases to 0.6, indicating stronger reliability in gaze estimation by the right eye in the head-down view. This scenario-specific confidence difference stems from the combined effects of corneal refraction, eye rotation angle, and image quality on both eyes under different gaze postures. During gaze vector fusion, the system can dynamically allocate the fusion ratio of binocular data based on the confidence weight of the graph. For example, when looking up to the left, the left eye data is the core and the right eye data is the auxiliary, while when looking down, the right eye data is the main data, thereby effectively improving the overall accuracy of gaze point estimation.

[0049] In one embodiment of the present invention, after obtaining the unit gaze direction vector in the camera coordinate system, the gaze point estimation module needs to calculate its intersection with the screen plane. The spatial position of the screen plane in the camera coordinate system has been obtained through system calibration and is usually represented by a plane equation, which is defined in the form of a normal vector and the distance from the plane to the origin. The equation of the line containing the gaze direction vector is combined with the screen plane equation. The equation of the line containing the gaze direction vector is defined by the coordinates of the eye center in the camera coordinate system and the gaze direction vector. By solving the equation of the line and the plane equation simultaneously, the coordinates of the intersection point of the line and the plane are calculated. These intersection point coordinates are the intersection point of the gaze optical axis and the physical plane of the screen, represented in three-dimensional coordinates of the camera coordinate system.

[0050] After obtaining the physical intersection coordinates, they need to be converted into pixel-level coordinates on the screen. The system establishes a bilinear mapping or projection transformation from the camera coordinate system to the screen pixel coordinate system by calibrating the known physical size and resolution of the screen, as well as the coordinates of the four corner points of the screen plane in the camera coordinate system. Using this mapping, the obtained physical intersection coordinates are transformed into the screen pixel coordinate system with the top-left corner of the screen as the origin. The resulting two-dimensional pixel coordinates represent the coordinates of the user's current gaze position on the screen, and these coordinates can be directly used in subsequent human-computer interaction applications.

[0051] In practical implementation, the gaze-point estimation module handles example scenarios where the user is looking at an icon or text area on the screen. The module obtains the normalized unit gaze direction vector in the camera coordinate system and the three-dimensional coordinates of the eye center in the camera coordinate system. To demonstrate the necessity of intersecting the gaze direction with the screen plane, and contrasting this with methods that directly use the gaze direction angle for rough estimation without intersection (which cannot provide precise two-dimensional pixel coordinates on the screen), the gaze-point estimation module obtains the plane equation of the screen plane in the camera coordinate system. This equation is determined through a pre-calibrated system process. The calibration process utilizes the positions of multiple screen corner points or feature points with known three-dimensional coordinates in the camera coordinate system and employs a plane fitting algorithm to calculate the point-normal equation of the screen plane. The equation of the line containing the gaze direction vector is then solved simultaneously with the screen plane equation. The equation of the line containing the gaze direction vector is defined by the three-dimensional coordinates of the eye center in the camera coordinate system and the gaze direction vector, and is expressed in parametric form.

[0052] The simultaneous solution process substitutes the parametric form of the linear equation into the point-normal form of the screen plane equation, obtaining a scalar parameter value. Substituting this scalar parameter value back into the linear equation, the three-dimensional coordinates of the intersection point of the line and the plane are calculated. These intersection point coordinates represent the physical location where the line-of-sight optical axis intersects the physical plane of the screen, expressed in three-dimensional coordinates of the camera coordinate system. The obtained physical intersection point coordinates are then transformed from the camera coordinate system to the screen pixel coordinate system. The screen pixel coordinate system is a two-dimensional coordinate system, with its origin typically defined at the upper left corner of the screen display area. The positive X-axis points horizontally to the right and the positive Y-axis points vertically downwards. This transformation requires a mapping relationship from three-dimensional points in the camera coordinate system to two-dimensional points in the screen pixel coordinate system. This mapping relationship is established during system calibration. This mapping relationship is usually obtained by solving a projection transformation matrix or bilinear transformation parameters. Using the known three-dimensional coordinates of the four corner points of the screen in the camera coordinate system and their known pixel coordinates in the screen pixel coordinate system, a 3x3 homography transformation matrix is ​​calculated to project points on the screen plane in the camera coordinate system to the pixel coordinates.

[0053] In some embodiments, when the screen plane is not parallel to the camera imaging plane, the projection transformation is performed directly using the aforementioned physical intersection coordinates. The projection transformation formula is expressed as: in: The X and Y components of the intersection of the line of sight and the screen plane in the camera coordinate system (the Z component is implicitly determined by the plane equation). This represents the 3x3 homography transformation matrix obtained through calibration; Represents the homogeneous pixel coordinates after projection; The scaling factor represents the homogeneous coordinates, and the final screen pixel coordinates are... Compared to methods that directly assume the screen plane is parallel to the image plane and perform similar triangle conversions, using a homography transformation matrix can accurately handle cases where the screen plane is in any spatial orientation. Optionally, the mapping relationship can also be implemented by establishing a bilinear interpolation lookup table from the camera coordinate system to the screen pixel coordinate system to handle non-ideal projection models.

[0054] It is understandable that transforming the physical intersection coordinates to the screen pixel coordinate system is a crucial step in generating the final interactive instructions. The resulting two-dimensional pixel coordinates are the coordinates of the gaze point on the screen, which can be directly used to control cursor movement or identify the user's gaze target. In some embodiments, the system performs time-domain filtering on the gaze point coordinates calculated over multiple consecutive frames, such as using a Kalman filter or moving average, to smooth noise and improve the stability of the gaze point coordinates. Optionally, for the gaze point coordinates calculated separately for the left and right eyes, the system calculates their weighted average or selects the coordinates of the side with higher confidence as the final output. The gaze point coordinates are provided to the upper-layer human-computer interaction application in the form of integer pixel coordinates or floating-point subpixel coordinates, completing the entire estimation process from eye images and head pose data to the specific position on the screen.

[0055] See Figure 4This figure presents the distribution characteristics of the original pixel landing points in the screen pixel coordinate system. The rectangles in the figure define the boundaries of the screen display area. The purple and gradient-colored scattered dots represent the original gaze point pixel coordinates corresponding to different experimental sample numbers, with the color gradation from dark purple to bright yellow corresponding to the increasing experimental sample numbers from 0 to 40. The horizontal axis is the screen pixel X coordinate (range 0-2000px), and the vertical axis is the screen pixel Y coordinate (range 0-1000px), completely covering the pixel space of the screen display area. From the distribution, gaze points are distributed in the upper-middle, middle, and lower parts of the screen, with some landing points close to the screen boundary, reflecting the spatial randomness of user gaze behavior; the changes in color gradation reflect the temporal distribution characteristics of the landing points under the experimental sample sequence. The figure not only verifies the effectiveness of the pixel landing points output by the coordinate transformation module within the screen display area but also provides an intuitive spatial distribution basis for subsequent analysis of gaze point stability and system calibration accuracy.

[0056] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A human-computer interaction gaze estimation system integrating eye tracking and head posture, characterized in that, include: The data acquisition and analysis module acquires user eye image sequences and head posture sensor data, performs feature point detection on the eye image sequences, extracts the image coordinate set of the corner of the eye and the center of the pupil, and analyzes the head posture sensor data to obtain posture data representing the head rotation angle. The three-dimensional coordinate calculation module establishes the transformation relationship between the head coordinate system, the camera coordinate system, and the screen coordinate system. Using the image coordinate set of the corner of the eye point and the camera intrinsic parameters, it calculates the three-dimensional spatial coordinates of the corner of the eye point in the head coordinate system through a three-dimensional reconstruction method. The center positioning module fits the initial position of the eyeball center in the head coordinate system based on the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system. According to the posture data, the initial position of the eyeball center is transformed from the head coordinate system to the camera coordinate system, and then transformed to the screen coordinate system according to the relative position relationship between the camera and the screen, so as to obtain the estimated position of the eyeball center in the screen coordinate system. The gaze point estimation module uses the image coordinate set of the pupil center point, camera intrinsic parameters, and the estimated position of the eyeball center in the screen coordinate system to calculate the gaze direction vector. The intersection of the gaze direction vector with the screen plane is then used to obtain the coordinates of the gaze point on the screen.

2. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 1, characterized in that, The step of performing feature point detection on the eye image sequence to extract the image coordinate set of the corner of the eye and the center of the pupil specifically includes: A face detection algorithm is used to locate each frame of the eye image sequence, and the region of interest containing the eyes is extracted. For each frame of the region of interest image, the image coordinates of the inner canthus of the left eye, the outer canthus of the left eye, the inner canthus of the right eye, and the outer canthus of the right eye are located using the gray-scale integral projection method or the shape model method. For each frame of the region of interest image, edge detection and ellipse fitting methods are used to locate the image coordinates of the center of the left pupil and the center of the right pupil, respectively. The image coordinates of the inner corner of the left eye, the outer corner of the left eye, the inner corner of the right eye, the outer corner of the right eye, the center of the left pupil, and the center of the right pupil are stored in chronological order to form the image coordinate set of the corner of the eye and the center of the pupil.

3. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 1, characterized in that, The step of calculating the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system using the image coordinate set of the corner of the eye and camera intrinsic parameters through a three-dimensional reconstruction method specifically includes: Based on the camera pinhole model, establish the equation relating the image coordinates of the corner of the eye to its projection coordinates in the three-dimensional space of the camera coordinate system; Select at least two frames of eye images at different times and obtain the pose data at the corresponding times; use the pose data to calculate the relative rotation and translation transformation of the head coordinate system between the two frames; Substitute the image coordinates of the same eye corner point in two frames, the camera intrinsic parameters, and the relative rotation and translation transformations into the projection relationship equation to construct an overdetermined system of equations. Solve the overdetermined system of equations to obtain the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system.

4. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 3, characterized in that, The process of fitting the initial position of the eyeball center in the head coordinate system based on the three-dimensional spatial coordinates of the corner of the eye in the head coordinate system specifically includes: Based on prior knowledge of human anatomy, a relative spatial position constraint model is established between the center of the eyeball and multiple corner points of the eye. Substitute the calculated three-dimensional spatial coordinates of the inner and outer corners of the left eye in the head coordinate system into the relative spatial position constraint model, and solve the three-dimensional spatial coordinates of the center of the left eyeball in the head coordinate system by the least squares method. Substitute the calculated three-dimensional spatial coordinates of the inner and outer corners of the right eye in the head coordinate system into the relative spatial position constraint model, and solve the three-dimensional spatial coordinates of the center of the right eyeball in the head coordinate system by the least squares method. The three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball together constitute the initial position of the eyeball center in the head coordinate system.

5. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 4, characterized in that, The step of transforming the initial position of the eyeball center from the head coordinate system to the camera coordinate system based on the posture data specifically includes: Based on the pitch angle, yaw angle, and roll angle contained in the attitude data, construct the rotation matrix from the head coordinate system to the camera coordinate system; Based on the rotation matrix and the preset translation vector of the head coordinate system origin relative to the camera coordinate system, construct a homogeneous transformation matrix; Using the homogeneous transformation matrix, the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the head coordinate system are transformed respectively to obtain the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the camera coordinate system.

6. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 5, characterized in that, The next step, based on the relative positional relationship between the camera and the screen, transforms to the screen coordinate system to obtain the estimated position of the eyeball center in the screen coordinate system. Specifically, this includes: The rotation and translation relationship between the camera coordinate system and the screen coordinate system is obtained through pre-calibration; Using the aforementioned rotation and translation relationship, the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the camera coordinate system are transformed to the screen coordinate system to obtain the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the screen coordinate system. The average of the three-dimensional spatial coordinates of the center of the left eyeball and the center of the right eyeball in the screen coordinate system is used as the reference position of the user's head center in the screen coordinate system. The positions of the center of the left eyeball and the center of the right eyeball are then finely adjusted in combination with the interpupillary distance to finally obtain the estimated position of the eyeball center in the screen coordinate system for line of sight calculation.

7. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 1, characterized in that, The step of calculating the gaze direction vector using the image coordinate set of the pupil center point, camera intrinsic parameters, and the estimated position of the eyeball center in the screen coordinate system specifically includes: Based on the image coordinates of the pupil center point and the camera intrinsic parameters, the direction of the projection ray of the pupil center point in the camera coordinate system is obtained by back projection. Based on the estimated position of the eyeball center in the screen coordinate system and the relative position of the camera and the screen, the three-dimensional coordinates of the eyeball center in the camera coordinate system are deduced. In the camera coordinate system, a spatial straight line is formed by connecting the three-dimensional coordinates of the eyeball center with a point on the projection ray; According to the principle of corneal reflection, the direction of the spatial straight line, after being corrected by the refraction of the corneal spherical surface, is the true direction of vision. By solving the geometric constraints that allow the spatial straight line to point to the estimated position of the eyeball center in the screen coordinate system after passing through a specific refractive model of the corneal sphere, the expression of the gaze direction vector in the camera coordinate system is calculated.

8. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 7, characterized in that, The process involves solving the geometric constraints that allow the spatial straight line to point to the estimated position of the eyeball center in the screen coordinate system after passing through a specific refractive model of the corneal sphere, and then calculating the expression of the gaze direction vector in the camera coordinate system. Specifically, this includes: A corneal spherical refractive model is established with the corneal surface as an ideal sphere. The corneal spherical refractive model includes corneal radius of curvature and refractive index parameters. In the camera coordinate system, the spatial straight line connecting the three-dimensional coordinates of the eyeball center and a point on the projection ray is extended to the spherical surface of the specific refractive model of the cornea. According to Snell's law of refraction, the normal vector of the spatial line at the incident point on the specific refractive model sphere of the cornea is calculated, and the exit direction of the light after refraction by the cornea is calculated based on the refractive index parameter. Construct geometric constraints that require the backward extension of the outgoing light rays after refraction through the cornea to pass through the three-dimensional coordinates of the camera coordinate system corresponding to the estimated position of the eyeball center in the screen coordinate system. A system of nonlinear equations is established based on the geometric constraints, where the unknowns of the nonlinear equations are the specific points on the projection ray of the spatial line. The nonlinear equations are solved using an iterative optimization algorithm to obtain the direction of the spatial line that satisfies the geometric constraints. The direction vector of the spatial line obtained by the solution is normalized to obtain the expression of the line of sight direction vector in the camera coordinate system.

9. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 7, characterized in that, The step of intersecting the gaze direction vector with the screen plane to obtain the coordinates of the gaze point on the screen specifically includes: Obtain the plane equation of the screen plane in the camera coordinate system; Solve the equation of the line containing the calculated line direction vector and the equation of the screen plane simultaneously. The coordinates of the intersection point between the line and the plane are calculated, and these coordinates are the physical intersection point between the line of sight optical axis and the screen. The coordinates of the physical intersection point are transformed into the screen pixel coordinate system with the top left corner of the screen as the origin to obtain the final coordinates of the landing point of the line of sight on the screen.

10. The human-computer interaction gaze estimation system integrating eye tracking and head posture as described in claim 1, characterized in that, After acquiring the user's eye image sequence and head posture sensor data, the process also includes data synchronization and timestamp alignment steps: A timestamp of the image acquisition time is appended to each frame of the eye image sequence; A timestamp of the sensor acquisition time is appended to each data packet of the head posture sensor data; A time alignment buffer is established. Based on the timestamps of the image acquisition time and the sensor acquisition time, an interpolation method is used to match the head pose data that is closest in time to each frame of eye image, ensuring that each set of data in subsequent processing is time-aligned.

Images