Virtual-real fusion dynamic calibration method based on binocular eye movement tracking

Abstract

The invention discloses a virtual-real fusion dynamic calibration method based on binocular eye movement tracking. The method comprises the following steps: step 1, constructing a virtual-real fusion dynamic calibration system based on binocular eye movement tracking; step 2, establishing a calibration coordinate system; 3, solving a conversion relation between the coordinate system of the eye movement tracking camera and the coordinate system of the AR equipment through offline joint calibration; 4, collecting eye movement tracking parameters and modeling user characteristics; 5, constructing a virtual display camera model, and solving an internal parameter matrix and external parameter transformation of a virtual display camera, so that the virtual display camera is geometrically consistent with the real watching behavior of the user; step 6, constructing a conversion relation between an external tracking sensor coordinate system and a virtual display coordinate system, so as to convert the position of the marker to a virtual display camera coordinate system; and step 7, evaluating a system registration error, carrying out on-line fine tuning on key external parameters, carrying out sight line prediction compensation, and adaptively adjusting the pose of a virtual object in a virtual display camera coordinate system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality technology, and in particular to a virtual-real fusion dynamic calibration method based on binocular eye movement tracking. BACKGROUND

[0002] Augmented reality (AR) technology realizes virtual-real fusion display by superimposing computer-generated virtual information on a real scene, and has been widely applied in education, industry, medical treatment and daily life. Wearable devices such as AR glasses and AR headsets, as important carriers of this technology, can present virtual objects to users without affecting their observation of the real environment, providing an immersive interactive experience.

[0003] In order to accurately present virtual objects in real space, AR devices must be registered in virtual-real space, that is, an accurate mapping relationship between the virtual image coordinate system and the real world coordinate system must be established through a virtual-real registration calibration method. Existing virtual-real fusion calibration methods mainly include the following categories: (1) Static calibration based on manual selection or visual features: users collect the correspondence between multiple image points and three-dimensional points through mouse clicking, gaze targeting and other methods, and calculate the coordinate conversion matrix. Although this method has a clear principle, it is tedious and dependent on manual operation, and its accuracy is limited by human eye discrimination ability and lacks dynamic adaptability.

[0004] (2) Automatic calibration method based on camera: a detection camera is placed at the position of the human eye, and the feature points are identified through the detection camera and the device tracking camera to collect images, so as to establish the coordinate relationship. Although this method does not require manual interaction, it ignores the influence of individual differences in human eye optics on registration accuracy, and when the user's head or AR device moves relatively, the calibration parameters are prone to drift and need to be frequently recalibrated. Moreover, most of the current calibration methods use the AR device's own camera to position and track to realize fusion calibration, which is not suitable for AR devices without self-positioning function or without built-in cameras.

[0005] Therefore, there is an urgent need for a virtual-real calibration method that is automated, high-precision, adaptive to individual differences and can correct drift in real time. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a virtual-real fusion dynamic calibration method based on binocular eye movement tracking.

[0007] The technical solution of the present application to solve the technical problem is to provide a virtual-real fusion dynamic calibration method based on binocular eye movement tracking, characterized in that the method comprises the following steps: Step 1, build a virtual-real fusion dynamic calibration system based on binocular eye movement tracking; The calibration system includes an external tracking sensor, an AR device, a left-eye eye-tracking camera, a right-eye eye-tracking camera, and a control and processing unit; The left and right eye-tracking cameras are fixed on the front sides of the optical transmissive display module of the AR device, facing the human eye; the external tracking sensor is placed in space and can always track the AR device during the calibration process; the external tracking sensor, the AR device, the left and right eye-tracking cameras are all connected to the control and processing unit. Step 2: Based on the binocular eye-tracking virtual-real fusion dynamic calibration system built in Step 1, establish a calibration coordinate system; The calibration coordinate system includes the world coordinate system. External tracking sensor coordinate system AR device coordinate system Eye-tracking camera coordinate system The display plane coordinate system of the display module of AR devices and virtual display camera coordinate system ; Step 3: Through offline joint calibration, based on the eye-tracking camera coordinate system constructed in Step 2. coordinate system with external tracking sensor Solve for the coordinate system of the eye-tracking camera. coordinate system with external tracking sensor Conversion relationship Then, based on the eye-tracking camera coordinate system... External tracking sensor coordinate system Conversion relationship and the coordinate system of the external tracking sensor for AR devices The posture of the lower Solve for the coordinate system of the eye-tracking camera. AR device coordinate system Conversion relationship ; Step 4: Eye-tracking parameter acquisition and user feature modeling: The left and right eye-tracking cameras respectively capture images of the user's left and right eyes, obtaining images of the user's left and right eyes; then, from these images, image processing and gaze modeling algorithms are used to establish gaze vectors for the user's left and right eyes. With display plane coordinate system The mapping relationship is then used to establish the coordinate system of the eye-tracking camera. With display plane coordinate system The conversion relationship between them; then based on the conversion relationship obtained in step 3 Thus, a display plane coordinate system is established. AR device coordinate system Conversion relationship ; Step 5: Based on the display plane coordinate system obtained in Step 4 AR device coordinate system Conversion relationship A virtual display camera model is constructed, and the intrinsic parameter matrix and extrinsic parameter transformation of the virtual display camera are solved to make the virtual display camera geometrically consistent with the user's real gaze behavior. Step 6: Based on the camera intrinsic and extrinsic parameters obtained in Step 5, set the intrinsic and extrinsic parameters for the virtual camera; then, based on the coordinate system of the external tracking sensor... Mark the object pose and construct the coordinate system of the external tracking sensor. With virtual display coordinate system The transformation relationship is used to transform the position of the marker to the virtual display camera coordinate system. Down; Step 7: Solve the problem as described in step 6. Based on achieving virtual-real alignment, the system registration error is evaluated, key extrinsic parameters are fine-tuned online, and line-of-sight prediction compensation is performed to adaptively adjust the virtual display camera coordinate system. The virtual object pose is adjusted to ensure the stability of virtual-real fusion under long-term wear and fast-moving scenarios.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention provides an integrated, automated, personalized, and real-time responsive virtual-real fusion calibration system that can adapt to individual gaze characteristics and posture changes in real time during the user's wearing process, greatly improving the immersive experience and interaction accuracy of AR devices.

[0009] (2) This invention integrates multi-source sensor coordinate systems to achieve module cascade fusion. This invention uses the AR device coordinate system as a transformation medium. By calibrating the binocular eye-tracking camera, external tracking sensor and AR device, it fuses the external tracking sensor coordinate system, eye-tracking camera coordinate system and display plane coordinate system to construct a mapping chain between the real world coordinate system, external tracking sensor coordinate system, eye-tracking camera coordinate system and virtual display camera coordinate system. This solves the problem of accurate calibration of AR devices without built-in positioning sensors and improves system consistency and scalability.

[0010] (3) This invention supports adaptive modeling of individual user features, thereby improving registration accuracy. By acquiring the left and right eye gaze vectors and eye center positions in real time through a binocular eye-tracking camera, and combining it with an optical model to dynamically estimate the physiological parameters of different individuals, an individual difference adaptive adjustment mechanism for the calibration system is realized, which is significantly better than the traditional method of using a unified model and monocular eye tracking.

[0011] (4) This invention achieves dynamic wearing posture drift correction and improves stability. Through dynamic registration and binocular gaze behavior analysis, when the user's head moves, eyes move or the wearing position changes slightly, the system can track the changes in gaze in real time and dynamically re-evaluate the relative transformation relationship between gaze projection direction and image display, and automatically update the coordinate mapping matrix without manual intervention, fundamentally solving the problem of "failure after drift" in static calibration methods.

[0012] (5) This invention supports an image rendering optimization mechanism driven by gaze behavior, which can identify different eye movement behaviors such as saccades, staring, and saccades. Based on the user's current gaze target area, the system can automatically adjust the virtual image rendering focus and perspective, significantly improving the perception quality of virtual content and providing support for the deployment of low-power AR head-mounted display devices. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a structural block diagram of the calibration system of the present invention; Figure 3 This is a schematic diagram of the various coordinate systems in the system of the present invention; Figure 4 The binocular gaze vector and display plane coordinate system described in this embodiment of the invention A schematic diagram for calibrating the projection relationship.

[0014] In the diagram, 1 is an external tracking sensor, 2 is an AR device, 3 is a left eye-tracking camera, 4 is a right eye-tracking camera, 5 is a control and processing unit, 6 is a virtual image displayed by a display module, and 7 is an actual object. Detailed Implementation

[0015] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the present invention.

[0016] This invention provides a method for dynamic calibration of virtual-real fusion based on binocular eye tracking (hereinafter referred to as the method), characterized by the following steps: Step 1: Build a virtual-real fusion dynamic calibration system based on binocular eye tracking (hereinafter referred to as the calibration system); The calibration system includes an external tracking sensor 1, an AR device 2, a left eye eye-tracking camera 3, a right eye eye-tracking camera 4, and a control and processing unit 5; The left eye movement tracking camera 3 and the right eye movement tracking camera 4 are respectively fixed on both sides of the front end of the optical transmission display module of the AR device 2, face the human eyes, are used for collecting the images of the eyes of the user, identifying the eyeball centers and the gaze directions and calculating the binocular interpupillary distance; the external tracking sensor 1 is placed in space, can always track the AR device 2 in the calibration process, is used for obtaining the three-dimensional pose of the AR device 2 and collecting the external environment information; the external tracking sensor 1, the AR device 2, the left eye movement tracking camera 3 and the right eye movement tracking camera 4 are in communication connection with the control and processing unit 5, the control and processing unit 5 is used for receiving the pose of the AR device 2 received by the external tracking sensor 1 and the images of the left eye movement tracking camera 3 and the right eye movement tracking camera 4, analyzing and calculating the pose conversion relationship, realizing the functions of image processing, coordinate calculation, projection transformation, dynamic updating and the like. The control and processing unit 5 can be realized by the software and / or hardware, can be composed of two or more physical entities, or can be composed of one physical entity.

[0017] Preferably, in step 1, the external tracking sensor 1 adopts a binocular optical positioner, is placed at a suitable height and is set at a pitch angle, so as to ensure that the tracking range can cover the calibration working interval.

[0018] Preferably, in step 1, the AR device 2 is an optical transmission near-eye display device, comprises an optical transmission display module (referred to as a display module), and the display module is a semi-transparent and semi-reflective optical system. Therefore, the human eye can observe the real world scene through the display module and can also see the virtual image from the image source of the display module, so as to observe the virtual-real combination effect.

[0019] Preferably, in step 1, the AR device 2 has a virtual display space and a real space, and the virtual-real fusion calibration enables the virtual display image and the actual real scene to be correctly registered and fused. In the embodiment, the virtual-real fusion calibration process specifically refers to obtaining the conversion relationship between the external tracking sensor coordinate system and the display plane coordinate system of the display module of the AR device 2, so as to realize the correct superposition display of the virtual-real image based on the conversion relationship.

[0020] Preferably, in step 1, the control and processing unit 5 is a computer device such as a notebook computer or a desktop computer for data analysis and calculation.

[0021] Step 2, based on the virtual-real fusion dynamic calibration system of binocular eye movement tracking established in step 1, a calibration coordinate system is established; The calibration coordinate system comprises a world coordinate system , an external tracking sensor coordinate system , an AR device coordinate system , an eye movement tracking camera coordinate system and a display plane coordinate system of the display module of the AR device 2 and virtual display camera coordinate system ; The three-dimensional coordinate system in which the actual object 7 is located is a world coordinate system , the origin of which is determined based on the location of the actual object 7; the three-dimensional coordinate system used by the external tracking sensor 1 to observe the display world is an external tracking sensor coordinate system , the origin of which is determined based on the location of the external tracking sensor 1; a marker is placed on the AR device 2, the marker has at least four calibration points, and the marker is fixedly connected to the AR device 2, and a three-dimensional coordinate system defined by the marker is an AR device coordinate system , the origin of the AR device coordinate system is determined based on the location of the AR device 2; the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4 are taken as a whole, and an eye movement tracking camera coordinate system is established with the camera optical center of the left-eye eye movement tracking camera 3 as the origin , the origin of which is determined based on its location; the two-dimensional coordinate system in which the display plane (referred to as the display plane) of the display module of the AR device 2 is located is the display plane coordinate system of the display module of the AR device 2 (referred to as the display plane coordinate system ), the origin of which is the center of the display plane; the three-dimensional coordinate system in which the virtual image 6 displayed by the display module is located is a virtual display camera coordinate system , the origin of which is defined by the display software.

[0022] Preferably, in step 2, the eye movement tracking camera coordinate system is constructed and the external tracking sensor coordinate system The specific steps are as follows: S21, place the AR device 2 in the tracking range of the external tracking sensor 1, place the calibration board in the common field of view of the left-eye eye movement tracking camera 3, the right-eye eye movement tracking camera 4 and the external tracking sensor 1, and simultaneously collect the left and right camera images of the external tracking sensor 1 and the images of the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4, and record the left and right eye movement tracking camera image frames , , Collect M≥25 frames of images, and the calibration board is in different spatial positions in each frame to ensure rich viewing angles and improve calibration robustness; Preferably, in step S21, the calibration board has an image for calibration, and the calibration board pattern has at least four calibration points. In this embodiment, the calibration board pattern is an asymmetric circular dot array, and the center points of the asymmetric circular dot array are taken as the calibration points.

[0023] S22, Based on the left and right eye tracking camera image frame pairs acquired in S21 ( , Image frame pairs acquired by the left and right cameras of the external tracking sensor 1. , ( ) The left eye-tracking camera 3 and the right eye-tracking camera 4 are calibrated to construct the eye-tracking camera coordinate system. The external tracking sensor 1 is calibrated to construct the external tracking sensor coordinate system. ; Preferably, in step S22, the left-eye eye-tracking camera 3 and the right-eye eye-tracking camera 4 are calibrated to construct an eye-tracking camera coordinate system. The steps are as follows: A221, based on the left and right eye tracking camera image frame pairs ( , The calibration points in the calibration board pattern in the images captured by the left and right eye-tracking cameras are detected, and the pixel coordinates of the calibration points in each frame of the left eye-tracking camera are obtained. : And the pixel coordinates of the calibration point in each frame of the right eye eye-tracking camera 4 : M represents the number of frames. A222, Based on the pixel coordinates of the calibration point and And the three-dimensional coordinates of the calibration point, respectively, to solve the coordinate system of the left eye tracking camera. Right eye eye tracking camera coordinate system With world coordinate system Conversion relationship ; A223. According to the conversion relationship The coordinate system of the right eye tracking camera was obtained. To the coordinate system of the left eye eye tracking camera rotation matrix Translation vector Determine the conversion relationship Using the left-eye eye-tracking camera 3 and the right-eye eye-tracking camera 4 as a binocular vision system, with the left-eye eye-tracking camera coordinate system as the coordinate system... For reference, the coordinate system of the right eye-tracking camera is... The coordinates in the table are transformed through the relationship. Unified association to Establish an eye-tracking camera coordinate system based on the left eye-tracking camera 3. (Right now ).

[0024] Preferably, in step S22, when calibrating the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4, the intrinsic matrix of the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4 and the radial and tangential distortion coefficients of the cameras can also be obtained, containing the physical parameters of the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4, which are used for subsequent left-eye and right-eye eye movement tracking camera image correction.

[0025] Preferably, in step S22, when calibrating the external tracking sensor 1 to construct the external tracking sensor coordinate system , the steps are as follows: B221, according to the left and right camera image frames collected by the external tracking sensor 1 , , the calibration points in the calibration board pattern in the images collected by the left and right cameras of the external tracking sensor 1 are detected, and the pixel coordinates of the calibration points in each frame of image of the left camera of the external tracking sensor 1 are obtained : and the pixel coordinates of the calibration points in each frame of image of the right camera of the external tracking sensor 1 are obtained : ; M is the number of frames; B222, according to the pixel coordinates of the calibration points and and the three-dimensional coordinates of the calibration points, the conversion relationship between the left camera coordinate system of the external tracking sensor 1 , the right camera coordinate system of the external tracking sensor 1 and the world coordinate system is solved respectively ; B223, according to the conversion relationship , the transformation matrix from the right camera coordinate system to the left camera coordinate system is obtained; taking the left-eye eye movement tracking camera 3 and the right-eye eye movement tracking camera 4 as a binocular vision system, taking the left camera coordinate system as a reference, the coordinates in the right camera coordinate system are unified and associated to through the conversion relationship , and the external tracking sensor coordinate system based on the left camera of the external tracking sensor 1 is established (i.e. ).

[0026] Preferably, in step S22, when calibrating the external tracking sensor 1, the intrinsic matrix of the left and right cameras of the external tracking sensor 1 and the radial and tangential distortion coefficients of the left and right cameras can also be obtained, containing the physical parameters of the left and right cameras of the external tracking sensor 1, which are used for subsequent left and right camera image correction of the external tracking sensor 1.

[0027] Step 3: Through offline joint calibration, based on the eye-tracking camera coordinate system constructed in Step 2. coordinate system with external tracking sensor Solve for the coordinate system of the eye-tracking camera. coordinate system with external tracking sensor Conversion relationship Then, based on the eye-tracking camera coordinate system... External tracking sensor coordinate system Conversion relationship And AR device 2 in the external tracking sensor coordinate system The posture of the lower Solve for the coordinate system of the eye-tracking camera. AR device coordinate system Conversion relationship ; Preferably, in step 3, since the left eye-tracking camera 3 and the right eye-tracking camera 4 are fixedly mounted on the AR device 2, the coordinate system of the eye-tracking cameras is fixed during the calibration process. AR device coordinate system Conversion relationship It remains unchanged.

[0028] Preferably, in step 3, the coordinate system of the eye-tracking camera is solved. coordinate system with external tracking sensor Conversion relationship The specific steps are as follows: A31. Based on the left eye tracking camera coordinate system in step 2. Right eye eye tracking camera coordinate system Transformation relationship with world coordinate system Obtain the coordinate system of the eye-tracking camera. With world coordinate system Conversion relationship: ; According to the left camera coordinate system of external tracking sensor 1 in step 2 The right camera coordinate system of external tracking sensor 1 Transformation relationship with world coordinate system The coordinate system of the external tracking sensor is obtained. With world coordinate system Conversion relationship: ; A32. Based on the conversion relationship and Establish the coordinate system of the eye-tracking camera coordinate system with external tracking sensor Conversion relationship: .

[0029] Preferably, in step 3, the coordinate system of the eye-tracking camera is solved. AR device coordinate system Conversion relationship The specific steps are as follows: B31. Using markers pasted on AR device 2, and employing image feature extraction and stereo matching algorithms, solve for the coordinate system of the external tracking sensor at each moment. The posture of AR device 2 ; B32, according to and Solve for the coordinate system of the eye-tracking camera at each moment. AR device coordinate system Conversion relationship ; B32, repeat B32 to obtain the eye-tracking camera coordinate system at multiple time points. AR device coordinate system Conversion relationship Then, the transformation relation... The Levenberg-Marquardt algorithm (LM algorithm) is used for global optimization to obtain the optimal solution. .

[0030] Step 4: Eye-tracking parameter acquisition and user feature modeling: The left eye-tracking camera 3 and the right eye-tracking camera 4 respectively capture images of the user's left and right eyes, obtaining images of the user's left and right eyes; then, from the user's left and right eye images, gaze vectors of the user's left and right eyes are established through image processing and gaze modeling algorithms. With display plane coordinate system The mapping relationship is then used to establish the coordinate system of the eye-tracking camera. With display plane coordinate system The conversion relationship between them; then based on the conversion relationship obtained in step 3 Thus, a display plane coordinate system is established. AR device coordinate system Conversion relationship ; Preferably, in step 4, a conversion relationship is established. The specific steps are as follows: S41, AR device 2 displays a calibration board pattern on its display plane. Left and right eye images are acquired using left-eye eye-tracking camera 3 and right-eye eye-tracking camera 4. A sequence of left and right eye images is then constructed based on these images. Finally, a sample pair is created by comparing the pixel coordinates of the display plane calibration points with the left and right eye image sequence. ; Preferably, in step S41, the specific steps of constructing the sample pair are as follows: S411, the user wears the AR device 2, displays a calibration board pattern in the display plane of the AR device 2, displays a preset calibration point with known center coordinates as a fixation point marker pattern at a time, and records the calibration point pixel coordinates of each calibration point in the display plane coordinate system ; ; S412, the user gazes at the marker point displayed in the display plane for 1-3 seconds; S413, synchronously collect the left eye movement tracking camera 3 and the right eye movement tracking camera 4 images to form an image pair; S414, repeat S411-S413 to obtain multiple sets of calibration point pixel coordinates and image pairs, exclude blinking samples, ensure that at least N≥10 sets of valid left and right eye image sequence pairs are recorded for each calibration point, and construct left and right eye image sequences .

[0031] S42, using the sample pairs collected in step S41 , construct an eye system model for each frame of left and right eye images of the user collected by the left eye movement tracking camera 3 and the right eye movement tracking camera 4; then extract the eyeball center of each frame of left and right eye images of the user based on the eye system model, and calculate the left eyeball center coordinates and the right eyeball center coordinates ; Preferably, in step S42, the steps of extracting the eyeball center of each frame of left and right eye images of the user based on the eye system model, and calculating the left eyeball center coordinates and the right eyeball center coordinates are as follows: S421, pre-process the left and right eye images of the user collected by the left eye movement tracking camera 3 and the right eye movement tracking camera 4 to obtain pre-processed images; Preferably, in step S421, the pre-processing includes image cropping, size normalization and grayscale processing.

[0032] S422, calculate the local grayscale sum of the pre-processed images according to the grid, detect the darkest region in the pre-processed image as the approximate region of the eyeball center, and take the darkest center in the approximate region of the eyeball center as the candidate pupil center ; S423, construct a fixed-size ROI region with the candidate pupil center , perform binaryzation in a multi-threshold manner to obtain multiple threshold maps Then, the connected component contours are extracted from each threshold map, and the extracted connected component contours are filtered to obtain valid contours; then, the outer contour with the largest area is retained as the candidate contour from the valid contours. Preferably, in step S423, the contour of the connected domain in each threshold map is as follows: Using area and aspect ratio constraints for filtering, the outline filtering conditions are defined as follows: ;Pick , .

[0033] S424. Perform ellipse fitting on the candidate contours extracted from each threshold image to obtain the ellipse center. Major and minor axes With rotation angle Then, based on the ellipse quality index... Select from all fitted ellipses The maximum value is the best-fit ellipse for the candidate contour.

[0034] Preferably, in step S424, the ellipse quality index is: Where Q is the elliptic coverage. R represents the number of pixels overlapping between the outline and the ellipse boundary, and R represents the outline-ellipse boundary matching ratio.

[0035] S425. Based on the best-fit ellipse obtained from multiple frames of images, take the center of the best-fit ellipse in each frame as an example. and rotation angle Determine the direction of the normal to the center of the eyeball as the direction of the minor axis of the optimal ellipse. And construct in-plane rays Then, randomly select N such rays from multiple frames of images and perform pairwise intersection calculations. For any two rays... The intersection point is calculated. Then, store the intersection points in the intersection point set Q, and take the average of the intersection point coordinates as the estimated coordinates of the image plane of the eyeball center. ; S426. For each set of eye image sequences, repeat steps S421 to S426, and calculate the image plane coordinates of the center of the left eyeball for each eye image. Image plane coordinates of the center of the right eyeball .

[0036] S43. For each frame of the image captured by the left and right eye-tracking cameras, based on the coordinates of the left eyeball center in each frame... and the coordinates of the center of the right eyeball Estimate the gaze vectors of the left and right eyes. , , used to indicate the estimated gaze direction of the left and right eyes; Preferably, in step S43, the eye center image plane coordinates O The specific steps of calculating the gaze vector are as follows: The specific steps of calculating the gaze vector are as follows: S431, the detected eye center pixel coordinates O The specific steps of calculating the gaze vector are as follows: Transformed into the field of view space and normalized, the corresponding coordinates in the field of view space are obtained , ; S432, the field of view space coordinates are mapped to the far clipping plane with a distance of from the eye movement tracking camera optical center and parallel to the camera image plane, and the mapped coordinates are , wherein , respectively, the width and height of the far clipping plane, representing the distance of the far clipping plane from the eye movement tracking camera optical center ; S433, according to the mapped three-dimensional coordinates, the unit directional vector from the eye movement tracking camera optical center to is constructed: ; ; S434, the eye is simplified as a sphere with the sphere center O and the radius r , and the sphere equation representing the eye is constructed ; then, according to the eye movement tracking camera optical center and the unit directional vector , the gaze ray is constructed; then, according to the relationship between the ray and the sphere surface , the intersection point of the gaze ray and the eye sphere model surface is calculated; then, the vector from the sphere center O to the intersection point is calculated and normalized to obtain the gaze vector .

[0037] Preferably, in step S434, the intersection point is calculated as follows: first, the relationship between the ray and the sphere surface intersection point is converted into the standard quadratic equation form: wherein ; then, the standard quadratic equation is solved to obtain the equation solution t : ; then, the positive larger solution is selected as the solution: ; then, according to the selected solution, the corresponding intersection point is obtained as: .

[0038] S44. Display the plane coordinate system for the multiple sets of left and right eye image sequences acquired in step S41 and the recorded fixation points. Pixel coordinates And calculate the left and right eye gaze vectors for each frame of the captured image. Construct sample pairs Then, based on the sample data, a neural network model is used to fit the nonlinear mapping relationship from the binocular gaze vector to the pixel coordinates of the display plane. ; Preferably, in step S44, the input to the neural network model is the image frames acquired by the left and right eye tracking cameras and the left and right eye gaze vectors estimated based on the image frames. The output is the display plane pixel coordinates of the gaze point. The loss function is defined as: .

[0039] S45. Using the left and right eye fixation vectors obtained in step S44 , The center of the left and right eyeballs Left and right eye fixation vectors , To display plane coordinate system The mapping relationship is used to solve the planar display coordinate system. AR device coordinate system Conversion relationship .

[0040] Preferably, in step S45, the planar display coordinate system is solved. AR device coordinate system Conversion relationship The specific steps are as follows: S451. Unify binocular eye-tracking data to the eye-tracking camera coordinate system Specifically: Transformation of the coordinates of the center of the left and right eyeballs: ; Left and right eye fixation vector transformation: ; S452, in the eye-tracking camera coordinate system Next, establish left and right eye fixation rays and solve for the fusion fixation point. Construct a fused gaze direction vector Then, using the eye-tracking camera coordinate system AR device coordinate system Conversion relationship Constructing a fusion gaze ray ; Preferably, in step S452, the step of constructing the fusion gaze ray is: S4521, constructing the fusion gaze ray S4521, constructing the fusion gaze ray Left eye gaze ray: ; Right eye gaze ray: ; Wherein, is the ray parameter; S4522, by minimizing the distance between the two rays, find its nearest point pair , satisfy the equation: ; Then solve the target is to minimize , the solution is :

[0041] The corresponding point pair is the nearest point pair , take the midpoint of the point pair as the fusion gaze point of the current frame : ; S4523, with the left eyeball center as the reference, construct the fusion gaze direction vector according to the fusion gaze point : ; S4524, use the conversion relationship between the eye movement tracking camera coordinate system and the AR device coordinate system , convert the fusion gaze vector and the left eyeball center to the AR device coordinate system , establish the fusion gaze ray , expressed as: , wherein ; S453, according to the fusion gaze ray constructed and the corresponding screen pixel coordinates, construct the equation group, solve the conversion relationship between the display plane coordinate system and the AR device coordinate system .

[0042] Preferably, in step S453, the conversion relationship between the display plane coordinate system and the AR device coordinate system is expressed as: ; Preferably, in step S453, the homogeneous coordinates of the user's screen pixel gaze point are: With fusion gaze rays satisfy: ; Preferably, in step S453, the LM algorithm is used to solve the system of equations to obtain the optimal transformation relationship. .

[0043] Step 5: Based on the display plane coordinate system obtained in Step 4 AR device coordinate system Conversion relationship A virtual display camera model is constructed, and the intrinsic parameter matrix and extrinsic parameter transformation of the virtual display camera are solved to make the virtual display camera geometrically consistent with the user's real gaze behavior. Preferably, in step 5, the specific steps for determining the intrinsic parameter matrix and extrinsic parameter transformation of the virtual display camera are as follows: S51. Based on the display system's viewing angle parameters, the image resolution is used to approximately estimate the intrinsic parameter matrix K of the virtual display camera:

[0044] in, The resolution of the virtual display image (unit: pixels). These are the field of view angles in the horizontal and vertical directions; S52. Based on the intrinsic parameter matrix K of the virtual display camera, align the fixed point i and treat its screen pixel coordinates as the pixel coordinates of the virtual display camera's image plane, constructing normalized coordinates. And virtually display the direction of the image plane ray within the camera. : , ; S53, Based on the display plane coordinate system AR device coordinate system Conversion relationship Construct an optimization problem and solve it to obtain the optimal solution. .

[0045] Preferably, in step S53, the optimal solution is obtained. The specific steps are as follows: S531, According to the conversion relationship For each calibration point Based on its coordinate system in AR devices coordinates below With in the display plane coordinate system coordinates below Establish a system of equations: ,in, is the first calibration point corresponding to the ray parameter; S532, according to the established equation group, an optimization objective function is constructed, and all parameters are combined as optimization variables , and an optimal solution is obtained by optimization solving , ; The optimization objective is .

[0046] Step 6, according to the camera internal and external parameters solved in step 5, the internal and external parameters of the virtual camera are set; and according to the marker pose in the external tracking sensor coordinate system , a conversion relationship between the external tracking sensor coordinate system and the virtual display coordinate system is constructed, so as to transform the marker position to the virtual display camera coordinate system . Preferably, in step 6, the specific steps of transforming the marker position to the virtual display camera coordinate system are as follows: S61, according to the virtual display camera internal and external parameters solved in step 5, the virtual display camera parameters are set in the virtual display camera coordinate system ; S62, a spatial marker containing marker points is placed in the actual scene, and according to the tracking result of the external tracking sensor 1, the pose of the marker points on the marker in the external tracking sensor coordinate system and the real-time pose of the AR device are calculated by using a visual algorithm; and a unified conversion relationship , , and between the virtual display camera coordinate system and the external tracking sensor coordinate system is constructed: ; ; S63, the spatial alignment of the virtual image and the real marker is realized by using the unified conversion relationship of step S62, and the coordinates of the object in the world coordinate system correspond to the coordinates of the virtual display camera coordinate system : ; .

[0047] Step 7, the calculation ​Based on achieving virtual-real alignment, the system registration error is evaluated, key extrinsic parameters are fine-tuned online, and line-of-sight prediction compensation is performed to adaptively adjust the virtual display camera coordinate system. The virtual object pose is adjusted to ensure the stability of virtual-real fusion under long-term wear and fast-moving scenarios.

[0048] Preferably, in step 7, the specific steps for adjusting the pose of the virtual object are as follows: S71. The user gazes at the marker point on the spatial marker, and simultaneously acquires images from the left and right eye-tracking cameras and the external tracking sensor 1; then, based on the calculation results of the external tracker, the coordinate system of the virtual display camera corresponding to the marker point is calculated. lower coordinate Based on the eye-tracking camera images, the screen gaze point is solved using the mapping relationship. Pixel coordinates, then based on the transformation relationship Solve for the pixel coordinates of the gaze point. Corresponding virtual display camera coordinate system lower coordinate ;according to and Real-time calculation of registration error and fusion error: ; S72. Using the registration error and fusion error from step S71, fine-tune key extrinsic parameters and update the coordinate transformation chain to achieve rapid adaptive calibration: When the system detects line-of-sight fluctuations exceeding the threshold, i.e., fusion drift, it pauses. Coordinates are updated, and the system prompts the user to gaze at the target point, utilizing the gaze vector at that moment. and corresponding reference point Perform a one-time, rapid fine-tuning optimization, with the following optimization goal: Update the temporary transformation matrix according to the optimization objective. , correct ; Preferably, in step S72, the calibration update can be initiated by the user, i.e., by inputting an update command to complete the update.

[0049] S73. To reduce the instantaneous misalignment caused by system latency in high-speed head and eye movement scenarios, a time-series-based gaze prediction mechanism is introduced. The predicted gaze direction is calculated using the gaze point and virtual space coordinates, enabling the rendering system to update virtual content in advance, thereby reducing the sense of virtual-real misalignment caused by dynamic latency. The rendering system performs viewpoint and depth compensation for virtual objects in advance, reducing the sense of misalignment caused by dynamic latency.

[0050] Preferably, in step S73, the system uses short-term sliding window prediction to predict the next gaze direction and calculates the virtual content projection in advance. The prediction formula is: .

[0051] The invention is applicable where not mentioned.

Claims

1. A method for dynamic calibration of virtual-real fusion based on binocular eye tracking, characterized in that, The method includes the following steps: Step 1: Build a virtual-real fusion dynamic calibration system based on binocular eye tracking; The calibration system includes an external tracking sensor (1), an AR device (2), a left eye eye tracking camera (3), a right eye eye tracking camera (4), and a control and processing unit (5). The left eye tracking camera (3) and the right eye tracking camera (4) are fixed on the front sides of the optical transmissive display module of the AR device (2), facing the human eye; the external tracking sensor (1) is placed in the space and can always track the AR device (2) during the calibration process; the external tracking sensor (1), the AR device (2), the left eye tracking camera (3) and the right eye tracking camera (4) are all connected to the control and processing unit (5) for communication. Step 2: Based on the binocular eye-tracking virtual-real fusion dynamic calibration system built in Step 1, establish a calibration coordinate system; The calibration coordinate system includes the world coordinate system. External tracking sensor coordinate system AR device coordinate system Eye-tracking camera coordinate system The display plane coordinate system of the display module of the AR device (2) and virtual display camera coordinate system ; Step 3: Through offline joint calibration, based on the eye-tracking camera coordinate system constructed in Step 2. coordinate system with external tracking sensor Solve for the coordinate system of the eye-tracking camera. coordinate system with external tracking sensor Conversion relationship Then, based on the eye-tracking camera coordinate system... External tracking sensor coordinate system Conversion relationship And AR devices (2) in the external tracking sensor coordinate system The posture of the lower Solve for the coordinate system of the eye-tracking camera. AR device coordinate system Conversion relationship ; Step 4: Eye-tracking parameter acquisition and user feature modeling: The left eye-tracking camera (3) and the right eye-tracking camera (4) respectively capture images of the user's left and right eyes to obtain the user's left and right eye images; then, from the user's left and right eye images, the user's gaze vectors are established through image processing and gaze modeling algorithms. With display plane coordinate system The mapping relationship is then used to establish the coordinate system of the eye-tracking camera. With display plane coordinate system The conversion relationship between them; then based on the conversion relationship obtained in step 3 Thus, a display plane coordinate system is established. AR device coordinate system Conversion relationship ; Step 5: Based on the display plane coordinate system obtained in Step 4 AR device coordinate system Conversion relationship A virtual display camera model is constructed, and the intrinsic parameter matrix and extrinsic parameter transformation of the virtual display camera are solved to make the virtual display camera geometrically consistent with the user's real gaze behavior. Step 6: Based on the camera intrinsic and extrinsic parameters obtained in Step 5, set the intrinsic and extrinsic parameters for the virtual camera; then, based on the coordinate system of the external tracking sensor... Mark the object pose and construct the coordinate system of the external tracking sensor. With virtual display coordinate system The transformation relationship is used to transform the position of the marker to the virtual display camera coordinate system. Down; Step 7: Solve the problem as described in step 6. Based on achieving virtual-real alignment, the system registration error is evaluated, key extrinsic parameters are fine-tuned online, and line-of-sight prediction compensation is performed to adaptively adjust the virtual display camera coordinate system. The virtual object pose is adjusted to ensure the stability of virtual-real fusion under long-term wear and fast-moving scenarios.

2. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 1, the left eye tracking camera (3) and the right eye tracking camera (4) are used to acquire images of the user's eyes, identify the center of the eyeball and the direction of gaze, and calculate the interpupillary distance; the external tracking sensor (1) is used to acquire the three-dimensional pose of the AR device (2) and acquire external environmental information; the control and processing unit (5) is used to receive the pose of the AR device (2) and the images of the left eye tracking camera (3) and the right eye tracking camera (4) received by the external tracking sensor (1), and to analyze and calculate the pose transformation relationship; In step 1, the external tracking sensor (1) is a binocular optical positioning instrument, which is placed at an appropriate height and has a pitch angle set to ensure that its tracking range can cover the calibrated working area. In step 1, the AR device (2) is an optical transmissive near-eye display device, which includes an optical transmissive display module; In step 1, the control and processing unit (5) is a computer device.

3. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 2, the three-dimensional coordinate system in which the actual object (7) is located is the world coordinate system. Its origin is determined based on the location of the actual object (7); the three-dimensional coordinate system used by the external tracking sensor (1) to observe and display the world is used as the coordinate system of the external tracking sensor. Its origin is determined based on the location of the external tracking sensor (1); a marker is placed on the AR device (2), the marker has at least four calibration points, the marker is fixed to the AR device (2), and the three-dimensional coordinate system defined by the marker is used as the coordinate system of the AR device. AR device coordinate system The origin is determined based on the location of the AR device (2); the left eye tracking camera (3) and the right eye tracking camera (4) are treated as a whole, and the eye tracking camera coordinate system is established with the optical center of the left eye tracking camera (3) as the origin. Its origin is determined based on its location; the two-dimensional coordinate system of the display plane of the display module of AR device (2) is the display plane coordinate system of the display module of AR device (2). Its origin is the center of the display plane; the three-dimensional coordinate system in which the virtual image (6) displayed by the display module is located is the virtual display camera coordinate system. Its origin is defined by the display software.

4. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 2, the coordinate system of the eye-tracking camera is constructed. coordinate system with external tracking sensor The specific steps are as follows: S21. Place the AR device (2) within the tracking range of the external tracking sensor (1), place the calibration board within the common field of view of the left eye tracking camera (3), the right eye tracking camera (4), and the external tracking sensor (1), and simultaneously acquire the left and right camera images of the external tracking sensor (1) and the images of the left eye tracking camera (3) and the right eye tracking camera (4), and record the left and right eye tracking camera image frames. , Record the left and right camera image frames of the external tracking sensor (1). , Acquire M≥25 frames of images, with the calibration board in a different spatial position in each frame; S22, Based on the left and right eye tracking camera image frame pairs acquired in S21 ( , Image frame pairs acquired by the left and right cameras of the external tracking sensor (1) and the external tracking sensor (1). , The left eye-tracking camera (3) and the right eye-tracking camera (4) are calibrated to construct the eye-tracking camera coordinate system. The external tracking sensor (1) is calibrated to construct the external tracking sensor coordinate system. ; In step S22, the left eye-tracking camera (3) and the right eye-tracking camera (4) are calibrated to construct the eye-tracking camera coordinate system. The steps are as follows: A221, based on the left and right eye tracking camera image frame pairs ( , ), detect the calibration points in the calibration board pattern in the images acquired by the left and right eye tracking cameras, and obtain the pixel coordinates of the calibration points in each frame of the left eye tracking camera (3). : And the pixel coordinates of the calibration point in each frame of the right eye eye-tracking camera (4) : M represents the number of frames. A222, Based on the pixel coordinates of the calibration point and And the three-dimensional coordinates of the calibration point, respectively, to solve the coordinate system of the left eye tracking camera. Right eye eye tracking camera coordinate system With world coordinate system Conversion relationship ; A223. According to the conversion relationship The coordinate system of the right eye tracking camera was obtained. To the coordinate system of the left eye eye tracking camera rotation matrix Translation vector Determine the conversion relationship Using the left eye-tracking camera (3) and the right eye-tracking camera (4) as a binocular vision system, with the left eye-tracking camera coordinate system as the coordinate system. For reference, the coordinate system of the right eye-tracking camera is... The coordinates in the table are transformed through the relationship. Unified association to Establish an eye-tracking camera coordinate system based on the left eye-tracking camera (3). ; In step S22, when calibrating the left eye tracking camera (3) and the right eye tracking camera (4), the intrinsic parameter matrix of the left eye tracking camera (3) and the right eye tracking camera (4) and the radial and tangential distortion coefficients of the camera can also be obtained, including the physical parameters of the left eye tracking camera (3) and the right eye tracking camera (4), which are used for subsequent image correction of the left and right eye tracking cameras; In step S22, the external tracking sensor (1) is calibrated to construct the external tracking sensor coordinate system. The steps are as follows: B221, Based on the image frame pairs acquired by the left and right cameras of the external tracking sensor (1) , The calibration points in the calibration board pattern in the images acquired by the left and right cameras of the external tracking sensor (1) are detected, and the pixel coordinates of the calibration points in each frame of the left camera of the external tracking sensor (1) are obtained. : The pixel coordinates of the calibration point in each frame of the right camera image of the external tracking sensor (1) : M represents the number of frames. B222, Based on the pixel coordinates of the calibration point and And the three-dimensional coordinates of the calibration point, respectively solve the coordinate system of the external tracking sensor (1) and the left camera. External tracking sensor (1) Right camera coordinate system With world coordinate system Conversion relationship ; B223, According to the conversion relationship Obtain the right camera coordinate system To the left camera coordinate system Transformation matrix Using the left eye-tracking camera (3) and the right eye-tracking camera (4) as a binocular vision system, with the left camera coordinate system as the coordinate system. For reference, the right camera coordinate system The coordinates in the table are transformed through the relationship. Unified association to Establish an external tracking sensor coordinate system with the left camera of the external tracking sensor (1) as the reference. ; In step S22, when calibrating the external tracking sensor (1), the intrinsic parameter matrix of the left and right cameras of the external tracking sensor (1) and the radial and tangential distortion coefficients of the left and right cameras can also be obtained, including the physical parameters of the left and right cameras of the external tracking sensor (1), which are used for subsequent image correction of the left and right cameras of the external tracking sensor (1).

5. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 3, the coordinate system of the eye-tracking camera is solved. coordinate system with external tracking sensor Conversion relationship The specific steps are as follows: A31. Based on the coordinate system of the left eye tracking camera in step 2. Right eye eye tracking camera coordinate system Transformation relationship with world coordinate system Obtain the coordinate system of the eye-tracking camera. With world coordinate system Conversion relationship: ; According to the left camera coordinate system of the external tracking sensor (1) in step 2 The right camera coordinate system of the external tracking sensor (1) Transformation relationship with world coordinate system The coordinate system of the external tracking sensor is obtained. With world coordinate system Conversion relationship: ; A32. According to the conversion relationship and Establish the coordinate system of the eye-tracking camera coordinate system with external tracking sensor Conversion relationship: ; In step 3, the coordinate system of the eye-tracking camera is solved. AR device coordinate system Conversion relationship The specific steps are as follows: B31. Using the markers pasted on the AR device (2), the coordinate system of the external tracking sensor at each moment is solved using image feature extraction algorithm and stereo matching algorithm. The posture of the AR device (2) ; B32, according to and Solve for the coordinate system of the eye-tracking camera at each moment. AR device coordinate system Conversion relationship ; B32, repeat B32 to obtain the eye-tracking camera coordinate system at multiple time points. AR device coordinate system Conversion relationship Then, the transformation relation... The LM algorithm is used for global optimization to obtain the optimal solution. .

6. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 4, establish the conversion relationship. The specific steps are as follows: S41. The display plane of the AR device (2) displays the calibration board pattern. The left and right eye images are acquired using the left eye tracking camera (3) and the right eye tracking camera (4). The left and right eye image sequences are then constructed based on the left and right eye images. Finally, the pixel coordinates of the display plane calibration point are compared with the left and right eye image sequences to construct sample pairs. ; S42. Using the samples collected in step S41 to... For each frame of user's left and right eye images captured by the left eye tracking camera (3) and the right eye tracking camera (4), a human eye system model is constructed; then, based on the human eye system model, the eyeball center of each frame of user's left and right eye images is extracted, and the coordinates of the left eyeball center are calculated respectively. and the coordinates of the center of the right eyeball ; S43. For each frame of the image captured by the left and right eye-tracking cameras, based on the coordinates of the left eyeball center in each frame... and the coordinates of the center of the right eyeball Estimate the gaze vectors of the left and right eyes. , , used to indicate the estimated gaze direction of the left and right eyes; S44. Display the plane coordinate system for the multiple sets of left and right eye image sequences acquired in step S41 and the recorded fixation points. Pixel coordinates And calculate the left and right eye gaze vectors for each frame of the captured image. Construct sample pairs ; Then, based on the sample data, a neural network model is used to fit the nonlinear mapping relationship from the binocular gaze vector to the display plane pixel coordinates. ; S45. Using the left and right eye fixation vectors obtained in step S44 , The center of the left and right eyeballs Left and right eye fixation vectors , To display plane coordinate system The mapping relationship is used to solve the planar display coordinate system. AR device coordinate system Conversion relationship .

7. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 6, characterized in that, In step S41, the specific steps for constructing sample pairs are as follows: S411. The user puts on the AR device (2). A calibration board pattern is displayed on the display plane of the AR device (2). A preset marker point with known center coordinates is displayed one at a time as the gaze point marker pattern. The coordinate system of each calibration point in the display plane is recorded. Pixel coordinates of calibration points in ; S412. The user gazes at the marker point displayed on the display plane for 1 to 3 seconds; S413. Simultaneously acquire images from the left eye-tracking camera (3) and the right eye-tracking camera (4). These form image pairs; S414. Repeat S411~S413 to obtain multiple sets of calibration point pixel coordinates and image pairs. Exclude blink samples and ensure that each calibration point records at least N≥10 sets of valid left and right eye image sequence pairs to construct left and right eye image sequences. ; In step S42, the eyeball centers of the user's left and right eyes in each frame are extracted based on the human eye system model, and the coordinates of the left eyeball center are calculated respectively. and the coordinates of the center of the right eyeball The steps are as follows: S421. Preprocess the user's left and right eye images captured by the left eye tracking camera (3) and the right eye tracking camera (4) to obtain preprocessed images; In step S421, preprocessing includes image cropping, size normalization, and grayscale processing; S422. Calculate the local grayscale sum of the preprocessed image according to the grid, detect the darkest region in the preprocessed image, and use it as the approximate center region of the eyeball. Then, use the darkest center in the approximate center region of the eyeball as the candidate pupil center. ; S423, using pupil candidate center Construct a fixed-size ROI region and perform binarization using a multi-threshold method to obtain multiple threshold maps. Then, the connected component contours are extracted from each threshold map, and the extracted connected component contours are filtered to obtain valid contours; then, the outer contour with the largest area is retained as the candidate contour from the valid contours. In step S423, the contours of the connected domains in each threshold map are as follows: Using area and aspect ratio constraints for filtering, the outline filtering conditions are defined as follows: ; S424. Perform ellipse fitting on the candidate contours extracted from each threshold image to obtain the ellipse center. Major and minor axes With rotation angle Then, based on the ellipse quality index... Select from all fitted ellipses The maximum value is the best-fit ellipse for the candidate contour; In step S424, the ellipse quality index is: Where Q is the elliptic coverage. R is the number of pixels overlapping between the outline and the ellipse boundary, and R is the matching ratio between the outline and the ellipse boundary. S425. Based on the best-fit ellipse obtained from multiple frames of images, take the center of the best-fit ellipse in each frame as an example. and rotation angle Determine the direction of the normal to the center of the eyeball as the direction of the minor axis of the optimal ellipse. And construct in-plane rays Then, randomly select N such rays from multiple frames of images and perform pairwise intersection calculations. For any two rays... The intersection point is calculated. Then, store the intersection points in the intersection point set Q, and take the average of the intersection point coordinates as the estimated coordinates of the image plane of the eyeball center. ; S426. For each set of eye image sequences, repeat steps S421 to S426, and calculate the image plane coordinates of the center of the left eyeball for each eye image. Image plane coordinates of the center of the right eyeball ; In step S43, based on the eyeball center image plane coordinates O Calculate gaze vector The specific steps are as follows: S431, The detected eyeball center pixel coordinates O By image size Transform the coordinates to the view space and normalize them to obtain the corresponding view space coordinates. , ; S432, Map the field of view spatial coordinates to the optical center of the distance eye-tracking camera. Distance is And on the far clipping plane parallel to the camera image plane, the mapped coordinates are... , ,in, , These are the width and height of the cut surface, respectively. Indicates the distance of the far clipping plane from the optical center of the eye-tracking camera. The distance; S433. Based on the mapped three-dimensional coordinates, use eye-tracking camera optical center... Using the origin as the optical center, construct a path from the eye-tracking camera's optical center to... unit direction vector : ; S434, Simplify the eyeball to its center. O , radius is r Construct a spherical equation to represent the eyeball. Then, based on the optical center of the eye-tracking camera... With solving for the unit direction vector Constructing gaze rays Then, based on the relationship between the ray and the sphere... Calculate the intersection point of the gaze ray with the surface of the eyeball model. Then calculate from the center of the ball. O to the intersection The vector is normalized to obtain the gaze vector. ; In step S434, the intersection points are solved. Specifically: First, determine the relationship between the intersection points of the ray and the sphere. Convert to standard quadratic equation form: ,in Then, solve the standard quadratic equation to obtain the solution. t : Then select the larger positive solution. Solution: Then, based on the selected solution, substitute the corresponding intersection point as follows: ; In step S44, the input to the neural network model is the image frames captured by the left and right eye tracking cameras and the left and right eye gaze vectors estimated based on the image frames. The output is the display plane pixel coordinates of the gaze point. The loss function is defined as: ; In step S45, the planar display coordinate system is solved. AR device coordinate system Conversion relationship The specific steps are as follows: S451. Unify binocular eye-tracking data to the eye-tracking camera coordinate system Specifically: Transformation of the coordinates of the center of the left and right eyeballs: ; Left and right eye fixation vector transformation: ; S452, in the eye-tracking camera coordinate system Next, establish left and right eye fixation rays and solve for the fusion fixation point. Construct a fused gaze direction vector Then, using the eye-tracking camera coordinate system AR device coordinate system Conversion relationship Constructing a fusion gaze ray ; In step S452, a fusion gaze ray is constructed. The steps are as follows: S4521. Constructing fusion fixation rays for both left and right eyes: Left eye fixation beam: ; Right eye fixation beam: ; in, For ray parameters; S4522. Find the closest pair of points by minimizing the distance between two rays. Satisfies the equation: ； Then the objective is to minimize Solving for the given information yields the following results. : ， Corresponding point pairs are the closest point pairs. The midpoint of the point pair is taken as the fusion gaze point of the current frame. : ; S4523, with the center of the left eyeball For reference, based on the fusion gaze point Constructing the fused gaze direction vector: ; S4524, Using the coordinate system of an eye-tracking camera AR device coordinate system Conversion relationship This will fuse the gaze vector with the center of the left eye. Transform to AR device coordinate system In the middle, establish fusion gaze rays , represented as: ,in ; S453, Based on the construction of fusion gaze rays Using the corresponding screen pixel coordinates, construct a system of equations and solve for the display plane coordinate system. AR device coordinate system Conversion relationship ; In step S453, the planar coordinate system will be displayed. AR device coordinate system Conversion relationship Represented as: ; In step S453, the homogeneous coordinates of the user's screen pixel gaze point are: With fusion gaze rays satisfy: ; In step S453, the LM algorithm is used to solve the system of equations to obtain the optimal transformation relation. .

8. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 5, the specific steps for determining the intrinsic parameter matrix and extrinsic parameter transformation of the virtual display camera are as follows: S51. Based on the display system's viewing angle parameters, the image resolution is used to approximately estimate the intrinsic parameter matrix K of the virtual display camera: ， in, For virtual display image resolution, These are the field of view angles in the horizontal and vertical directions; S52. Based on the intrinsic parameter matrix K of the virtual display camera, align the fixed point i and treat its screen pixel coordinates as the pixel coordinates of the virtual display camera's image plane, constructing normalized coordinates. And virtually display the direction of the image plane ray within the camera. : ， ； S53, Based on the display plane coordinate system AR device coordinate system Conversion relationship Construct an optimization problem and solve it to obtain the optimal solution. ; In step S53, the optimal solution is found. The specific steps are as follows: S531, According to the conversion relationship For each calibration point Based on its coordinate system in AR devices coordinates below With in the display plane coordinate system coordinates below Establish a system of equations: ,in, It is the first Ray parameters corresponding to each calibration point; S532. Based on the established system of equations, construct the objective function, and then combine all parameters into optimization variables. The optimal solution is obtained by optimization. , ; The optimization goal is .

9. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 6, the marker position is transformed to the virtual display camera coordinate system. The specific steps are as follows: S61. Based on the virtual display camera intrinsic and extrinsic parameters obtained in step 5, in the virtual display camera coordinate system... Configure the virtual display to show camera parameters; S62. Place spatial markers containing marker points in the actual scene, and use a visual algorithm to calculate the coordinates of the marker points on the markers in the coordinate system of the external tracking sensor based on the tracking results of the external tracking sensor (1). neutral posture Real-time pose of AR devices Then utilize the transformation relationship , and Construct a virtual display camera coordinate system Relative to the external tracking sensor coordinate system Unified transformation relationship : ; S63. Utilize the unified transformation relationship from step S62. To achieve spatial alignment between virtual images and real markers in the world coordinate system. Object coordinates in lower space Corresponding virtual display camera coordinate system lower coordinate for: .

10. The method for dynamic calibration of virtual-real fusion based on binocular eye tracking according to claim 1, characterized in that, In step 7, the specific steps for adjusting the pose of the virtual object are as follows: S71. The user gazes at the marker point on the spatial marker, and simultaneously acquires the left and right eye tracking camera images and the external tracking sensor (1) image; then, based on the external tracker's calculation results, the coordinate system of the virtual display camera corresponding to the marker point is calculated. lower coordinate Based on the eye-tracking camera images, the screen gaze point is solved using the mapping relationship. Pixel coordinates, then based on the transformation relationship Solve for the pixel coordinates of the gaze point. Corresponding virtual display camera coordinate system lower coordinate ;according to and Real-time calculation of registration error and fusion error: ; S72. Using the registration error and fusion error from step S71, fine-tune key extrinsic parameters and update the coordinate transformation chain to achieve rapid adaptive calibration: When the system detects line-of-sight fluctuations exceeding the threshold, i.e., fusion drift, it pauses. Coordinates are updated, and the system prompts the user to gaze at the target point, utilizing the gaze vector at that moment. and corresponding reference point Perform a one-time, rapid fine-tuning optimization, with the following optimization goal: Update the temporary transformation matrix according to the optimization objective. , correct ; S73. To reduce the instantaneous misalignment caused by system latency in high-speed head and eye movement scenarios, a time-series-based gaze prediction mechanism is introduced. The predicted gaze direction is calculated using the gaze point and virtual space coordinates, enabling the rendering system to update virtual content in advance, thereby reducing the sense of virtual-real misalignment caused by dynamic latency. The rendering system performs viewpoint and depth compensation for virtual objects in advance, reducing the sense of misalignment caused by dynamic latency. In step S73, the system uses short-term sliding window prediction to predict the next gaze direction and pre-calculates the virtual content projection. The prediction formula is as follows: .

Images