Near-eye AR setup with implicit adaptive calibration method, system, device, and medium

Abstract

The application discloses a near-eye AR device viewing implicit adaptive calibration method, system, equipment and medium, and the near-eye AR device viewing implicit adaptive calibration method comprises line-of-sight estimation, gesture perception and classification, hand-eye space-time consistency analysis and sample library construction and parameter adaptive updating. The application aims at the technical problem that a real gaze label is difficult to be reliably acquired in a near-eye AR open interaction scene, innovatively models hand-eye coupling differences in a gesture type conditional manner, uses speed Pearson correlation after mutual correlation time delay compensation as a sample access threshold, selects representative frames in a stable gesture event interval to construct multi-dimensional quality weighted calibration samples, and introduces harmful update detection and parameter rollback mechanism, continuously optimizes personalized gaze estimation parameters based on a PCCR three-dimensional eyeball geometric model without any explicit calibration operation of a user, realizes long-term stable online adaptive gaze calibration.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and wearable computing technology, and in particular to an implicit adaptive calibration system and method for gaze of near-eye augmented reality devices based on hand-eye coordination behavior priors. Background Technology

[0002] Eye tracking is the core perceptual capability for near-eye AR devices to achieve gaze-based interaction. Current mainstream solutions fall into two categories: end-to-end methods based on deep learning and geometric methods based on PCCR principles and 3D eye geometry models. The former offers high accuracy but requires significant computational power and has weak cross-user generalization; the latter is computationally efficient, highly interpretable, and suitable for low-cost, lightweight deployment.

[0003] The accuracy of geometric methods is highly dependent on individualized calibration parameters: by detecting the reflective point formed by the infrared LED on the cornea and the pupil center, the three-dimensional gaze direction is estimated from the PCCR geometric relationship, and then projected onto the screen coordinate system through a polynomial mapping function. The calibration parameters degrade with wear offset and changes in user state. Traditional methods rely on frequent multi-point explicit calibration (usually requiring viewing 9 to 13 screen points), which severely disrupts the continuity of interaction.

[0004] In recent years, implicit calibration methods have begun to utilize natural user interactions (clicks, smooth tracking, hand-eye coordination) as weak supervision signals. However, existing methods still suffer from the following major drawbacks: Defect 1: Lack of system modeling for hand-eye consistency conditions. Existing methods assume that hand-eye consistency always holds, failing to distinguish the differences in coupling strength under different gesture types, leading to the inclusion of low-quality samples and parameter drift.

[0005] Defect 2: Lack of a learnable update decision mechanism. Existing methods use fixed thresholds or rules to determine when to update, which cannot dynamically adapt to individual user differences and interaction scenarios, resulting in insufficient robustness.

[0006] Deficiency 3: Limited sample quality assessment dimensions. Existing methods mostly use fixed time windows or motion amplitude thresholds as sample quality indicators, without comprehensively utilizing multi-dimensional information such as velocity correlation, gesture stability, and temporal alignment.

[0007] Defect 4: Lack of anomaly detection and parameter rollback mechanisms. When erroneous samples cause the system to deviate, there is a lack of online evaluation and recovery capabilities, posing a risk of irreversible degradation.

[0008] Defect 5: Reliable acquisition of true gaze labels in near-eye AR scenarios. Unlike desktop screen interaction, the interactive objects of near-eye AR devices are not GUI elements fixed in known screen positions. The true gaze targets of users in natural interactions are often not directly observable. Existing implicit calibration methods based on the salience of clicked interactive elements or screen content rely on known target positions as supervision labels, which are not suitable for the open interactive scenarios of near-eye AR. Therefore, how to reliably filter out high-confidence calibration pseudo-labels from natural interaction behavior is the core technical challenge that distinguishes near-eye AR implicit calibration from other scenarios. Summary of the Invention

[0009] The technical problem this invention aims to solve is to address the shortcomings of the existing technologies by providing an implicit adaptive gaze calibration method, system, device, and medium for near-eye AR devices based on hand-eye coordination behavior priors. Its core technological contribution lies in: modeling hand-eye coupling differences under different gestures through gesture type conditionalization; using the velocity Pearson correlation after cross-correlation delay compensation as the real-time sample entry threshold; constructing multi-dimensional quality-weighted calibration samples by taking representative frames within stable gesture event intervals; and ensuring the long-term stability of the calibration system under uncertain observation conditions through harmful update detection and parameter rollback mechanisms. Thus, continuous online adaptive gaze calibration for open interactive scenarios in near-eye AR is achieved without requiring any explicit calibration operations from the user.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: This invention first provides an implicit adaptive calibration method for near-eye AR devices based on hand-eye coordination behavior priors, comprising the following steps: Step 1: Gaze estimation, including gaze estimation and gesture type recognition; Gaze estimation: The original gaze direction vector is obtained by acquiring infrared images of the eye through a near-eye camera, based on the detection of the pupil center and corneal reflection point and the three-dimensional eye geometry calculation. The original gaze direction vector is then mapped to the calibrated gaze direction through the current quadratic polynomial mapping function, which is used as the gaze estimation result for this frame. The calibration parameters of the quadratic polynomial mapping function are initially set to default values ​​and are replaced with the latest parameters each time the decision is successfully updated in step 4. Gesture type recognition: The system captures hand images using the front-facing RGB camera, extracts hand features, identifies the current gesture type, and extracts the three-dimensional coordinates of the index fingertip. Step 2: Gesture perception and classification: Based on the recognized current gesture type, the index fingertip velocity sequence and gaze direction velocity sequence are extracted separately. When the fingertip velocity is lower than the velocity threshold and continuously exceeds the time threshold, the corresponding gesture type is determined as a stable candidate gesture event. Based on cross-correlation analysis, the optimal time delay of the index fingertip velocity sequence and gaze direction velocity sequence is estimated and compensated, and the signals from the near-eye camera and the front-facing RGB camera are aligned in time. After time alignment, the Pearson correlation coefficient C is calculated within a sliding window. The corresponding differential threshold Thr_C is selected according to the current gesture type. If C ≥ Thr_C, the hand-eye spatiotemporal consistency condition is satisfied; otherwise, the current frame is discarded. Step 3: Hand-eye spatiotemporal consistency analysis: When both stable candidate gesture events and hand-eye spatiotemporal consistency conditions are met, the frame with the highest correlation coefficient in the event interval is taken as the representative frame. A weighted calibration sample pair of pupil feature vector and gaze direction pseudo-true value is constructed (pseudo-true value is obtained by normalizing the three-dimensional coordinates of the index fingertip and the center of the eyeball on the same side). Each sample is assigned a comprehensive quality weight and stored in the sliding window sample library. Step 4: Sample library construction and adaptive parameter update: When the cumulative number of samples in the sliding window sample library reaches the trigger threshold, the calibration parameters of the quadratic polynomial mapping model in step 1 are updated based on the weighted least squares method. The weighted average angle error before and after the update is calculated using the highest quality sample in the sliding window sample library as the validation set. If the increase in the weighted average angle error exceeds the error threshold, it is determined to be a harmful update, and the calibration parameters are automatically rolled back to the previous version. The trigger samples in this batch are removed and the process enters the protection waiting period. Otherwise, the update is accepted and the cycle from step 1 to step 4 continues.

[0011] This invention also provides an implicit adaptive calibration system for near-eye AR devices based on hand-eye coordination behavior priors, comprising: The gaze estimation subsystem acquires infrared images of the user's eyes using a near-eye camera and calculates the original gaze direction vector; The gesture perception subsystem detects stable pointing events using the front-facing RGB camera; The hand-eye spatiotemporal consistency analysis subsystem extracts the gaze direction velocity sequence and the fingertip movement velocity sequence respectively, and analyzes and judges the hand-eye consistency condition. The implicit calibration sample construction and quality assessment subsystem, when both stable pointing event and consistency conditions are met, takes the frame with the largest correlation coefficient in the event interval as the representative frame, constructs a calibration sample pair of pupil feature vector and gaze direction pseudo-true value (pseudo-true value is obtained by normalizing the three-dimensional coordinates of the index fingertip and the center of the ipsilateral eyeball), and assigns a comprehensive quality weight that integrates hand-eye consistency and fingertip stability. The personalized calibration model training and online update subsystem updates the parameters of the quadratic polynomial mapping model based on the weighted least squares method when the cumulative number of high-value samples reaches the trigger threshold, and feeds back the updated calibration parameters to the line-of-sight estimation subsystem. The decision-making mechanism subsystem is updated to calculate the conditional trust score of gesture type. The weighted angle error change before and after the update is evaluated using high-quality samples as the validation set. If the error increment after the update exceeds the error threshold, it is judged as a harmful update, and the system is automatically rolled back to the previous version of parameters and enters a protection waiting period to prevent irreversible parameter degradation.

[0012] The present invention also provides an interactive device, comprising: One or more processors; Memory, used to store one or more programs; Near-eye camera, used to capture infrared images of the eye; Front-facing RGB camera for capturing hand images; When the one or more programs are executed by the one or more processors, the device performs the steps of the method as described above.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0014] Compared with the prior art, the beneficial effects of the present invention are: The zero-disruption implicit calibration experience requires no explicit calibration operation throughout the process. Calibration is integrated into natural directional interaction, making it suitable for continuous AR wearable scenarios around the clock.

[0015] Gesture-conditional fine-grained sample management models the hand-eye coupling strength for different gesture types and combines a multi-dimensional quality weighting strategy to effectively avoid low-quality samples from contaminating model parameters.

[0016] The interpretable geometric model has strong compatibility. Based on the PCCR geometric method, the computational cost of incremental update of quadratic polynomial parameters is extremely low (closed-form solution), making it suitable for real-time operation of low-power AR glasses.

[0017] Stable convergence with rollback is guaranteed, and trust score evaluation, anomaly detection and parameter rollback mechanisms ensure the long-term stability of the system under uncertain observation conditions.

[0018] Rapid and personalized adaptation: Experimental verification shows that only 5 to 15 valid pointing gesture calibration samples are needed to complete the first personalized parameter convergence, which is far lower than the threshold of traditional explicit calibration. Attached Figure Description

[0019] Figure 1The overall system architecture diagram shows the hierarchical relationship of the six main functional subsystems of this invention and the online continuous update feedback loop; Figure 2 : Flowchart of PCCR 3D eye geometry model and gaze estimation algorithm; where (a) is the PCCR 3D eye geometry model, labeled with near-eye camera, infrared LED, corneal reflective point, pupil center, corneal curvature center, eyeball center and optical axis / visual axis angle; (b) is the gaze estimation algorithm flowchart; Figure 3 The flowchart of hand-eye spatiotemporal consistency analysis and implicit sample screening shows the branch logic of velocity sequence extraction, gesture type conditional branching, Pearson correlation calculation and threshold judgment, as well as the generation of weighted effective sample pairs and sample discarding. Figure 4 The flowchart of the online calibration update decision mechanism shows the effective and stable gesture event detection, weighted sample construction, multinomial weighted regression parameter update, anomaly detection and parameter rollback. Figure 5 The overall flowchart of the calibration method of this invention shows the processing flow of four steps: step 1 gaze estimation, step 2 gesture perception and classification, step 3 hand-eye spatiotemporal consistency analysis, and step 4 sample library construction and parameter adaptive update, as well as the implicit closed-loop feedback relationship of continuous update of parameter θ*. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings: This invention relates to an implicit adaptive calibration system for near-eye AR devices based on hand-eye coordination behavior priors, such as... Figure 1 As shown, it includes six software subsystems: gaze estimation subsystem, gesture perception subsystem, hand-eye spatiotemporal consistency analysis subsystem, implicit calibration sample construction and quality assessment subsystem, personalized calibration model training and online update subsystem, and update decision mechanism subsystem.

[0021] The hardware perception layer includes a near-eye infrared camera, a front-facing RGB camera, and an IMU inertial sensor. Six software subsystems operate in a pipeline: the gaze estimation subsystem (module 1) and the gesture perception subsystem (module 2) acquire perception data in parallel; the hand-eye spatiotemporal consistency analysis subsystem (module 3) fuses the two datasets for correlation analysis; the implicit calibration sample construction and quality assessment subsystem (module 4) selects high-quality samples; the personalized calibration model training and online update subsystem (module 5) performs parameter updates; and the update decision mechanism subsystem (module 6) controls the update timing and ensures stability. The updated calibration parameters are fed back to the gaze estimation subsystem, forming a closed loop of continuous online updates.

[0022] (a) Line of sight estimation subsystem The gaze estimation subsystem is the fundamental sensing front-end of the adaptive calibration system of this invention. The gaze estimation subsystem includes a pupil center detection module, a three-dimensional eyeball geometry modeling module, a three-dimensional gaze direction calculation module, and a polynomial mapping calibration module.

[0023] 1.1 Pupil Center Detection Module: The pupil center detection module is used to calculate the pixel coordinates and diameter of the pupil center based on the acquired infrared image of the user's eye; and to detect the bright reflective points formed by the infrared LED on the cornea and extract the pixel coordinates of the corneal reflective points.

[0024] A near-eye camera captures infrared images of the user's eyes at a fixed frame rate (preferably 30fps or higher). A lightweight object detection network (preferably YOLOv8-nano) is used to extract the pupil bounding box and calculate the pixel coordinates of the pupil center. and diameter The high-brightness reflective points formed by infrared LEDs on the cornea are detected, and the pixel coordinates of the corneal reflective points are extracted. .

[0025] 1.2 Three-dimensional eyeball geometry modeling module: The three-dimensional eyeball geometry modeling module is used to calculate the center of the eyeball.

[0026] Based on the PCCR (Pupil Center Corneal Reflection) principle, using the camera intrinsic parameter matrix Position of infrared light source Establish a system of corneal spherical constraint equations to solve for the corneal curvature center. (corneal curvature radius) (Preferred size: 7.8mm); from this, the center of gravity of the eyeball can be calculated. :

[0027] in, The three-dimensional coordinates of the pupil center are determined by the intrinsic parameter matrix. The intersection of the back projection and the eyeball sphere is obtained; The radius of the eyeball, 12mm is preferred. For example... Figure 2 As shown.

[0028] 1.3 Three-dimensional gaze direction calculation module: The three-dimensional gaze direction calculation module is used to calculate the original gaze direction vector.

[0029] The original gaze direction vector is calculated from the center of the pupil and the center of the eyeball:

[0030] in, This is the original gaze direction vector; The Euclidean norm of a vector (i.e., the second norm, which is the length of the vector).

[0031] The original gaze direction vector is processed by the glasses-camera extrinsic parameter matrix. Switch to the front camera coordinate system to align with the hand coordinates in time and space.

[0032] 1.4 Polynomial Mapping Calibration Module: The polynomial mapping calibration module is used to map a quadratic polynomial function F... poly The original gaze direction vector is mapped to a calibrated gaze direction unit vector.

[0033] Using a quadratic polynomial (Poly2) mapping function F poly The original gaze direction vector is calibrated and mapped to a calibrated gaze direction unit vector. Specifically, let the original gaze direction vector... The projection onto the normalized image plane is , ;F poly Composed of two quadratic polynomial components and end-normalization, the calibrated direction vector is:

[0034] Among them, the 12 coefficients a0, a1, …, a5 and b0, b1, …, b5 constitute the calibration parameter vector. It is updated online by the weighted least squares update submodule.

[0035] The optimization objective of this calibration update is weighted least squares: θ new cal = arg min θ ∑ i=1 N w i ‖F poly (p i ; θ) -g i || 2 Where, θ new cal The updated Poly2 mapping coefficient vector obtained by solving in the weighted least squares sense is θ after this calibration. cal ; The projection of the original gaze direction corresponding to the i-th sample onto the normalized image plane is obtained by taking the ratio of the first two dimensions of the 6-dimensional pupil feature vector output by the implicit calibration sample construction and quality assessment module after three-dimensional eye geometry calculation; g iLet w be the pseudo-true value direction unit vector of the i-th sample, obtained by connecting the three-dimensional coordinates of the index fingertip at the corresponding moment of the sample with the center of the i-side eyeball and normalizing the vector; i The overall quality weight output by the implicit calibration sample construction and quality assessment module; N is the current sample library capacity; θ is the parameter vector of the Poly2 mapping function (i.e., the aforementioned θ). cal (General notation used as an independent variable in the optimization process).

[0036] Due to F poly The two components are linear with respect to their coefficients, and terminal normalization does not introduce the parameters to be estimated. Therefore, a closed-form solution can be obtained at the level of the unnormalized Poly2 components. Let (u) i , v i ) for g i Denormalized two-dimensional supervised target (i.e., g) i Dividing the first two dimensions by the third dimension respectively, the parameters can be divided into... , Solve the two components independently:

[0037] In the formula, X ∈ To design a matrix (regression matrix), its i-th row... W = diag(w1, w2, …, w N ) ∈ This is a diagonal weighted matrix composed of the overall quality weights of the samples; , ; This represents the matrix transpose, (·). -1 This represents the inverse of the matrix. Stacking the two together yields... As shown in Figure 1.

[0038] (II) Gesture Perception Subsystem 2.1 Hand Key Point Detection Module: The hand key point detection module is used to extract the three-dimensional coordinates of hand joints in real time.

[0039] The MediaPipe Hands algorithm (or an equivalent algorithm) is used to extract the 3D coordinates of 21 key points of the hand in real time (1 at the wrist, 4 at each finger × 5 fingers = 20, including 5 at the fingertips), and outputs a set of key points. .

[0040] 2.2 Gesture Type Classification Module: The gesture type classification module is used for gesture classification.

[0041] The gesture type classification module classifies gesture types based on the geometric relationships of key points. To differentiate between different types of gestures in subsequent gaze estimation sample quality assessment and hand-eye consistency determination, this module presets two parameters for each type of gesture: (1) Gesture type conditionalization coefficient ρ (ρ ∈ [0,1]): Characterizes the prior confidence coefficient of the samples generated by this type of gesture in the comprehensive quality weight w. The larger ρ is, the stronger the hand-eye coupling of this type of gesture and the higher the confidence as an implicit calibration signal. ρ is given a priori by the degree of hand-eye spatial geometric coupling and participates in the calculation of comprehensive quality weight in module 4 (sample library construction). That is, the comprehensive weight of this type of sample is multiplied by ρ for weighting. When ρ=0, this type of gesture does not generate effective calibration samples.

[0042] (2) Differential consistency threshold Thr_C: This is the minimum threshold of the Pearson correlation coefficient C required to determine whether a gesture is "hand-eye spatiotemporally consistent" (the definition of C is given in Module 3). Only when C ≥ Thr_C is the sample determined to be hand-eye consistent and included in the subsequent sample library; the stronger the hand-eye coupling of the gesture, the higher its C value will be, so the value of Thr_C can be relatively lenient. Thr_C is used for consistency determination in Module 3 (hand-eye spatiotemporal consistency analysis module), and together with ρ, it completes the differential screening and weighting of samples of different gesture types.

[0043] (a) Pointing gesture: The index finger is fully extended (proximal-mid joint angle less than 30°), and the other fingers are bent (greater than 90°); strong hand-eye coupling, serving as the primary calibration signal source. (The highest trust weight is used as the full value coefficient in the overall quality weight.) (The consistency threshold is relatively lenient because the hand-eye Pearson correlation coefficient is usually high when pointing to an action.)

[0044] (b) Pinch gesture: The distance between the tips of the thumb and index finger is less than 20mm; although this gesture has a clear hand-eye gaze direction (the user usually gazes at the pinch point), the finger extension directional relationship is weaker compared to the pointing gesture, and the hand-eye spatial coupling is moderate. Therefore, it is used as an auxiliary calibration signal source. =0.5 (medium trust weight, the sample is weighted by half in the overall quality weight to avoid the excessive influence of pinch noise samples on the global mapping), Thr_C=0.5 (the consistency threshold is stricter than the pointing gesture because the hand-eye Pearson correlation coefficient distribution is wider under the pinch action, and a higher threshold is needed to filter out weakly coupled samples).

[0045] (c) Other gestures (including unrecognized gestures, transitional movements, gestures without a clear gaze direction, etc.): lack a stable spatial geometric coupling relationship with the gaze point, and are therefore considered... =0, meaning that the overall quality weight of any candidate sample generated by this type of gesture is always 0, it does not enter the sample library, does not participate in calibration updates, and is only recognized and explicitly filtered in Module 2; Thr_C is not set for this type of gesture (because its samples are always discarded, the consistency threshold is meaningless).

[0046] 2.3 Fingertip 3D Coordinate Extraction Module: The fingertip 3D coordinate extraction module is used to extract the 3D coordinates of the index fingertip.

[0047] Extract the 3D coordinates of the index fingertip (key point index 8). It also transforms the wrist coordinate system to the world coordinate system to eliminate drift caused by user movement.

[0048] 2.4 Stability Detection Module: The stability detection module is used to detect the motion stability of key points of gestures (especially fingertips) over time. It identifies continuous segments in the gesture sequence where the hand is in a low-speed, near-static state, which serve as the triggering condition and time window definition basis for the subsequent "hand-eye spatiotemporal consistency analysis module". That is, only when the user's hand is stable for a certain period of time is the gaze sample within that period considered to have the prerequisite for entering the implicit calibration process, thereby avoiding the mistaken inclusion of gaze samples during rapid hand swiping and transitional movements into the sample library.

[0049] Gaussian smoothing of the fingertip coordinate sequence ( Calculate fingertip movement speed after (frame):

[0050] in, The instantaneous velocity of the fingertip of the index finger (or the primary tracking finger) at time t (unit: m / s); Let t be the 3D coordinate vector of the fingertip keypoint in the near-eye camera coordinate system or the world coordinate system at time t (output by the gesture perception subsystem, unit: m); ||·|| is the Euclidean norm; Δt is the time interval between two adjacent frames (determined by the near-eye camera sampling rate, typical value). = 1 / 60 s ≈ 16.7 ms); v_thr is the velocity stability threshold; t_thr is the stability duration threshold.

[0051] when (Preferred speed 0.1 m / s) and this condition is continuously met for more than [time period missing] When the time is (preferably 300ms), it is determined to be a "valid and stable gesture event" and serves as a prerequisite for triggering hand-eye spatiotemporal consistency analysis.

[0052] (III) Hand-eye Spatiotemporal Consistency Analysis Subsystem The hand-eye spatiotemporal consistency analysis subsystem is one of the core innovative modules of this invention. The hand-eye spatiotemporal consistency analysis subsystem includes a velocity sequence extraction module, a correlation calculation module, and a consistency condition judgment module.

[0053] 3.1 Velocity Sequence Extraction Module: This module extracts the gaze direction velocity sequence and fingertip movement velocity sequence. The extracted gaze direction velocity sequence and fingertip movement velocity sequence (both smoothed by 5-frame moving average) are as follows:

[0054] in, The rate of change of the gaze direction (line of sight) at time t per unit time is the instantaneous angular velocity characterizing the gaze motion (unit: rad / s or rad / frame, depending on whether it is divided by Δt, see ③ below). Let t be the instantaneous linear velocity of the fingertip movement (unit: m / s); for The original gaze direction unit vector output by the 1.3 three-dimensional eye geometry solution module in step 1 at any given time ( This expresses the three-dimensional orientation of the gaze in the world coordinate system / head-mounted device coordinate system; the original gaze direction is used here. Calibration gaze direction instead of quadratic polynomial mapping To avoid Pearson-related calculations affecting the current calibration parameters (and ensure that the signal on which the trigger criterion is based remains stable during parameter updates, thus preventing circular dependencies). Δt is the 3D coordinate vector of the fingertip key point in the camera / world coordinate system at time t (output by the gesture perception subsystem, unit: m); Δt is the time interval between two adjacent frames (determined by the sampling rate of the near-eye camera, typical value 1 / 60 s ≈ 16.7 ms); ‖·‖ represents the Euclidean norm.

[0055] 3.2 Correlation Calculation Module: The correlation calculation module is used to perform two calculations between the gaze direction velocity sequence v_gaze(t) and the fingertip movement velocity sequence v_hand(t) output by the velocity sequence extraction module (3.1): First, it estimates the optimal time delay τ* between the two signals through cross-correlation analysis to compensate for the physiological / systemic time delay between human eye and hand movements; Second, it calculates the Pearson correlation coefficient C on the two sequences after time delay alignment, as a quantitative basis for judging hand-eye spatiotemporal consistency, and compares it with the gesture type differentiation threshold Thr_C to give the judgment result of whether the sample within the time window meets the hand-eye consistency.

[0056] Specifically, define the search interval [-T] max , T max Normalized cross-correlation function within:

[0057] The optimal delay estimate is:

[0058] Where τ is the time delay variable to be estimated; τ* is the estimated time delay value that maximizes R(τ); T max is the maximum time delay range for cross-correlation search (preferably 300 ms); W is the analysis time window defined by the current stable gesture event; argmax represents the value of the independent variable corresponding to the maximum value of the subsequent function; R(τ) is the normalized cross-correlation function value between v_gaze and v_hand after time shift τ, with a value range of [-1, 1].

[0059] After obtaining τ*, time-shift compensation is performed on the fingertip motion velocity sequence, letting Then, the Pearson correlation coefficient C is calculated on the time-delay aligned sequence.

[0060] in, , , respectively, are the mean values ​​of the corresponding sequences on window W; C ∈ [-1, 1], the closer to 1, the stronger the linear coupling between the two signals after time delay alignment.

[0061] Compare C with the differential threshold Thr_C corresponding to the current gesture type: if C ≥ Thr_C, it is determined that the hand-eye spatiotemporal consistency condition is met within the current time window, and the gaze samples within the window are entered into the sample library construction module (module 4); otherwise, the samples within the window are discarded.

[0062] 3.3 Consistency Condition Judgment Module: The consistency condition judgment module is used to select the corresponding differential consistency threshold Thr_C according to the gesture type to which the current stable gesture event belongs (output by the gesture type classification module 2.2), and compare the Pearson correlation coefficient C output by the correlation calculation module (3.2) with the threshold to determine whether the samples in the current time window W meet the hand-eye spatiotemporal consistency condition; the samples that meet the condition are entered into the sample library construction module (module 4) as valid calibration times, and the samples that do not meet the condition are discarded, thereby realizing the pre-filtering of low-quality / invalid samples.

[0063] Specifically, select the corresponding differentiation threshold Thr_C from the gesture type parameter table preset in Module 2.2 according to the current gesture type; compare C with this threshold: if C ≥ Thr_C, it is determined that the current time window W is a "Valid Calibration Moment (VCM)". The fixation samples (and their corresponding gesture key points and fixation anchors) output by the line-of-sight estimation subsystem within this window enter the sample library construction module as a candidate calibration sample; if C < Thr_C, it is determined that the hand-eye consistency does not meet the standard, and all samples within this window are discarded and do not participate in subsequent calibration updates.

[0064] (4) Implicit Calibration Sample Construction and Quality Assessment Subsystem The implicit calibration sample construction and quality assessment subsystem includes a sample pair pairing module, a correlation weighted scoring module, and a sample library management module.

[0065] 4.1 Sample Pair Pairing Module: The sample pair pairing module is used to automatically generate a (feature vector, true direction) sample pair for each VCM output by the consistency condition judgment module (3.3) as the input for subsequent weighted least squares parameter update (Module 4.3). Its core function is to strictly align and bind the eye geometric features output by the line-of-sight estimation subsystem and the true direction obtained by normalizing the connection line between the three-dimensional coordinates of the fingertip output by the gesture perception subsystem and the center of the ipsilateral eyeball in time to form a calibration sample, so as to continuously provide implicitly labeled calibration data for the system under the condition that the user does not actively cooperate at all.

[0066] Specifically, within the time window W when the gesture perception subsystem detects a valid stable gesture event and the hand-eye spatio-temporal consistency analysis subsystem determines that this event meets the consistency condition (C ≥ Thr_C), select the frame that makes the Pearson correlation coefficient C obtain the local maximum value within this window as the "representative frame". Generate a calibration sample from this representative frame. The purpose of only selecting the representative frame is to avoid over-fitting of the subsequent least squares caused by high redundancy of adjacent frames and ensure that the selected representative frame reaches the peak value of the hand-eye coupling strength of this event.

[0067] The form of the calibration sample pair corresponding to each representative frame is:

[0068] Among them, The input feature vector (6-dimensional, output by module 1.2) is used for the gaze estimation of the i-th sample: x_p and y_p are the two-dimensional pixel coordinates of the pupil center in the near-eye camera image plane; D_p is the pupil diameter; X_{C_o}, Y_{C_o}, and Z_{C_o} are the three-dimensional coordinates of the eyeball center C_o in the world coordinate system. The pseudo-true value unit vector g_i (three-dimensional, unit length) of the gaze direction of the i-th sample is obtained by normalizing the line connecting the three-dimensional coordinates P_{t,i} of the index fingertip of the representative frame to the center of the eyeball C_o on the same side, i.e., g_i = (P_{t,i} - C_o) / ‖P_{t,i} - C_o‖; F_proj is obtained by the factory calibration of the near-eye AR device and remains unchanged during the operation of this invention.

[0069] At each valid calibration time, the VCM outputs a calibration sample pair, denoted as s_i = (p_i, g_i, t_i, τ_i, C_i), where t_i is the timestamp of the representative frame, τ_i is the optimal delay of the event, and C_i is the Pearson correlation coefficient of the representative frame. All sample pairs {s_i} constitute the input to the sample library construction module (module 4.2). No active user intervention is required throughout the process.

[0070] 4.2 Correlation-Weighted Scoring Module Assign a comprehensive quality weight to each sample:

[0071] The weights comprehensively reflect hand-eye consistency and fingertip stability, with high-quality samples contributing a larger gradient.

[0072] 4.3 Sample Library Management Module Capacity A sliding window is used to manage samples by weight; when the library is full, the sample with the lowest weight is replaced; for samples exceeding [a certain weight], [the system continues to manage samples]. The sample weights are aged and decayed (preferably every 5 minutes) by a decay factor of 0.8 to continuously track the user's current status.

[0073] (v) Personalized calibration model training and online update subsystem 5.1 Poly2 Weighted Regression Module The weighted least squares objective function and the closed-form solution have been described in Section 1.4. Fittings are then performed separately. shaft and The axial components are estimated, with a total of 12 polynomial coefficients.

[0074] 5.2 MLP Online Fine-tuning Module When the number N of samples used for training in the sample library exceeds the threshold Nmlp (Preferred N) mlp After (= 20), this module uses the output of the aforementioned Poly2 mapping function as a reference, with a low learning rate lr = 10. -4 Incremental weighted fine-tuning of multilayer perceptron (MLP) neural networks is used to help refine high-dimensional nonlinear residuals that the Poly2 mapping function fails to characterize.

[0075] The structure of the MLP neural network is as follows: the input layer has a dimension of 6, receiving the aforementioned 6-dimensional pupil feature vector p. i ∈ Hidden layer 1 contains 64 neurons and uses the ReLU activation function; hidden layer 2 contains 64 neurons and uses the ReLU activation function; the output layer has a dimension of 2, corresponding to the projection of the calibrated gaze direction onto the normalized image plane. The two components, and the aforementioned Poly2 mapping function F poly The two-dimensional intermediate outputs before normalization are in the same geometric space, which facilitates the subsequent weighted fusion of the two branches.

[0076] The fine-tuned weighted least squares loss function is:

[0077] in, The weighted mean squared error loss function optimized for online fine-tuning of the MLP; N is the total number of samples currently participating in the fine-tuning (satisfying N ≥ N). mlp ); w i The overall quality weight of the i-th sample is output by the aforementioned implicit calibration sample construction and quality assessment module; MLP(p i Let p represent the 6-dimensional pupil feature vector of the i-th sample. i The 2D normalized image plane projection prediction value is obtained by forward propagation through the above "6→64→64→2" feedforward neural network (both hidden layers use ReLU activation); Let g be the unit vector in the direction of the false true value of the i-th sample. i The two-dimensional supervised target obtained after denormalization (i.e., g) i The first two dimensions are divided by its third dimension, which is consistent with the supervision objective used in the aforementioned Poly2 weighted least squares solution submodule; N mlp The minimum number of samples (preferably N) required to trigger online fine-tuning of the MLP. mlp = 20); lr is the learning rate used for MLP fine-tuning (preferably lr = 10). -4 ), to avoid catastrophic amnesia caused by drastic parameter updates; ||·|2 represents Euclidean... Norm; ∑ represents the summation over all samples involved in the fine-tuning.

[0078] The 2D normalized image plane projection output of the MLP branch and the intermediate output of the same dimension of the Poly2 branch are weighted and fused on the normalized image plane, and then normalized again to obtain the final calibrated gaze direction unit vector d. cal .

[0079] 5.3 Calibration Parameter Management Module Maintain a historical parameter version stack with a depth of at least 3; push new parameters onto the stack after each update for backup, and pop up a log to restore to the previous version in case of an error.

[0080] (vi) Update the decision-making mechanism subsystem like Figure 4 As shown.

[0081] 6.1 Trust Score Calculation Module For each sample in the sample library Calculate their gesture-conditional trust score :

[0082] The Expanding on the definition, the above equation is equivalently written as:

[0083] in, For the first The gesture conditional trust score of each sample comprehensively reflects the credibility of the sample in three dimensions: gesture type, hand-eye spatiotemporal consistency, and fingertip stability. For gesture type conditionalization coefficients, For the first Each sample represents the type of gesture identified in the frame. Take when pointing gesture Take when making a pinching gesture Take other gestures This is used to de-weight or remove gestures that are not subject to supervision. For the first The overall quality weight of each sample; , Both are non-negative and preferably satisfy ; For the first Each sample represents a scalar of the instantaneous fingertip velocity at a given frame, and its value is taken from the sequence of fingertip movement velocities. The modulus value on that representative frame; This is the normalized upper bound for fingertip speed.

[0084] Only when At that time, the first The sample was identified as a high-value sample and added to the update trigger counter. :

[0085] in, This is the lower bound threshold for the trust score, used to determine whether a single sample qualifies for inclusion in the parameter update trigger count. A higher threshold indicates better hand-eye consistency of the sample. ) and fingertip stillness ( The stricter the requirements, the more conservative the system updates, resulting in better parameter stability but slower update response; conversely, the lower the threshold, the more frequent the update triggers, leading to greater sensitivity to changes in the computing environment but reduced noise robustness. Optimal value: In typical near-eye use cases of head-mounted AR, a balance can be struck between "update response speed" and "parameter robustness"; the corresponding physical intuition is: when samples simultaneously satisfy the Pearson correlation coefficient... And the fingertips are almost still ( When a sample meets the criteria, it can be accepted as a high-value sample by this module.

[0086] 6.2 Update the trigger decision module When the cumulative number of high-value samples n hv Reaching the trigger threshold N trig (Preferred) N trig When = 20), this module triggers a Poly2 parameter update; the specific solution for the parameter update adopts the weighted least squares method described in the aforementioned Poly2 weighted least squares solution submodule:

[0087] Solving for the updated parameter vector yields the solution. θ new cal Then, the original parameter vector θ old cal Retained in the parameter version stack with a depth of at least 3. θ new cal Push onto the stack; after the update is complete, this module clears the trigger counter. n hv ← 0.

[0088] in, n hv The cumulative number of high-value samples output by the aforementioned Trust Score Calculation Module 6.1;N trig To update the trigger threshold, its value must be significantly greater than the number of coefficients to be estimated for each component of the Poly2 mapping function (i.e., 6 dimensions, totaling 12 dimensions for both components) to ensure the design of the matrix in the aforementioned weighted least squares solution. (Size 6×6) Numerically invertible and with good condition numbers; preferably selected N trig = 20, which can achieve a balance between "solution stability" and "update response speed".

[0089] 6.3 Convergence Monitoring Module Based on the overall quality weight in the current sample library highest The sample constitutes the validation set (Preferred) ,in (for the current sample library capacity), calculated under given parameters. Download the validation set Weighted average angle error:

[0090] in, For parameters Download the validation set Weighted average angle error (unit: degrees); The validation set is composed of comprehensive quality weights from the sample library. highest The sample consists of [number] bars; To verify the set capacity (preferred) ); This represents the current sample library capacity. For the verification set The overall quality weight of each sample; For parameters Next The angular error between the predicted gaze direction and the true gaze direction for each sample, in degrees, is calculated by mapping the Poly2 output. With truth value Each vector is back-projected into a three-dimensional gaze unit direction vector, and the inverse cosine of the angle between them is taken.

[0091] The weighted average angle error before and after this parameter update is defined as follows:

[0092] in, The parameters before the update as described in S42 In the validation set The weighted average angle error; To use the updated parameters In the same verification set The weighted average angle error. The difference between the two is defined as the error increment:

[0093] The change in angle error before and after the update (unit: degrees). This indicates that the error has increased after this update (parameter performance has deteriorated). This indicates that the error decreased after the update (parameter performance improved).

[0094] like (Preferred) The update was determined to be harmful, triggering a parameter rollback; the rollback step S44 describes: popping the parameter version stack. And restored to Remove the triggering samples from this batch. The system enters a protection waiting period and needs to accumulate more samples. (Preferred) Only after obtaining a high-value sample will the next update be allowed.

[0095] in, The threshold for determining harmful updates (preferred) This means that when the increase in the weighted average angle error of the validation set caused by this parameter update exceeds... When a new parameter is deemed to have significantly degraded performance compared to the original parameter, a parameter rollback mechanism is triggered. A smaller threshold provides stricter protection for update quality, making the system more sensitive to error fluctuations (more frequent rollbacks, higher parameter stability but decreased adaptability to environmental changes); a larger threshold increases the system's update acceptance (more aggressive response but decreased robustness to abnormal samples). Optimal value... The basis for this is: the typical post-calibration angle error of mainstream PCCR-type eye-tracking systems is... ,by The threshold roughly corresponds to a minimum distinguishable gaze offset, which can effectively distinguish between "performance degradation caused by parameter mismatch" and "normal fluctuations caused by pupil detection noise".

[0096] 6.4 Parameter Rollback Module The top parameter of the pop-up version stack is restored to the previous version; the current batch of trigger samples is removed; the system enters protection waiting (further accumulation is required). Only a high-value sample can be allowed for the next update.

[0097] One embodiment of the present invention includes: near-eye AR glasses (such as HoloLens 2 or an equivalent device), the device being equipped with a near-eye infrared camera and a front-facing RGB camera. When users wear the AR glasses for everyday interactions, naturally occurring pointing and pinching gestures are automatically collected and analyzed by the system without any additional operation.

[0098] The present invention provides an implicit adaptive calibration method for near-eye AR devices based on hand-eye coordination behavior priors, as shown in Figure 5, comprising the following steps: Step 1: Identify gaze features and hand features, specifically including: Step 11: Acquire infrared images of the eye at a fixed frame rate using a near-eye camera and perform PCCR feature extraction: Use a lightweight object detection network to extract the pupil bounding box, calculate the pixel coordinates and diameter of the pupil center; detect the pixel coordinates of the reflective points formed by the infrared LED on the cornea. Using the camera intrinsic parameter matrix and the position of the infrared light source, establish a corneal spherical constraint equation system to solve for the corneal curvature center and the eyeball center, and then calculate the original gaze direction vector. Output the calibrated gaze direction through the current polynomial mapping parameters.

[0099] Step 12: Acquire scene images using the front-facing RGB camera, and extract the 3D coordinates of 21 hand joints in real time using a hand keypoint detection algorithm (such as MediaPipe Hands). Identify the gesture type based on the geometric relationships of the keypoints. Extract the 3D coordinates of the index fingertip and transform them to the world coordinate system, then calculate the fingertip movement velocity sequence. Determine whether the stable pointing event conditions are met.

[0100] Step 2: Calculate hand-eye movement correlation and determine consistency conditions, specifically including: Step 21: Extract the gaze direction velocity sequence and fingertip movement velocity sequence respectively, and smooth them using moving average. Estimate and compensate for the optimal time delay of the two signals based on cross-correlation analysis to eliminate the fixed sampling delay difference between the sensors.

[0101] Step 22: Calculate the Pearson correlation coefficient of the velocity series within a sliding window after time alignment. Select the corresponding threshold based on the current gesture type. (Pointing gesture is 0.3, pinching gesture is 0.5), if If the consistency condition is met, then skip the current frame; otherwise, skip the current frame.

[0102] Step 3: Construct a weighted calibration sample, which includes: Step 31: Among the frames that satisfy the stable pointing event and the consistency condition, take the frame with the largest correlation coefficient in the event interval as the representative frame to avoid redundancy between adjacent frames.

[0103] Step 32: Automatically construct calibration sample pairs in representative frames. Pupil feature vector The pseudo-true value g_i of the gaze direction is obtained by normalizing the three-dimensional coordinates of the fingertip and the center of the eyeball on the same side. A comprehensive quality weight is assigned. .

[0104] Step 33: Store the weighted sample pairs in a container with a capacity of [capacity missing]. The sliding window sample library is sorted and managed by weight. When the library is full, the sample with the lowest weight is replaced, and aging and decay are performed on samples that have expired.

[0105] Step 4: Perform calibration parameter updates and anomaly detection, specifically including: Step 41: Calculate the gesture conditional trust score Only when Samples exceeding a trust threshold (preferably 0.15) are included in the high-value sample count. When the cumulative number of high-value samples reaches... At that time, a quadratic polynomial mapping model is trained based on the weighted least squares method, and the calibration parameters are updated.

[0106] Step 42: Select the highest quality sample from the database (Preferably, K = min(10, N / 2)) samples are used as the validation set. The weighted average error before and after the update is calculated. If the error increment after the update exceeds the error threshold (preferably 0.5°), it is determined to be a harmful update. The version stack is popped, the system is restored to the previous version parameters, the triggering samples in this batch are removed, and the system enters a protection waiting period (further accumulation is required). (Only 10 high-value samples are preferred for the next update).

[0107] Step 43: Feed the updated calibration parameters back to the polynomial mapping calibration module of the gaze estimation subsystem to achieve continuous online improvement of gaze estimation accuracy. Optionally, when the number of samples exceeds 20, perform low-learning-rate incremental weighted fine-tuning of the MLP with the Poly2 output as a reference to help refine the nonlinear residuals.

[0108] Experimental verification To validate the system and method, a complete offline eye-tracking calibration and evaluation system was built to complete prototype verification.

[0109] In real-world pointing gesture interaction scenarios, hand-eye consistency in the velocity dimension was verified through fingertip stability analysis: in the stable range where the fingertip is stationary, the velocity synchronization in the gaze direction tends to stabilize, supporting the core assumption that pointing stability events can be used as reliable calibration anchors.

[0110] Ablation experiments were conducted for four calibration data variants (Raw, LagComp, CorrFiltered, and LagComp_CorrFiltered), and the results are shown in Table 1.

[0111] Table 1

[0112] Ablation experiments established "CorrFiltered plus Poly2 weighted training" as the optimal solution: the average angle error was reduced to 3.60°, an improvement of about 0.12° compared to the uncalibrated baseline; with the combination of velocity correlation filtering and weighted training, the Z-axis position correlation reached 0.70 (the highest among the three axes), and the velocity correlation peak reached 0.82, indicating that the correlation filtering strategy effectively eliminated the interference of low-quality samples on calibration accuracy.

[0113] View area error analysis (angle error field heatmap) shows that the average angle error in the central region of the field of view (Yaw about -10°, Pitch about 0°) is generally less than 2°, while the error at the edge of the field of view (Pitch above 10°) is higher (about 10° or more). This reflects the inherent accuracy limitation of the geometric model at the edge of the field of view and also indicates the key areas for subsequent MLP refinement.

Claims

1. A method for implicit adaptive calibration of gaze in near-eye AR devices based on hand-eye coordination behavior priors, characterized in that, Includes the following steps: Step 1: Gaze estimation, including gaze estimation and gesture type recognition; Gaze estimation: The original gaze direction vector is obtained by acquiring infrared images of the eye through a near-eye camera, based on the detection of the pupil center and corneal reflection point and the three-dimensional eye geometry calculation. The original gaze direction vector is then mapped to the calibrated gaze direction through the current quadratic polynomial mapping function, which is used as the gaze estimation result for this frame. The calibration parameters of the quadratic polynomial mapping function are initially set to default values ​​and are replaced with the latest parameters each time the update decision in step 4 is successfully passed. Gesture type recognition: The system captures hand images using the front-facing RGB camera, extracts hand features, identifies the current gesture type, and extracts the three-dimensional coordinates of the index fingertip. Step 2: Gesture perception and classification: Based on the recognized current gesture type, extract the index fingertip velocity sequence and gaze direction velocity sequence respectively. When the fingertip velocity is lower than the velocity threshold and continues to exceed the time threshold, the corresponding gesture type is determined as a stable candidate gesture event. Based on cross-correlation analysis, the optimal time delay is estimated and compensated for the index fingertip velocity sequence and the gaze direction velocity sequence, and the two signals of the near-eye camera and the front RGB camera are aligned in time sequence. After temporal alignment, the Pearson correlation coefficient C is calculated within the sliding window. The corresponding differential threshold Thr_C is selected according to the current gesture type. If C ≥ Thr_C, the hand-eye spatiotemporal consistency condition is satisfied; otherwise, the current frame is discarded. Step 3: Hand-eye spatiotemporal consistency analysis: At the moment when both stable candidate gesture events and hand-eye spatiotemporal consistency conditions are met, the frame with the largest correlation coefficient in the event interval is taken as the representative frame, and a weighted calibration sample pair of pupil feature vector and gaze direction pseudo-true value is constructed; the pseudo-true value is obtained by normalizing the three-dimensional coordinates of the index fingertip and the line connecting the center of the ipsilateral eyeball. Each sample is assigned a comprehensive quality weight and stored in a sliding window sample library; Step 4: Sample library construction and adaptive parameter update: When the cumulative number of samples in the sliding window sample library reaches the trigger threshold, the calibration parameters of the quadratic polynomial mapping model in step 1 are updated based on the weighted least squares method. The weighted average angle error before and after the update is calculated using the highest quality sample in the sliding window sample library as the validation set. If the increase in the weighted average angle error exceeds the error threshold, it is determined to be a harmful update, and the calibration parameters are automatically rolled back to the previous version. The trigger samples in this batch are removed and the process enters the protection waiting period. Otherwise, the update is accepted and the cycle from step 1 to step 4 continues.

2. The near-eye AR device gaze implicit adaptive calibration method according to claim 1, characterized in that, In step 1: The original gaze direction vector obtained based on pupil center and corneal reflector detection and three-dimensional eyeball geometry calculation is: in, This is the original gaze direction vector; The Euclidean norm of a vector; For the center of the eyeball; The three-dimensional coordinates of the pupil center; The original gaze direction vector is mapped to the calibrated gaze direction using the current quadratic polynomial mapping function, including: make Original gaze direction vector The projection onto the normalized image plane is denoted as ,but: The above is from to d cal The mapping process is denoted by the quadratic polynomial mapping function F. poly ,Right now ; θ cal It projects the original gaze direction onto the normalized image plane. With calibration parameter vector θ cal As input, the output is the calibrated gaze direction unit vector d. cal ;in, This is the gaze direction vector calibrated by the current quadratic polynomial mapping function; The abscissa component of the projection of the gaze direction onto the normalized image plane after calibration; The ordinate component of the projection of the gaze direction onto the normalized image plane after calibration; All of these are the coefficients of a quadratic polynomial mapping function; From coefficients Constructing the calibration parameter vector : Calibration parameter vector It is generated by the weighted least squares update in step 4.

3. The near-eye AR device gaze implicit adaptive calibration method according to claim 1, characterized in that, In step 2, the optimal time delay estimation and compensation are performed on the index fingertip velocity sequence and the gaze direction velocity sequence based on cross-correlation analysis as follows: Define the search interval [-T] max , T max Normalized cross-correlation function within: The optimal delay estimate is: Where τ is the time delay variable to be estimated; τ* is the estimated time delay value that maximizes R(τ); T max is the maximum time delay range for the cross-correlation search; W is the analysis time window defined by the current stable gesture event; argmax represents the value of the independent variable corresponding to the subsequent function reaching its maximum value; R(τ) is... With time shift τ The normalized cross-correlation function values ​​between them, with a range of [-1, 1]; For the gaze direction velocity sequence, This is a sequence of fingertip movement speeds; , These are the mean values ​​of the corresponding sequences over window W; After obtaining τ*, time-shift compensation is performed on the fingertip motion velocity sequence to obtain the compensated fingertip motion velocity sequence. .

4. The near-eye AR device gaze implicit adaptive calibration method according to claim 3, characterized in that, In step 2, the Pearson correlation coefficient C, calculated within the sliding window after time alignment, is: in, The velocity sequence in the gaze direction within the window The value of the frame; The time-delay compensated fingertip movement velocity sequence within the window The value of the frame; , They are respectively and In length The mean over the sliding window; The closer to 1, the stronger the linear coupling between the two signals after time delay alignment.

5. The gaze-implicit adaptive calibration method for near-eye AR devices according to claim 3, characterized in that, Step S3 includes: S31. Generate a calibration sample pair from the representative frames. Each representative frame corresponds to a calibration sample pair. The format is: in, For the first The input feature vector is used to estimate the gaze of a sample. , The two-dimensional pixel coordinates of the pupil center on the near-eye camera image plane; The diameter of the pupil; For the eyeball center Three-dimensional coordinates in the world coordinate system; For the first The pseudo-true value unit vector g_i of the gaze direction of a sample is obtained by normalizing the line connecting the three-dimensional coordinates of the fingertip P_{t,i} and the center of the eyeball C_o on the same side, i.e. g_i = (P_{t,i} - C_o) / ‖P_{t,i} - C_o‖; The values ​​of the three-dimensional coordinates of the fingertip described in step 1 on this representative frame; S32. Output one calibration sample pair at each valid calibration time, denoted as... ,in To represent the frame timestamp, The optimal delay for this event is... The Pearson correlation coefficient for the representative frame; all sample pairs Construct a sample library; S33. Assign a comprehensive quality weight to each sample in the sample library: in, For the first The overall quality weight of each sample; The weighting coefficients for the hand-eye spatiotemporal consistency term reflect the Pearson correlation coefficient. Contribution to sample quality; The weighting coefficient for the fingertip stability term reflects the contribution of fingertip stillness to sample quality; , Both are non-negative and preferably satisfy ; For the first Each sample represents a scalar of the instantaneous fingertip velocity at a given frame, and its value is taken from the sequence of fingertip movement velocities. The modulus value on that representative frame; This is a normalized upper bound for fingertip speed, used to normalize the stability term to... interval; The larger the value, the higher the value. The higher the quality of the sample, the better.

6. The gaze-implicit adaptive calibration method for near-eye AR devices according to claim 5, characterized in that, Step S4 includes: S41. For each sample in the sample library Calculate the gesture-conditional trust score : in, The first as described in claim 5 Overall quality weight of each sample; For gesture type conditionalization coefficients, when When using a pointing gesture When making a pinching gesture, take When taking other gestures Only when At that time, the first The sample was identified as a high-value sample and included in the update trigger count. ; S42. When the cumulative number of high-value samples... Reaching the trigger threshold At that time, a calibration parameter update is triggered; all samples in the sample library are used as the training set, to... To fit the weights, the updated calibration parameter vector is obtained by solving the quadratic polynomial mapping function using weighted least squares. : in, The quadratic polynomial mapping function is described above; The current sample library capacity is denoted as ; the original calibration parameter vector is denoted as . And retain it in the parameter version stack with a depth of at least 3. Push onto the top of the stack; then clear the trigger counter. ; S43. Using the weights in the sample library highest 1 sample as validation set Calculate the weighted average angle error on the validation set before and after the update: in, In calibration parameters Next The angular error between the predicted fixation and the true fixation of a sample; let the error increment be... ; S44, If Determined as a harmful update, it is popped from the parameter version stack. And restored to Remove the triggering samples from this batch. The system enters a protection waiting period and needs to accumulate more samples. Only after obtaining a high-value sample will the next update be allowed; otherwise, accept this update and... Feedback to step 1 replaces the current calibration parameters of its quadratic polynomial mapping function Return to step 1 and continue the loop from step 1 to step 4.

7. A near-eye AR device gaze implicit adaptive calibration system based on hand-eye coordination behavior prior, characterized in that, include: The gaze estimation subsystem acquires infrared images of the user's eyes using a near-eye camera and calculates the original gaze direction vector; The gesture perception subsystem detects stable pointing events using the front-facing RGB camera; The hand-eye spatiotemporal consistency analysis subsystem extracts the gaze direction velocity sequence and the fingertip movement velocity sequence respectively, and analyzes and judges the hand-eye consistency condition. The implicit calibration sample construction and quality assessment subsystem, when both stable pointing event and consistency conditions are met, takes the frame with the largest correlation coefficient in the event interval as the representative frame, constructs calibration sample pairs of pupil feature vector and gaze direction pseudo-true value, and assigns a comprehensive quality weight that integrates hand-eye consistency and fingertip stability; wherein, the pseudo-true value is obtained by normalizing the three-dimensional coordinates of the index fingertip and the line connecting the center of the ipsilateral eyeball; The personalized calibration model training and online update subsystem updates the parameters of the quadratic polynomial mapping model based on the weighted least squares method when the cumulative number of high-value samples reaches the trigger threshold, and feeds back the updated calibration parameters to the line-of-sight estimation subsystem. The decision-making mechanism subsystem is updated to calculate the conditional trust score of gesture type. The weighted angle error change before and after the update is evaluated using high-quality samples as the validation set. If the error increment after the update exceeds the error threshold, it is judged as a harmful update, and the system is automatically rolled back to the previous version of parameters and enters a protection waiting period to prevent irreversible parameter degradation.

8. The near-eye AR device gaze implicit adaptive calibration system based on hand-eye coordination behavior prior as described in claim 7, characterized in that, The gaze estimation subsystem includes: a pupil center detection module, a three-dimensional eyeball geometry modeling module, a gaze direction calculation module, and a polynomial mapping calibration module; The pupil center detection module is used to extract the pixel coordinates of the pupil center from infrared images of the eye captured by a near-eye camera. and pupil diameter Simultaneously, the pixel coordinates of the reflective points formed by the infrared LEDs on the cornea are detected. ; A 3D eyeball geometry modeling module is used to model the eyeball based on the PCCR principle, utilizing the camera intrinsic parameter matrix. Position of infrared light source Establish a system of corneal spherical constraint equations to solve for the corneal curvature center. And determined by the radius of the eyeball Calculate the center of gravity of the eyeball ; The gaze direction calculation module is used to calculate the original gaze direction vector from the three-dimensional coordinates of the pupil center and the center of the eyeball. And then transformed to the coordinate system of the front camera via an external parameter matrix; The polynomial mapping calibration module is used to map the original gaze direction vector to a calibrated gaze direction unit vector using a quadratic polynomial mapping function F_poly, and to update the mapping parameters with weighted least squares as the optimization objective. The gesture perception subsystem includes a hand key point detection module, a gesture type classification module, a fingertip 3D coordinate extraction module, and a fingertip stability detection module. The hand keypoint detection module is used to extract the 3D coordinates of 21 joints of the hand in real time, including one joint of the wrist and four joints of each finger, and outputs a set of keypoints. ; The gesture type classification module is used to classify gestures into three categories based on the geometric relationship of key points: pointing gestures, pinching gestures, and other gestures, and to set hand-eye coupling weights for each type of gesture. and consistency threshold The pointing gesture involves fully extending the index finger while bending the other fingers. , The pinching gesture is defined as the distance between the tips of the thumb and index finger being less than 20mm. , Other gestures No samples are generated; The fingertip 3D coordinate extraction module is used to extract the 3D coordinates of the index fingertip. And it transforms from the wrist coordinate system to the world coordinate system to eliminate drift caused by user movement; The fingertip stability detection module is used to calculate the fingertip movement speed after Gaussian smoothing the fingertip coordinate sequence. When the fingertip movement speed is lower than a speed threshold, the system detects the fingertip movement speed. And the condition is met continuously for more than a time threshold. When this occurs, it is determined to be a valid and stable pointing event; The hand-eye spatiotemporal consistency analysis subsystem includes: a velocity sequence extraction module, a temporal alignment module, a Pearson correlation calculation module, and a consistency condition judgment module; The velocity sequence extraction module is used to extract the velocity sequences in the gaze direction. and fingertip movement speed sequence All values ​​have been smoothed using moving averages. The timing alignment module is used to estimate the optimal time delay of two signals based on cross-correlation analysis. And compensation is provided to eliminate the fixed sampling delay difference between the two sensor streams; The Pearson correlation calculation module is used to perform time-series alignment within a sliding window. Internal calculation of Pearson correlation coefficient of velocity series , The closer the value is to 1, the stronger the positive correlation between hand and eye movements; The consistency condition judgment module is used to set corresponding thresholds based on the gesture type. ,when If the time is determined to be a valid calibration time, the sample construction process begins; otherwise, it is discarded. The implicit calibration sample construction and quality assessment subsystem includes: a stable event detection module, a sample pair matching module, a correlation-weighted scoring module, and a sample library management module; The stable event detection module is used to extract the correlation coefficient within the event interval in frames where the gesture perception subsystem detects a stable pointing event and the hand-eye spatiotemporal consistency analysis subsystem determines that the consistency condition is met. The largest frame is used as the representative frame to generate a calibration sample; The sample pairing module is used to automatically construct calibration sample pairs in valid sampled frames, where the pupil feature vector True or false value of gaze direction It is obtained by normalizing the three-dimensional coordinates of the fingertip with the line connecting the center of the eyeball on the same side; The correlation-weighted scoring module is used to assign a comprehensive quality weight to each sample. ,in , The weighting comprehensively reflects hand-eye consistency and fingertip stability; The sample library management module is used to manage the sample library using a sliding window strategy, sort the samples by weight, replace the lowest weighted sample when the library is full, and age and decay samples that have exceeded the aging time limit by multiplying their weights by a decay coefficient. The update decision mechanism subsystem includes: a trust score calculation module, an update trigger decision module, a convergence monitoring module, and a parameter rollback module; The trust score calculation module is used to calculate the gesture-conditional trust score. Only when Samples are only included in the update trigger count as high-value samples when they exceed the trust threshold. Update the trigger decision module to trigger when the cumulative number of high-value samples reaches the trigger threshold. A polynomial parameter update is triggered once, and the counter is cleared. The convergence monitoring module is used to calculate the weighted average error using the highest quality samples in the sample library as the validation set. If the error increment after the update exceeds the error threshold, the module will monitor the result. This is then considered a harmful update; The parameter rollback module is used to pop the top parameter of the version stack and restore it to the previous version when a harmful update is determined, remove the trigger samples in this batch, and the system enters the protection waiting period.

9. An interactive device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Near-eye camera, used to capture infrared images of the eye; Front-facing RGB camera for capturing hand images; When the one or more programs are executed by the one or more processors, the device performs the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images