Near-eye AR device gaze depth disambiguation method, interaction device and storage medium

Abstract

This invention discloses a method for resolving gaze depth ambiguity in near-eye AR devices, an interactive device, and a storage medium. The method includes: constructing a sparse scene plane table; continuously extracting temporal features of eye movement patterns based on pupil image sequences acquired by a near-eye camera; identifying gesture action types and hand depth information; fusing eye movement pattern temporal features and gesture state information to infer the user's current task stage and corresponding gaze depth range; intersecting the current gaze direction vector with each plane in the sparse scene plane table; selecting the candidate plane with the highest confidence as the depth plane of the current gaze target, and outputting the estimated two-dimensional image coordinates and three-dimensional world coordinates of the gaze intersection point. This invention replaces the binocular convergence scheme with a monocular near-eye camera, significantly reducing hardware costs and power consumption while improving the gaze depth disambiguation accuracy in multi-target depth overlap scenes through task state semantic constraints.

Description

Technical Field

[0001] This invention belongs to the fields of augmented reality, human-computer interaction and computer vision technology, and specifically relates to a method and device for multimodal fusion using eye-tracking pattern temporal features, gesture interaction state and scene geometric information to achieve multi-target depth ambiguity resolution of staring rays under monocular near-eye camera conditions. Background Technology

[0002] With the increasing popularity of near-eye wearable devices such as augmented reality (AR) glasses in the consumer market, eye-tracking systems have received widespread attention as a core interaction method. Eye-tracking systems estimate the user's gaze direction using a near-eye camera and project this gaze direction onto a real-world image captured by a front-facing camera to determine the specific object the user is looking at. However, in real-world environments, multiple objects at different depths often exist along the same gaze direction. For example, when a user is facing a classroom, they might simultaneously see a laptop on a desk (approximately 0.5m), the back of a student's head in front of them (approximately 1.5m), and a blackboard in the distance (approximately 4m) in the same gaze direction. This problem, known as the gaze depth ambiguity problem, is one of the core bottlenecks restricting the interaction accuracy and user experience of AR glasses.

[0003] Existing technologies have the following main drawbacks in resolving gaze depth ambiguity: Drawback 1: Binocular vergence-based methods suffer from high hardware costs and power consumption. Represented by the vergence-based approach used by companies like Meta, this method uses near-eye cameras in both eyes to calculate gaze depth by the intersection of the gaze direction vectors. This approach requires two complete near-eye cameras, pupil detection, and gaze estimation circuitry, resulting in high hardware costs, computational demands, and power consumption, contradicting the lightweight and low-power design goals of consumer-grade AR glasses.

[0004] Defect 2: Monocular schemes lack a reliable depth determination mechanism. Existing monocular staring estimation schemes typically only output two-dimensional image coordinates or project by setting a fixed depth plane, failing to distinguish multiple targets at different depths along the same staring ray direction, resulting in a significant drop in accuracy in multi-target overlapping scenarios.

[0005] Deficiency 3: Existing solutions neglect the constraint of user behavior semantics on depth judgment. The task a user is currently performing (such as manipulating objects at hand, interacting with distant targets, observing the environment, etc.) strongly constrains their gaze depth. For example, when a user is manipulating an object on a table, their gaze target is likely located near the depth of their hand manipulation. However, existing solutions do not incorporate task state semantic information into the gaze depth estimation framework.

[0006] To address the aforementioned shortcomings, this invention proposes a gaze depth ambiguity resolution scheme based on multimodal fusion of eye movement patterns and gesture states, which achieves reliable gaze depth judgment without adding hardware to the near-eye side. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology in judging gaze depth on near-eye AR devices, and to provide a gaze depth ambiguity resolution method and device based on multimodal fusion of eye movement mode and gesture state, so as to achieve more efficient and accurate gaze depth disambiguation.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye-tracking patterns and gesture states includes the following steps: Step 1: In a stable scene, based on the front-facing camera image, identify stable geometric planes and their depth information in the scene, and construct a sparse scene plane table; Step 2: Based on the pupil image sequence acquired by the near-eye camera, continuously extract the following eye movement features: fixation stabilization period, saccade events, smooth tracking, cross-depth layer jumps, and relative change rate of pupil diameter; Step 3: Based on the front-facing camera image, use a hand key point detection algorithm to extract the coordinates of the three-dimensional key points of the hand and identify the type of hand gesture. The types of hand gesture include near-field operation gestures, far-field pointing gestures, and stationary states. When the gesture is detectable, extract the three-dimensional depth information of the hand as a strong depth prior. Step 4: Combine the fixation stabilization period, saccade events, and relative change rate of pupil diameter output from Step 2 with the gesture type and three-dimensional depth information of the hand output from Step 3 to infer the current task stage of the user and the corresponding fixation depth range. Step 5: Intersect the current gaze direction vector with each plane in the sparse scene plane table constructed in Step 1 to obtain a candidate target depth set composed of candidate planes; based on the user's current task stage inferred in Step 4, use a multi-signal fusion method to calculate the posterior confidence of each candidate plane in the candidate target depth set; select the candidate plane with the highest posterior confidence as the depth plane where the current gaze target is located, output the coordinates of the intersection point between the gaze direction vector and the selected plane, and generate the corresponding two-dimensional image coordinates and three-dimensional world coordinate estimates of the front camera.

[0009] In step 5, the posterior confidence of each candidate plane in the candidate target depth set is calculated using a multi-signal fusion method:

[0010] in, Candidate plane The posterior confidence level; The prior probability of the task state; For the depth of the gesture; Weighted by visual saliency; For the stability of the gaze direction in time sequence; This is an auxiliary signal for pupil diameter; This indicates a proportional relationship, i.e., unnormalized posterior probability. Gesture depth proximity. for:

[0011] in, Candidate plane depth, This is an uncertainty parameter for gesture depth, whose value is determined based on the depth estimation accuracy of the hand keypoint detection algorithm, and its range is 0.05m to 0.3m; when the gesture is unavailable, Set it to a uniform distribution.

[0012] In step 5, the selected plane is:

[0013] in, Select a plane; In step 6, the coordinates of the intersection point between the output gaze direction vector and the selected plane are... for:

[0014] in, The coordinates of the eyeball position, This is the current gaze direction vector. Selected planes The normal vector and the distance to the camera origin; The three-dimensional world coordinates of the gaze intersection are:

[0015] The two-dimensional pixel coordinates of the gaze intersection point in the front-facing camera image are obtained through the camera projection model:

[0016] in, The coordinates of the gaze intersection point in the front-facing camera image are two-dimensional pixel coordinates. This is the intrinsic parameter matrix for the front-facing camera. The depth component represents the gaze intersection.

[0017] Step 4 integrates the fixation stabilization period, saccade events, and relative pupil diameter change rate output from Step 2 with the gesture type and hand 3D depth information output from Step 3 to infer the user's current task stage and corresponding fixation depth range, including: Define the set of hidden states in a Hidden Markov Model. These correspond to the observation phase, orientation phase, decision-making phase, and execution phase, respectively. The eye-tracking features extracted in step 2 and the gesture information obtained in step 3 are encoded into an observation feature vector:

[0018] in, For observing feature vectors; This refers to the duration of the current period of stable fixation. The video rate is the rate of scanning within the sliding window; This is a gesture state encoding vector; Indicates vector transpose; Based on the observed feature vectors mentioned above, the optimal hidden state sequence is solved using the Viterbi algorithm to determine the current task stage of the user.

[0019] in, This is the optimal hidden state sequence; This is a sequence of candidate hidden states; For the probability of launch, Let be the state transition probability. For observing feature vectors; These are the corresponding hidden state variables; The timing length; When gestures are available: the task stage is determined based on the gesture type identified in step 3, where near-field gestures correspond to the execution stage, and far-field pointing gestures correspond to the orientation stage. In a stationary state, the aforementioned Hidden Markov Model is still used for inference. Simultaneously, the gesture depth is used as a strong anchor point, limiting the candidate depth range to [value missing]. , For depth tolerance parameters, For hand depth; When gestures are unavailable: Encode the gesture state in the observed feature vector. Set to an empty vector, utilizing only eye-tracking feature components. As the observation input, the optimal hidden state at the current moment is solved using the Viterbi algorithm described above to determine the user's task stage; the candidate depth ranges corresponding to each task stage are as follows: Observation phase: Prioritize the selection of far-field candidate planes, with the candidate depth range being the top few planes with the largest depth in the sparse scene plane table; Orientation phase: No depth limit is imposed; all candidate planes are retained. Decision-making phase: Locate the candidate plane depth near the current fixation period; the candidate depth range is... ; Execution phase: Prioritize the selection of near-field candidate planes, with the candidate depth range being the top few planes with the smallest depth in the sparse scene plane table.

[0020] Step 2 includes: Calculate the angular velocity in the gaze direction based on the temporal sequence of pupil images acquired by a near-eye camera. :

[0021] in, for The unit vector of the gaze direction at any given moment. The sampling interval; The detection conditions for the fixation stabilization period are: continuous The threshold for determining whether the angular velocity of the gaze direction is lower than the velocity of the gaze angle within a given time period is reached. ,Right now:

[0022] The detection condition for a saccade event is: the angular velocity in the gaze direction suddenly exceeds the threshold for determining the saccade angle velocity. ,Right now:

[0023] The detection condition for smooth tracking is: using the Pearson correlation coefficient between the gaze direction change and the target motion direction within the sliding window. Discrimination:

[0024] in, and The first in the window The change in gaze direction and the change in target motion direction at each sampling point For window size; when It is determined to be in a smooth tracking state. To determine the threshold for correlation coefficient in smooth tracking; when it is determined to be a smooth tracking state, lock the depth of the candidate plane where the current tracking target is located, and assign an additional gaze direction temporal stability weight to the plane in step 5. The detection condition for cross-depth transitions is: after a saccade event, the difference between the candidate plane depth corresponding to the fixation stabilization period and the candidate plane depth of the previous fixation stabilization period exceeds the depth transition threshold. :

[0025] in, The candidate plane depth corresponding to the current gaze stabilization period; The candidate plane depth corresponding to the previous gaze stabilization period; Relative change rate of pupil diameter: continuous monitoring of pupil diameter Calculate the relative rate of change within the sliding window:

[0026] in, The relative rate of change within the sliding window. This represents the average pupil diameter within the window.

[0027] In step 2, the method for extracting the relative rate of change of pupil diameter is as follows: Continuous monitoring of pupil diameter Calculate within the sliding time window Relative rate of change within:

[0028] in, The average pupil diameter within the window. The sliding time window length is used to calculate the short-term relative rate of change of pupil diameter; After eliminating the interference of changes in illumination, when At that time, the relative constriction trend of the pupil was used as a weak confidence auxiliary signal for near-field focusing. This is the threshold for determining the relative constriction trend of the pupil. When the relative rate of change of the pupil is lower than this negative threshold, it is determined to be a near-field focusing trend. In step 1, the construction process of the sparse scene plane table is as follows: Perform scene plane detection on the front-facing camera image and extract the scene plane. The sparse scene plane is denoted as a principal stable plane:

[0029] Where the feature description parameter tuple of the i-th scene plane From the plane normal vector Distance from the plane to the camera origin and planar semantic tags composition:

[0030] Scene transition detection uses the cosine similarity criterion of scene semantic feature vectors from consecutive frames of images from the front-facing camera. Frame cosine similarity All are below the switching threshold When a scene switch occurs, it is determined that the scene has changed; after the switch, it continues... The cosine similarity of all frames was higher than the threshold. The scene is determined to have reached a stable state.

[0031] In step 3, when the 3D coordinates of the hand key points are available, the planes corresponding to the depth range in the sparse scene plane table are corrected in real time using the hand depth information, and the plane depth estimates are dynamically updated. The specific correction formula is as follows:

[0032] in The corrected plane depth, To correct the step size parameter, The depth difference threshold that triggers correction.

[0033] The present invention also provides an interactive device, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states.

[0034] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states.

[0035] Compared with the prior art, the beneficial effects of the present invention are: Effect 1: Monocular approach replaces binocular convergence, significantly reducing hardware cost and power consumption. This invention requires only a monocular near-eye camera, eliminating the need for dual near-eye cameras and dual-path gaze estimation links. Compared to binocular convergence solutions, it offers significant advantages in power consumption and hardware complexity, aligning with the lightweight design goals of consumer-grade AR glasses.

[0036] Effect 2: Task state semantic constraints improve depth disambiguation accuracy. By incorporating user behavior semantics (gesture task state, eye-tracking task stage) into the depth estimation framework, the disambiguation accuracy is significantly improved in multi-object depth overlap scenarios compared to solutions that rely solely on visual signals.

[0037] Effect 3: Triggered lightweight planar perception meets low power consumption requirements. Scene planar perception is triggered only once during scene switching, and the results are persistently cached. It does not continuously perform depth estimation or 3D mapping, which greatly reduces real-time computing overhead.

[0038] Effect 4: Multi-signal layered fusion enhances system robustness. Gesture depth (strong signal), eye movement pattern (medium signal), and visual salience (weak signal) constitute a layered fusion architecture, which allows the system to smoothly degrade and ensure availability when various signal components are missing.

[0039] Effect 5: The output includes both 2D and 3D coordinates. This meets the diverse needs of downstream applications for different coordinate formats, and the depth accuracy of the 3D coordinates is significantly improved compared to the fixed-plane assumption scheme, especially in scenarios with multiple targets overlapping at depth. Attached Figure Description

[0040] Figure 1 System overall architecture diagram.

[0041] Figure 2 : Flowchart for resolving ambiguity in depth of gaze.

[0042] Figure 3 : Schematic diagram of multi-target depth ambiguity.

[0043] Figure 4 Bayesian posterior confidence fusion framework diagram. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings: Example 1 This embodiment provides a method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye-tracking patterns and gesture states, such as... Figure 2 As shown, the complete method flow is illustrated from scene plane perception, eye-tracking pattern extraction, gesture state acquisition, to multimodal fusion inference of task state, weighted intersection of candidate planes, and final output of gaze coordinates, as well as the degradation processing path when no gestures are available.

[0045] The method for resolving gaze depth ambiguity in near-eye AR devices provided in this embodiment includes the following steps: Step 1: Scene transition detection and lightweight planar perception.

[0046] Continuously monitor scene semantic changes in front-facing camera images. Specifically, use a lightweight scene classification network (such as MobileNet) to extract the scene semantic feature vector of the current frame. Calculate the cosine similarity of feature vectors from adjacent frames:

[0047] in, Cosine similarity of feature vectors from adjacent frames; Representing the eigenvector The L2 norm is used to normalize the cosine similarity calculation; When continuous Frame (e.g.) The cosine similarity of all of them is below the switching threshold. When a scene change occurs (such as moving from outdoors to indoors, or from a corridor to a classroom), the system determines if a scene has changed. Frame (e.g.) The similarity of all samples was higher than the stability threshold. When the scene is deemed to have reached a stable state, a scene plane perception process is triggered. Geometric plane recognition is performed on the current scene image, extracting several major stable planes and their corresponding depth estimates. For example, in a classroom scene, the blackboard plane (approximately 4m), desktop plane (approximately 0.7m), and screen plane (approximately 0.5m) can be identified. This plane information is presented in a sparse plane table. Cache in the form of:

[0048] in, Let N be the feature description parameter tuple for the i-th scene plane, and N be the number of detected main stable planes. ;

[0049] in, It is a plane normal vector. For plane depth, These are semantic tags. This sparse scene plane table is persistently cached when the scene remains unchanged, without being updated, significantly reducing computational overhead.

[0050] Acquiring planar depth information is a conventional technique in this field, and can be achieved through one or a combination of the following methods: detection of straight line features in the front-facing camera image and geometric plane fitting (e.g., extracting line segments using an LSD line segment detector and then fitting the plane using RANSAC); end-to-end planar segmentation deep learning networks (e.g., PlaneRCNN); and depth recovery based on inter-frame optical flow or disparity information generated by the user's natural head micro-movements (walking, slight head turning, etc.). All of the above methods do not rely on dedicated depth sensors and are compatible with lightweight AR glasses hardware. The above methods are all conventional techniques in this field: line segment detection and geometric plane fitting can be found in the literature Von Gioi et al. (IEEE TPAMI, 2010, 32(4): 722-732); end-to-end planar segmentation networks can be found in the literature Liu et al. (CVPR, 2019); and depth recovery methods based on inter-frame optical flow or disparity can be found in the literature Hartley and Zisserman (Cambridge University Press, 2003).

[0051] Step 2: Eye movement pattern extraction.

[0052] Based on the pupil image sequence captured by a near-eye camera, the following features are extracted in real time: Fixation: A period of several consecutive frames during which the gaze direction changes below a threshold indicates that the user is steadily fixating on a specific target. Definition of gaze direction angular velocity:

[0053] in, Angular velocity in the direction of gaze; for The unit vector of the gaze direction at any given moment; The sampling interval; When continuous Within a time (like This period is considered the fixation stabilization period. At this time, the confidence level for task status assessment is high. The shortest duration for determining gaze stability. The threshold for determining the viewing angle speed; A saccade event is a rapid and significant change in the direction of the gaze within a short period of time. (like This indicates that the user is switching their attention target. At this point, the depth judgment confidence is low, and it is necessary to wait for subsequent fixation to stabilize. Threshold for determining the scanning angle speed; Smooth Pursuit: The gaze direction tracks the moving target in a smooth and continuous manner. The Pearson correlation coefficient between the gaze direction change and the target's motion direction within a sliding window is used for discrimination.

[0054] in, To smooth the tracking correlation coefficient; This represents the change in gaze direction at the k-th sampling point within the window; This represents the mean value of the change in gaze direction within the window. The change in the direction of the target's motion; W represents the mean value of the change in the target's direction of motion within the window; W is the size of the sliding window. when (like When a certain condition is met, it is determined to be in a smooth tracking state, indicating that the user is tracking a specific object, and the target depth can be determined accordingly. A threshold is determined to smoothly track the correlation coefficient.

[0055] Vergence saccade: A rapid shift in gaze from a near-field target to a far-field target or vice versa, i.e., the difference in depth between candidate planes during the fixation period after a saccade event exceeds a threshold.

[0056] in, The candidate plane depth corresponding to the current gaze stabilization period; The candidate plane depth corresponding to the previous gaze stabilization period; The depth transition threshold is set; after a cross-depth layer transition event is detected, step 4, task state re-inference, and step 5, candidate plane confidence re-evaluation are triggered.

[0057] This event is a critical moment for deep disambiguation to occur, and the detection of this event triggers the deep disambiguation reassessment process.

[0058] Relative rate of change in pupil diameter: Continuous monitoring of pupil diameter Calculate the relative rate of change within the sliding window:

[0059] in, The relative rate of change within the sliding window. The average pupil diameter within the window; After eliminating interference from changes in lighting (by synchronously monitoring the average brightness of the front-facing camera), when At that time, the relative shrinkage trend of the pupil diameter is used as a weak auxiliary signal for near-field focusing. The threshold for determining a relative pupil constriction trend represents the minimum significant decrease in the relative rate of change of the pupil diameter, typically ranging from 0.03 to 0.05. That is, a near-field focusing trend is defined as a relative decrease in pupil diameter exceeding 3% to 5% within the sliding window. The method for eliminating interference from changes in illumination is to simultaneously acquire the average brightness of the front-facing camera image. When the rate of change in brightness Exceeding the threshold When suppressing the weight of the pupil diameter change signal.

[0060] Step 3: Gesture state acquisition.

[0061] Based on the images from the front-facing camera, the coordinates of 21 hand joints in the 3D camera coordinate system are extracted using a hand key point detection algorithm (MediaPipe Hands is preferred in this embodiment, but not limited to it). It identifies the type of gesture and extracts hand depth information.

[0062] Hand depth Defined as the key point of the wrist Axial components : .

[0063] Gesture types are categorized by task depth meaning as follows: Near-field gestures: pinching, touching, and swiping fingers to indicate near-field tasks.

[0064] The method for judging finger pinching is:

[0065] in, The Euclidean distance between the tip of the thumb and the tip of the index finger; The coordinates of the key point at the tip of the thumb; The coordinates of the key point at the tip of the index finger; The threshold for determining kneading; Far-field pointing gesture: Extending the index finger and pointing into the distance indicates a far-field task. The method of judgment is: the degree of index finger extension. And the other fingers are bent. This is the threshold for determining the pointer; It is the sum of the extension angles of all joints of the index finger; Still state: No obvious hand movements, refer to historical mission states.

[0066] When a gesture is detectable, MediaPipe Hands provides the hand's 3D keypoint coordinates, directly indicating the hand's depth. This serves as a strong constraint signal for the current task's depth and can simultaneously perform real-time correction on the corresponding depth region in the sparse scene planar table.

[0067] in, This is the corrected plane depth; The plane depth before correction; To correct the step size parameter; The depth difference threshold that triggers correction; the corrected depth value. The depth parameters of the corresponding plane in the sparse scene plane table are updated in real time. In subsequent step 5, the intersection of the gaze direction vector and the candidate plane and the confidence calculation are all based on the updated plane depth.

[0068] Step 4: Task state inference and candidate plane weighting.

[0069] By fusing temporal features of eye-tracking patterns with gesture state information, the system infers the user's current task stage and corresponding gaze depth range. The task stage comprises four phases: observation (far-field browsing), orientation (mid-field search), decision-making (target locking), and execution (near-field manipulation).

[0070] Define the set of hidden states in a Hidden Markov Model. These correspond to the observation phase (far-field browsing), orientation phase (mid-field search), decision-making phase (target locking), and execution phase (near-field operation), respectively. The observation feature vector is:

[0071] in, For observing feature vectors; This refers to the duration of the current period of stable fixation. The video rate is the rate of scanning within the sliding window; This is a gesture state encoding vector; Indicates vector transpose; Solving for the optimal state sequence using the Viterbi algorithm:

[0072] in, This is the optimal hidden state sequence; This is a sequence of candidate hidden states; For the probability of launch, Let be the state transition probability. For observing feature vectors; These are the corresponding hidden state variables; The timing length; When gestures are available: use gesture depth as a strong anchor point to limit the candidate depth range. , This is a depth tolerance parameter used to control the search range of candidate planes around the gesture depth anchor point. A typical value is 0.15m to 0.3m, meaning that candidate planes are searched within a range of 15cm to 30cm before and after the hand depth. For example, when a near-field gesture is detected and the hand depth is 0.5m, the plane near 0.5m in the sparse plane table is preferentially selected as the gaze target plane.

[0073] When gestures are unavailable: Based on the aforementioned Hidden Markov Model, the temporal sequence of eye movement patterns is analyzed to infer the user's current task stage, thereby constraining the candidate depth range. For example, long-term stable fixation combined with previous near-field operation history infers that the user is still in a near-field task, and near-field candidate planes are selected preferentially.

[0074] The candidate plane confidence scores are weighted using a Bayesian posterior probability framework. In this embodiment, a Bayesian posterior probability weighting method is preferred for fusing the five sub-species confidence scores, but it is not limited to this; other multi-signal fusion methods such as weighted voting and fuzzy logic can also be used. The following uses the Bayesian framework as an example. Figure 4 As shown, this illustrates how the confidence scores of five sub-items—task state prior, gesture depth proximity, visual salience, gaze stability, and pupil auxiliary signal—are fused into the posterior confidence score of the candidate plane.

[0075] The posterior confidence level is calculated by considering the following five factors:

[0076] in, Candidate plane The posterior confidence level; The prior probability of the task state; For the depth of the gesture; Weighted by visual saliency; For the stability of the gaze direction in time sequence; This is an auxiliary signal for pupil diameter; This indicates a proportional relationship, i.e., the unnormalized posterior probability.

[0077] in: (1) Prior probability of task state Prior preferences for planes of different depth ranges are determined based on the current task stage; specifically, based on the current task stage inferred by the Viterbi algorithm, values ​​are assigned according to the following rules: Observation phase: Assign a higher prior to the far-field plane. With candidate plane depth Positive correlation; Orientation phase: All candidate planes have equal priors. Set it to a uniform distribution; Decision-making phase: Assign a higher prior to the plane near the depth corresponding to the current gaze stabilization period; Execution phase: Assign higher priors to the near-field plane. With candidate plane depth Negative correlation.

[0078] (2) Gesture depth proximity The degree of closeness between the gesture depth and the candidate plane depth; Candidate plane depth, This is an uncertainty parameter for gesture depth; when gestures are unavailable, Set it to a uniform distribution; (3) Visual salience weight Visual saliency scores are calculated for image regions at the intersection points of the gaze direction vector and each candidate plane. A saliency calculation method based on image contrast is used to extract color, brightness, and texture contrast features in the local region around the intersection points and normalize them to a saliency score in the range of [0, 1]. The higher the saliency, the more likely the region is to attract the user's attention. (4) Temporal stability of gaze direction The built-in reliability is high during the fixational period and low during the saccades. Defined as:

[0079] in, and These are the weighting coefficients for the temporal stability of the gaze direction during the fixation stabilization period and the saccade period, respectively. Typical values ​​are , ; (5) Pupil diameter auxiliary signal Based on the relative rate of change of pupil diameter Calculate, when Furthermore, when illumination interference has been eliminated, a gain is applied to the near-field candidate plane: (when hour), (Other situations); among which This is the near-field gain coefficient, typically ranging from 0.1 to 0.3. This is the near-field depth threshold; when the pupil constriction trend is not significant, All candidate planes are set to 1, which does not affect the fusion result.

[0080] The final output is the candidate plane with the highest confidence: in, Select a plane; Step 5: Output gaze depth.

[0081] like Figure 3 As shown, in a typical gaze depth ambiguity scenario, a single gaze ray sequentially passes through a near-field target (e.g., a laptop, approximately 0.5m), a mid-field target (e.g., a student in the front row, approximately 1.5m), and a far-field target (e.g., a blackboard, approximately 4m). This invention selects the correct depth layer by fusing multimodal signals such as eye movement patterns and gesture states.

[0082] Current gaze direction vector With the selected plane Intersection. A gaze ray is defined as... The intersection parameters are:

[0083] in, The coordinates of the eyeball position; For ray parameters; Select a plane The normal vector; Select a plane Distance to the camera origin; The candidate plane with the highest confidence level in step 4; The following results will also be output: (1) Three-dimensional world coordinates of the gaze intersection ; (2) Two-dimensional pixel coordinates of the gaze intersection point in the front-facing camera image Projecting the model through a camera get; This is the intrinsic parameter matrix for the front-facing camera. The depth component of the gaze intersection; (3) Confidence score of current depth judgment This information is provided for downstream application decision-making reference.

[0084] Example 2 This embodiment provides a gaze depth ambiguity resolution system for near-eye AR devices based on multimodal fusion of eye-tracking patterns and gesture states. It employs a four-layer architecture. (See [link to documentation]). Figure 1 It comprises a scene perception layer, a signal extraction layer, a multimodal fusion and disambiguation layer, and an output layer. The scene perception layer includes a scene transition detection module, a lightweight plane perception module, and a sparse plane table caching module; the signal extraction layer includes an eye-tracking pattern classification module and a gesture state detection module; the multimodal fusion and disambiguation layer includes a task state inference module and a candidate plane confidence weighting module; and the output layer includes a gaze depth output module and a 2D / 3D coordinate generation module.

[0085] The functions of each module are summarized below: The scene switching detection module is used to monitor scene semantic changes in the front camera image and triggers the planar perception process after the scene switches and reaches stability. The lightweight plane perception module is used to perform geometric plane recognition on the front camera image, extract several main stable planes and their corresponding depth estimates, and generate a sparse scene plane table. The sparse scene plane table caching module is used to store the sparse scene plane table of the current scene and cache it persistently when the scene does not change. The eye-tracking pattern classification module, based on the temporal sequence of pupil images acquired by the near-eye camera, continuously classifies and extracts fixation stabilization periods, saccade events, smooth tracking, cross-depth layer jump events, and the relative rate of change of pupil diameter. The gesture state detection module extracts the coordinates of three-dimensional key points of the hand, identifies the type of gesture and hand depth information in real time based on the image from the front-facing camera. The task state inference module is used to fuse the temporal features of eye movement patterns and gesture state information to infer the user's current task stage and corresponding gaze depth range; The candidate plane confidence weighting module is used to find the intersection of the gaze direction vector with each plane of the sparse scene plane table, and assign confidence weights to each candidate plane based on the task state inference results. The gaze depth output module is used to select the candidate plane with the highest confidence and output the estimated two-dimensional image coordinates and three-dimensional world coordinates of the gaze intersection point.

[0086] For details on the specific implementation of each of the above modules, please refer to the description of the corresponding steps in Example 1.

[0087] Example 3 This embodiment provides an interactive device, including: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the near-eye AR device gaze depth ambiguity resolution method based on eye movement pattern and gesture state multimodal fusion provided in Embodiment 1.

[0088] Example 4 This embodiment provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the steps of the near-eye AR device gaze depth ambiguity resolution method based on eye movement pattern and gesture state multimodal fusion provided in Embodiment 1.

Claims

1. A method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye-tracking patterns and gesture states, characterized in that, Includes the following steps: Step 1: In a stable scene, based on the front-facing camera image, identify stable geometric planes and their depth information in the scene, and construct a sparse scene plane table; Step 2: Based on the pupil image sequence acquired by the near-eye camera, continuously extract the following eye movement features: fixation stabilization period, saccade events, smooth tracking, cross-depth layer jumps, and relative change rate of pupil diameter; Step 3: Based on the front-facing camera image, use a hand key point detection algorithm to extract the coordinates of the three-dimensional key points of the hand and identify the type of hand gesture. The types of hand gesture include near-field operation gestures, far-field pointing gestures, and stationary states. When the gesture is detectable, extract the three-dimensional depth information of the hand as a strong depth prior. Step 4: Combine the fixation stabilization period, saccade events, and relative change rate of pupil diameter output from Step 2 with the gesture type and three-dimensional depth information of the hand output from Step 3 to infer the current task stage of the user and the corresponding fixation depth range. Step 5: Intersect the current gaze direction vector with each plane in the sparse scene plane table constructed in Step 1 to obtain a candidate target depth set composed of candidate planes; based on the user's current task stage inferred in Step 4, use a multi-signal fusion method to calculate the posterior confidence of each candidate plane in the candidate target depth set; select the candidate plane with the highest posterior confidence as the depth plane where the current gaze target is located, output the coordinates of the intersection point between the gaze direction vector and the selected plane, and generate the corresponding two-dimensional image coordinates and three-dimensional world coordinate estimates of the front camera.

2. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 1, characterized in that, In step 5, the posterior confidence of each candidate plane in the candidate target depth set is calculated using a multi-signal fusion method: in, Candidate plane The posterior confidence level; The prior probability of the task state; For the depth of the gesture; Weighted by visual saliency; For the stability of the gaze direction in time sequence; This is an auxiliary signal for pupil diameter; This indicates a proportional relationship, i.e., the unnormalized posterior probability.

3. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 2, characterized in that, Gesture depth proximity for: in, Candidate plane depth, This is an uncertainty parameter for gesture depth, whose value is determined based on the depth estimation accuracy of the hand keypoint detection algorithm, and its range is 0.05m to 0.3m; when the gesture is unavailable, Set it to a uniform distribution.

4. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 3, characterized in that, In step 5, the selected plane is: in, Select a plane; In step 5, the coordinates of the intersection point between the output gaze direction vector and the selected plane are... for: in, The coordinates of the eyeball position, This is the current gaze direction vector. Selected planes The normal vector and the distance to the camera origin; The three-dimensional world coordinates of the gaze intersection are: The two-dimensional pixel coordinates of the gaze intersection point in the front-facing camera image are obtained through the camera projection model: in, The coordinates of the gaze intersection point in the front-facing camera image are two-dimensional pixel coordinates. This is the intrinsic parameter matrix for the front-facing camera. The depth component represents the gaze intersection.

5. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 3, characterized in that, Step 4 integrates the fixation stabilization period, saccade events, and relative pupil diameter change rate output from Step 2 with the gesture type and hand 3D depth information output from Step 3 to infer the user's current task stage and corresponding fixation depth range, including: Define the set of hidden states in a Hidden Markov Model. These correspond to the observation phase, orientation phase, decision-making phase, and execution phase, respectively. The eye-tracking features extracted in step 2 and the gesture information obtained in step 3 are encoded into an observation feature vector: in, For observing feature vectors; This refers to the duration of the current period of stable fixation. The video rate is the rate of scanning within the sliding window; This is a gesture state encoding vector; Indicates vector transpose; Based on the observed feature vectors mentioned above, the optimal hidden state sequence is solved using the Viterbi algorithm to determine the current task stage of the user. in, This is the optimal hidden state sequence; This is a sequence of candidate hidden states; For the probability of launch, Let be the state transition probability. For observing feature vectors; These are the corresponding hidden state variables; The timing length; When gestures are available: the task stage is determined based on the gesture type identified in step 3, where near-field gestures correspond to the execution stage, and far-field pointing gestures correspond to the orientation stage. In a stationary state, the aforementioned Hidden Markov Model is still used for inference. Simultaneously, the gesture depth is used as a strong anchor point, limiting the candidate depth range to [value missing]. , For depth tolerance parameters, For hand depth; When gestures are unavailable: Encode the gesture state in the observed feature vector. Set to an empty vector, utilizing only eye-tracking feature components. As the observation input, the optimal hidden state at the current moment is solved using the Viterbi algorithm described above to determine the user's task stage; the candidate depth ranges corresponding to each task stage are as follows: Observation phase: Prioritize the selection of far-field candidate planes, with the candidate depth range being the top few planes with the largest depth in the sparse scene plane table; Orientation phase: No depth limit is imposed; all candidate planes are retained. Decision-making phase: Locate the candidate plane depth near the current fixation period; the candidate depth range is... ; Execution phase: Prioritize the selection of near-field candidate planes, with the candidate depth range being the top few planes with the smallest depth in the sparse scene plane table.

6. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 3, characterized in that, Step 2 includes: Calculate the angular velocity in the gaze direction based on the temporal sequence of pupil images acquired by a near-eye camera. : in, for The unit vector of the gaze direction at any given moment. The sampling interval; The detection conditions for the fixation stabilization period are: continuous The threshold for determining whether the angular velocity of the gaze direction is lower than the velocity of the gaze angle within a given time period is reached. ,Right now: The detection condition for a saccade event is: the angular velocity in the gaze direction suddenly exceeds the threshold for determining the saccade angle velocity. ,Right now: The detection condition for smooth tracking is: using the Pearson correlation coefficient between the gaze direction change and the target motion direction within the sliding window. Discrimination: in, and The first in the window The change in gaze direction and the change in target motion direction at each sampling point For window size; when It is determined to be in a smooth tracking state. To determine the threshold for correlation coefficient in smooth tracking; when it is determined to be a smooth tracking state, lock the depth of the candidate plane where the current tracking target is located, and assign an additional gaze direction temporal stability weight to the plane in step 5. The detection condition for cross-depth transitions is: after a saccade event, the difference between the candidate plane depth corresponding to the fixation stabilization period and the candidate plane depth of the previous fixation stabilization period exceeds the depth transition threshold. : in, The candidate plane depth corresponding to the current gaze stabilization period; The candidate plane depth corresponding to the previous gaze stabilization period; Relative change rate of pupil diameter: continuous monitoring of pupil diameter Calculate the relative rate of change within the sliding window: in, The relative rate of change within the sliding window. This represents the average pupil diameter within the window.

7. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 6, characterized in that, In step 2, the method for extracting the relative rate of change of pupil diameter is as follows: Continuous monitoring of pupil diameter Calculate within the sliding time window Relative rate of change within: in, The average pupil diameter within the window. The sliding time window length is used to calculate the short-term relative rate of change of pupil diameter; After eliminating the interference of changes in illumination, when At that time, the relative constriction trend of the pupil was used as a weak confidence auxiliary signal for near-field focusing. This is the threshold for determining the relative constriction trend of the pupil. When the relative rate of change of the pupil is lower than this negative threshold, it is determined to be a near-field focusing trend. In step 1, the construction process of the sparse scene plane table is as follows: Perform scene plane detection on the front-facing camera image and extract the scene plane. The sparse scene plane is denoted as a principal stable plane: Where the feature description parameter tuple of the i-th scene plane From the plane normal vector Distance from the plane to the camera origin and planar semantic tags composition: Scene transition detection uses the cosine similarity criterion of scene semantic feature vectors from consecutive frames of images from the front-facing camera. Frame cosine similarity All are below the switching threshold When a scene switch occurs, it is determined that the scene has changed; after the switch, it continues... The cosine similarity of all frames was higher than the threshold. The scene is determined to have reached a stable state.

8. The method for resolving gaze depth ambiguity in near-eye AR devices based on multimodal fusion of eye movement patterns and gesture states according to claim 1, characterized in that, In step 3, when the 3D coordinates of the hand key points are available, the planes corresponding to the depth range in the sparse scene plane table are corrected in real time using the hand depth information, and the plane depth estimates are dynamically updated. The specific correction formula is as follows: in The corrected plane depth, To correct the step size parameter, The depth difference threshold that triggers correction.

9. An interactive device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the near-eye AR device gaze depth ambiguity resolution method based on eye movement pattern and gesture state multimodal fusion as described in any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the near-eye AR device gaze depth ambiguity resolution method based on eye movement pattern and gesture state multimodal fusion as described in any one of claims 1 to 8.

Images