A faceless interaction method, device, and medium based on eye-tracking gesture coordination

Abstract

This specification discloses a faceless interaction method, device, and medium based on eye-tracking gesture coordination, relating to the field of interaction technology. The method includes: acquiring user eye-tracking data collected by an eye-tracking sensor of a head-mounted device and a sequence of user hand images collected by an image acquisition device of the head-mounted device to determine the user's gaze focus coordinates and user gesture data in three-dimensional space, wherein the user gesture data includes a three-dimensional coordinate sequence of hand skeletal joints; in response to the gaze focus coordinates falling within the interaction area of ​​a target virtual object and the user gesture data matching a valid gesture template in a preset gesture dictionary, generating a spatial operation instruction corresponding to the valid gesture template; generating a rendering instruction for a virtual control based on the spatial operation instruction and the gaze focus coordinates, and executing the rendering instruction to render and display the virtual control at the spatial position associated with the gaze focus coordinates.

Description

Technical Field

[0001] This specification relates to the field of interactive technology, and in particular to a faceless interactive method, device and medium based on eye-tracking gesture coordination. Background Technology

[0002] In the fields of augmented reality (AR) and virtual reality (VR), human-computer interaction is evolving from physical controllers to more natural and immersive interface-less interactions. Current technologies primarily rely on the following solutions to achieve this type of interaction: Firstly, a common approach is to use handheld controllers or data gloves. Users trigger actions in the virtual world by operating buttons, joysticks on physical devices, or bending sensors on sensing gloves. However, this approach disrupts the spatial consistency between the user's perception and the object being manipulated. Users need to be distracted by the controller's position and operational logic, undermining the immersive experience that the virtual environment strives to create. Furthermore, the controller requires additional gripping motions, increasing the burden on the user's arms and making it unsuitable for extended use.

[0003] Secondly, in pursuit of more natural interaction, existing technologies employ computer vision-based gesture recognition schemes. These schemes typically utilize a single ordinary camera to capture images of the user's hand and then use image processing algorithms to recognize predefined gestures. However, this approach is extremely sensitive to ambient lighting conditions. In complex environments with strong light, low light, or uneven lighting, extracting hand contour and texture features becomes difficult, leading to a significant drop in recognition rates. Furthermore, when the hand is partially obscured by other objects (such as the user themselves, virtual interfaces, or objects in the real environment), the recognition system fails or makes misjudgments due to the inability to obtain complete hand information, resulting in insufficient robustness of the interaction.

[0004] Furthermore, to further enhance the directness and efficiency of interaction, existing technologies attempt to introduce eye-tracking technology and combine it with gesture recognition to form a collaborative mode of eye-tracking localization and gesture operation. However, at the implementation level, eye-tracking modules and gesture recognition cameras are usually independent devices, and the timestamps of the data collected by the two are difficult to synchronize precisely, resulting in inherent hardware latency differences. On the other hand, eye-tracking data (gaze direction) and gesture data (hand spatial coordinates) often exist in different sensor coordinate systems, requiring complex spatial registration and fusion calculations. Existing fusion algorithms are inefficient, resulting in a high overall latency from the user's intention to gaze and perform a gesture to the system's response and update of the virtual screen. Moreover, the spatial alignment accuracy is limited, leading to a user experience of lag and inaccurate operation, and the smoothness and accuracy of collaborative interaction cannot meet user needs.

[0005] In the process of locking virtual objects through eye tracking and simultaneously operating them with natural gestures, the existing gesture recognition technology is not robust enough to environmental interference and occlusion, resulting in high response latency and poor spatial positioning accuracy. This leads to low efficiency in executing interactive intentions, frequent misoperations, and affects the user's natural interactive experience. Summary of the Invention

[0006] This specification provides one or more embodiments of a faceless interaction method, device, and medium based on eye-tracking gesture collaboration to solve the following technical problem: In the process of locking virtual objects through eye tracking and simultaneously operating them with natural gestures, the existing gesture recognition technology is not robust enough to environmental interference and occlusion, resulting in high response latency and poor spatial positioning accuracy, which leads to low efficiency in executing interaction intentions, frequent misoperations, and affects the user's natural interaction experience.

[0007] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a faceless interaction method based on eye-tracking gesture coordination. The method includes: acquiring user eye-tracking data collected by an eye-tracking sensor of a head-mounted device and a sequence of user hand images collected by an image acquisition device of the head-mounted device to determine the user's gaze focus coordinates and user gesture data in three-dimensional space, wherein the user gesture data includes a three-dimensional coordinate sequence of hand skeletal joints; in response to the gaze focus coordinates falling within the interaction area of ​​a target virtual object and the user gesture data matching a valid gesture template in a preset gesture dictionary, generating a spatial operation instruction corresponding to the valid gesture template; generating a rendering instruction for a virtual control based on the spatial operation instruction and the gaze focus coordinates, and executing the rendering instruction to render and display the virtual control at the spatial position associated with the gaze focus coordinates.

[0008] This specification provides one or more embodiments of a faceless interactive device based on eye-tracking gesture coordination, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0009] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0010] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: To address the issues of high latency and poor spatial positioning accuracy in collaborative interaction caused by asynchronous eye-tracking and gesture data acquisition and inconsistent coordinate systems in existing technologies, this method synchronously acquires and processes both signals, providing precise input with synchronized time and unified coordinates for subsequent fusion and judgment. High-precision eye tracking captures the coordinates of the gaze focus, identifying the three-dimensional spatial anchor points of the user's cognitive attention. Simultaneously, image sequence analysis extracts the three-dimensional coordinate sequence of hand skeletal joints, thus completely reconstructing the user's hand spatial posture and movement trajectory in data form. This integrates previously discrete and disparate physiological signals into a spatiotemporally consistent and computable interactive state. The vector, in response to the gaze focus coordinates falling within the interaction area of ​​the target virtual object and the user gesture data matching the valid gesture templates in the preset gesture dictionary, generates spatial operation instructions corresponding to the valid gesture templates. This overcomes the problems of unnatural interaction, disruption of immersion, and easy misoperation in ambiguous scenarios caused by traditional controllers. By using gaze focus as a precondition for interaction triggering and combining it with intuitive gestures, its beneficial effect is to realize an intuitive interaction logic of "seeing is doing," significantly improving the directness of interaction and the accuracy of intent expression, while eliminating physical controllers and enhancing user immersion. Finally, based on the spatial operation instructions and the gaze focus coordinates, a rendering instruction for the virtual control is generated. The rendering instruction is executed to render and display the virtual control at the spatial position associated with the gaze focus coordinates. This solves the problems of low interaction efficiency and fatigue caused by the need for users to actively search and move their arms to operate traditional graphical user interfaces or fixed spatial controls. It realizes the dynamic and spatially adaptive interaction interface, accurately projecting the control to the user's current visual attention center, significantly reducing the user's visual search and limb movement range, thereby reducing the physiological and cognitive load of interaction and improving operation efficiency and comfort. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a faceless interaction method based on eye-tracking gesture coordination provided in an embodiment of this specification; Figure 2 This is a schematic diagram of the structure of a faceless interactive device based on eye-tracking gesture collaboration, provided as an embodiment of this specification. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] This specification provides a user-friendly interaction method based on eye-tracking gesture coordination. It should be noted that the execution entity in this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart illustrating a faceless interaction method based on eye-tracking gesture coordination provided in this specification is shown below. Figure 1 As shown, the main steps include the following: Step S101: Acquire user eye movement data collected by the eye-tracking sensor of the head-mounted device and user hand image sequence collected by the image acquisition device of the head-mounted device to determine the user's gaze focus coordinates and user gesture data in three-dimensional space.

[0014] The user gesture data includes a three-dimensional coordinate sequence of hand skeletal joints; To determine the user's gaze focus coordinates and gesture data in three-dimensional space, the method involves: acquiring user eye-tracking data collected by the eye-tracking sensor of a head-mounted device and user hand image sequences collected by the high-precision camera of the same device via a hardware synchronization signal; setting the same timestamp for each frame of user eye-tracking data and user hand image data acquired synchronously to obtain time-stamped eye-tracking data streams and hand data streams; pairing eye-tracking data and hand data belonging to the same timestamp according to the timestamp to determine paired data; inputting the paired data into a preset eye-tracking gesture fusion model; and using the eye-tracking gesture fusion model to perform time alignment and spatial coordinate system calibration on the paired data to determine the synchronized gaze focus coordinates and synchronized user gesture data.

[0015] In traditional solutions, eye tracking and gesture recognition often operate as two independent subsystems. Eye sensors capture subtle eye movements at a high frequency, while cameras capture hand images at a different frequency. Because their acquisition starts at different times and have different sampling periods, even user intentions generated at the same physical moment, such as "looking at a button and simultaneously pinching a finger," appear as two time-displaced signals at the data level. This underlying temporal asynchrony introduces unpredictable lags and biases in subsequent fusion processing. Further complicating matters, eye tracking data (vectors describing the direction of gaze) and gesture data (a set of three-dimensional coordinates describing hand joints) belong to the local coordinate systems of the two sensors, resulting in multiple spatial transformations with the global coordinate system used to ultimately present virtual content. If used directly without calibration, the user's perceived location and the location of their hand operation will be separated in virtual space, leading to a loss of responsiveness.

[0016] In one embodiment of this specification, a periodic hardware synchronization signal, typically a high-precision electronic pulse, is generated by a central processing unit integrated within the head-mounted device or a dedicated synchronization controller. This pulse signal is simultaneously distributed to the drive circuits of both the eye-tracking sensor module and the high-precision camera module. This hardware synchronization signal triggers a complete exposure and data readout of the infrared light source of the eye-tracking sensor and the high-speed image sensor, thereby acquiring a frame of eye-tracking data containing an image of the pupil and corneal reflection. The synchronization signal also triggers the image sensor of the high-precision camera to expose, acquiring a frame of color or multispectral image containing the user's hand area. To ensure strict synchronization, the design ensures that the circuit delay between pulse emission and the actual exposure of the two sensors is measured and fixed. After each synchronized acquisition, the generated frame of eye-tracking data and the frame of hand image data are assigned the same timestamp, derived from a high-precision clock counter within the device. This results in a time-stamped eye-tracking data stream and a time-stamped hand image data stream.

[0017] Subsequently, for the eye-tracking data stream, the processing unit (such as the device's built-in processor or a subsequently connected server) first detects and locates the pupil center and corneal reflector point in each frame of the image. Then, based on pre-calibrated user eye model parameters (such as corneal curvature, the relative position of the pupil and corneal reflector point), it calculates the user's gaze direction vector in the current head-mounted device posture. This vector is described in the eye-tracking sensor coordinate system. For the hand image data stream, the processing unit extracts eye-tracking data and hand image data belonging to the same moment based on the timestamp, constructing paired data packets. A pre-trained deep learning gesture recognition model then identifies the hand images. The deep learning gesture recognition model first performs hand detection and segmentation, locating the hand region in the image, and then extracts the corresponding depth features for the hand region. It should be noted that if the camera is a multispectral imaging device, the model will simultaneously process sub-images from different spectral bands (e.g., visible light and near-infrared) and fuse features from these different bands to enhance robustness under complex lighting conditions. Based on these features, a deep learning gesture recognition model regresses the positions of multiple key skeletal joints of the hand (such as fingertips, knuckles, and wrist) in the two-dimensional coordinate system of the frame image. Then, combining the camera's intrinsic parameters (such as focal length and principal point) and depth information (obtainable through binocular vision, structured light, or time-of-flight methods), the two-dimensional joint coordinates are transformed into the high-precision camera's own three-dimensional coordinate system, thus obtaining the three-dimensional coordinates of the hand's skeletal joint set at the current moment—that is, the preliminary gesture data.

[0018] Next, the eye-tracking gesture fusion model first performs fine-tuning for time alignment. Although the data is synchronized via hardware during acquisition, slight differences in sensor processing and data transmission times can result in minute discrepancies in the actual physical moments represented by the two types of data in the paired data packets, ranging from nanoseconds to milliseconds. The eye-tracking gesture fusion model uses interpolation algorithms to compensate for this. For example, using the hand joint coordinates of immediately preceding and following timestamps, kinematic interpolation (such as linear interpolation or spherical linear interpolation) is used to calculate more accurate hand joint coordinates that perfectly match the timestamps of the eye-tracking data. Similarly, similar interpolation may be performed on the eye-tracking data. The goal is to achieve theoretically perfect synchronization between the gaze direction and hand posture data used for the final judgment at any given time.

[0019] After time alignment, the eye-tracking gesture fusion model undergoes spatial coordinate system calibration. The time-aligned eye-tracking data (gaze direction vector, based on the eye-tracking sensor coordinate system) and gesture data (hand joint coordinates, based on the high-precision camera coordinate system) are uniformly transformed to the same stable reference coordinate system. In one embodiment of this specification, either the world coordinate system of the head-mounted device itself or the global coordinate system of the virtual scene is selected. Key transformation relationships, precisely measured and stored during the calibration process before shipment or during initial user calibration, are obtained. These include a rotation and translation matrix from the eye-tracking sensor coordinate system to the head-mounted device's world coordinate system, and another rotation and translation matrix from the high-precision camera coordinate system to the head-mounted device's world coordinate system. The direct transformation relationship between the two sensor coordinate systems is also included (for verification). During actual operation, the eye-tracking gesture fusion model loads the pre-stored calibration parameters. For each set of time-aligned data, the gaze direction vector is first transformed from the eye-tracking sensor coordinate system to the head-mounted device's world coordinate system using the rotation and translation matrix from the eye-tracking sensor coordinate system to the head-mounted device's world coordinate system. After the transformation, combined with the known real-time position and attitude of the head-mounted device in the world coordinate system (usually provided by the head-mounted device's inertial measurement unit and SLAM algorithm), the gaze direction vector can be extended in reverse and intersected with the virtual scene in the world coordinate system to calculate the three-dimensional coordinates of the user's gaze focus in the world coordinate system, i.e., the synchronized gaze focus coordinates. Simultaneously, the three-dimensional coordinates of each hand joint are transformed from the high-precision camera coordinate system to the same head-mounted device world coordinate system using a rotation and translation matrix from the high-precision camera coordinate system to the same head-mounted device world coordinate system, resulting in synchronized user gesture data. At this point, the user's gaze point, hand position, and attitude are all in the same world coordinate system. In one example, the eye-tracking gesture fusion model performs another round of optimization, such as using the constraint of smooth changes in the relative position of the hand and head within a short time to filter and smooth the transformed coordinates to further suppress noise. Finally, the output is a set of fully spatiotemporally synchronized, coordinate system-unified synchronized gaze focus coordinates and synchronized user gesture data.

[0020] Hardware synchronization signals fundamentally guarantee the simultaneity of data source generation. Coupled with high-precision timestamps and interpolation alignment within the fusion model, true synchronization from the physical layer to the application layer is achieved. Precise pre-calibration and the application of rigorous mathematical transformations from various sensor coordinate systems to a unified world coordinate system ensure the absolute positional relationship between the gaze focus and hand joints within the same three-dimensional spatial measurement system. This allows virtual controls to be accurately rendered at the user's gaze location, and user gestures (such as grasping and dragging) to act on target virtual objects with millimeter-level precision, greatly improving the accuracy and predictability of the interaction and meeting the needs of precision operation scenarios. Furthermore, this deep collaborative fusion enhances the overall robustness of the system. When a sensor signal experiences brief noise or loss (e.g., a gesture is momentarily occluded), the fusion model can use stable data from another sensor (such as continuous eye tracking) and a spatiotemporal correlation model for prediction and compensation, maintaining stable judgment of the interaction state and avoiding false triggers or interaction interruptions.

[0021] Determining the user's gaze focus coordinates and gesture data in three-dimensional space specifically includes: acquiring multispectral images of the user's hand simultaneously acquired by the image acquisition device in multiple different spectral bands; inputting the multispectral images into a pre-trained deep learning gesture recognition model to extract the fusion features of the multispectral images in the spatial and spectral dimensions; outputting the three-dimensional coordinates of the hand skeletal joints in the current frame image based on the fusion features; and generating a three-dimensional coordinate sequence of the hand skeletal joints based on the three-dimensional coordinates of the hand skeletal joints output from multiple consecutive frames in the user's hand image sequence.

[0022] Real-world lighting conditions are extremely complex and dynamic. Users may be outdoors in direct sunlight (producing bright reflections and dark shadows), indoors with only weak indirect lighting (resulting in image noise and low contrast), or in complex light fields containing multiple different colored light sources (such as a mixture of warm-colored lamp light and cool-colored screen light). Traditional recognition schemes that rely on a single visible light band (RGB) camera are directly limited in image quality and information integrity by ambient lighting. In insufficient light, hand texture and contour information are severely lost; under strong light or reflections, images are locally overexposed, and key joint features are distorted; under a light source with a single spectral characteristic, the hand and background may be difficult to distinguish due to similar colors. This fragile dependence on lighting causes traditional solutions to plummet in reliability in many application scenarios outside the laboratory (such as industrial inspection, outdoor navigation, and home entertainment).

[0023] In one embodiment of this specification, the image acquisition device is a customized image acquisition device integrating multispectral imaging capabilities. It is not a traditional monocular RGB camera, but rather includes a precisely optically designed and calibrated imaging sensor and corresponding filters or a beam splitting system. These sensors are configured to simultaneously acquire images of the user's hand at the same physical moment, but in different specific spectral bands.

[0024] Typical spectral band combinations include, but are not limited to: visible light bands (covering red, green, and blue, used to acquire color and everyday texture information) and near-infrared bands (sensitive to the venous structures under the skin and unaffected by changes in visible light, enabling the formation of stable biometric images). The system's drive circuitry ensures that the image acquisition from these multiple channels is strictly synchronized in terms of exposure time and spatial resolution, generating a multispectral image pre-registered at pixel locations.

[0025] Multispectral images are recognized using a pre-trained deep learning gesture recognition model. This model employs a specially designed convolutional neural network architecture. The model's input layer is designed to receive and process input data with multiple channels, each corresponding to a spectral band. At the front end of the model, the network first uses multiple sets of convolutional kernels in parallel to perform preliminary feature extraction on the image for each spectral channel, learning low-level features unique to that specific spectral domain. For example, it learns skin texture and nail contours in the visible light channel and unique patterns of vein networks in the near-infrared channel. Next, the model enters the spatial and spectral dimension fusion stage. The network structure includes a specially designed feature fusion module (e.g., using cross-channel attention mechanisms or specific fusion convolutional layers). The function of the feature fusion module is to integrate the feature maps extracted from different spectral channels. The fusion module evaluates the spectral channels whose features are reliable and information-rich under the current lighting conditions. For example, in dim environments, visible light channel features may be weaker, while near-infrared channel features remain stable; the fusion module adaptively assigns higher weights to near-infrared features. Simultaneously, the fusion process also occurs in the spatial dimension, ensuring that features of the same finger joint seen from different spectral perspectives (e.g., the fingertip contour under visible light and the vascular pattern of the fingertip region under near-infrared light) are precisely aligned in spatial location and mutually reinforce each other. Through this deep fusion, the model constructs a high-dimensional fused feature representation that is robust to changes in illumination. This fused feature simultaneously encodes the geometry, texture, and intrinsic biometric information of the hand. The organic fusion of the hand's surface optical properties (color, reflectivity), subcutaneous structural properties (vein pattern), and depth geometric information allows the model to more clearly distinguish the hand from complex backgrounds and more accurately locate joints even in low-contrast or partially occluded conditions. For example, when the finger and background color are similar, visible light features may fail, but near-infrared vein features can still provide unique biometric cues. This complementarity and cross-validation of multi-dimensional information significantly improves the accuracy, robustness, and anti-interference ability of gesture recognition.

[0026] Based on the fused feature representation, the subsequent branch of the model (the regression head network) begins to perform the keypoint localization task. This regression head network learns to decode the precise positions of a predefined set of skeletal joints of the hand from the fused features, outputting the 3D coordinates of these hand joints. In other words, for each input frame of multispectral image, the model outputs the set of coordinates of all hand joints in the current frame within the 3D coordinate system of the image acquisition device. Finally, the 3D joint coordinates output from multiple consecutive frames (consisting of a sequence of user hand images) are arranged chronologically to form a 3D coordinate sequence of the hand's skeletal joints. This sequence records the static posture of the hand at each moment and also acquires the continuous dynamic trajectory of the gesture in 3D space, providing a precise spatiotemporal data foundation for subsequent gesture classification, intent understanding, and collaborative analysis with eye-tracking data. The additional depth cues provided by multispectral information, whether directly from the depth band or indirectly learned through multispectral features, enable the output three-dimensional coordinate sequence of hand skeletal joints to have higher spatial accuracy and stability. In other words, the virtual hand model can achieve a higher fidelity match with the user's real hand movements, and the measurement of gesture trajectories (such as dragging paths and pinching distances) is more accurate. This improves the accuracy and predictability of interactive operations based on three-dimensional gestures (such as fine grasping, model rotation, and spatial drawing), meeting the stringent requirements for operational precision in professional fields.

[0027] After extracting the fusion features of the multispectral image in the spatial and spectral dimensions using the deep learning gesture recognition model, the method further includes: obtaining the coordinates of historical hand bone joints corresponding to multiple consecutive frames preceding the current frame; performing spatiotemporal correlation analysis on the historical hand bone joint coordinates and the fusion features extracted from the current frame image to determine the analysis result; when the analysis result indicates that a joint is occluded in the current frame image, predicting the current coordinates of the occluded joint based on the motion trajectory of the historical hand bone joint coordinates to determine the predicted joint coordinates of the occluded joint in the current frame; and correcting the three-dimensional coordinates of the hand bone joint using the predicted joint coordinates of the current frame.

[0028] In an ideal, undisturbed laboratory environment, the user's hand is always presented clearly and completely in front of the camera. However, in real-world AR / VR applications, such as industrial maintenance (where the user's hand may be obscured by tools or equipment parts), medical training (where surgical instruments may obscure fingers), or everyday immersive entertainment (where the user's hand may be obscured by their body, another hand, or the virtual interface itself due to a relaxed posture), partial occlusion is the norm, not the exception. When key joints of the hand are obscured, the sensor cannot directly capture the obscured physical part. Traditional processing methods either discard the frame data, causing the interaction to "stutter" or interrupt, or make inferences based on incomplete image information. Both methods compromise the smoothness and reliability of the interaction and can lead to serious misoperations in delicate operation scenarios.

[0029] In one embodiment of this specification, after the deep learning gesture recognition model completes the extraction of spatial and spectral dimension fusion features from the current frame's multispectral image, it immediately initiates temporal enhancement processing to address potential joint occlusion issues. First, it retrieves the coordinates of historical hand skeleton joints from a fixed-size first-in-first-out buffer, corresponding to multiple consecutive frames preceding the current frame. These coordinates are high-precision timestamped three-dimensional coordinate data output and stored by the model in previous frames, constituting the motion trajectory of each joint in a short temporal domain.

[0030] Subsequently, spatiotemporal correlation analysis is performed to deeply fuse and verify the consistency between the prior motion information contained in the historical trajectory and the observational evidence presented by the visual features of the current frame. Specifically, spatiotemporal correlation analysis is achieved by processing two data sources in parallel. On the one hand, using the fused features extracted from the current frame, a spatially resolved feature response map is generated through the regression head network or an additional lightweight convolutional layer within the gesture recognition model. The feature response map characterizes the probability strength of each position belonging to a hand joint in the image domain. For each position projected from historical coordinates onto the current image plane, the peak intensity and distribution pattern of the feature response map in the surrounding local region are extracted. If the peak response of the region is lower than the adaptive threshold and the distribution is scattered, it indicates that the visual evidence of the current frame supports the joint weakly.

[0031] On the other hand, based on the historical coordinate sequence, a simple kinematic model (e.g., a uniform velocity or uniform acceleration model) is established for each joint. The least squares method is used to fit its motion trajectory, and the 3D coordinates of the joint at the current frame timestamp are extrapolated and predicted. Simultaneously, the 3D coordinates of the joints are obtained directly from the preliminary regression of the fused features in the current frame model. The prediction regression bias is obtained by calculating the Euclidean distance between the predicted coordinates and the preliminary regression coordinates of each joint. This bias is then combined with the confidence score of the corresponding feature response region for a comprehensive judgment. When the confidence score is lower than a preset threshold, the feature response confidence is considered low; conversely, if the confidence score is not lower than the preset threshold, the feature response confidence is considered high. If the feature response confidence of a joint is low and the prediction regression bias is greater than the error range expected based on the historical motion speed, while most other joints show high confidence and the prediction regression bias is not greater than the error range, then the joint is determined to be occluded in the current frame image.

[0032] When the analysis determines that a specific joint is occluded, a dynamic state vector is maintained for each joint, including its three-dimensional position, velocity, and acceleration. Using historical coordinate sequences, the state vector is updated and predicted through a Kalman filter or a similar recursive Bayesian estimator. The filter estimates the joint's motion parameters based on historical observations (i.e., historical coordinates) and considers physiological constraints of hand movements (such as joint angle limitations and motion smoothness), predicting the current state through a process model. During prediction, attention is paid to the change in the curvature of the occluded joint's trajectory just before occlusion occurs to adjust the prediction model parameters. Finally, the filter outputs the optimal estimated coordinates of the occluded joint in the current frame, i.e., the predicted joint coordinates for the current frame.

[0033] Finally, for joints marked as occluded, their predicted coordinates are replaced with the coordinates initially regressed by the model. For joints marked as unoccluded but with small prediction-regression bias, a weighted average of the initial regression coordinates and predicted coordinates is calculated, with the weight proportional to the feature response confidence level to achieve a smooth transition. For completely unoccluded joints with high confidence, the initial regression coordinates are retained unchanged. This correction process ensures that the topological structure of the entire hand skeleton in three-dimensional space maintains physiological rationality and temporal continuity. The corrected three-dimensional coordinate sequence of hand skeletal joints serves as robust gesture data for subsequent interaction logic, thus providing stable and coherent hand posture information even when visual information is partially missing.

[0034] Step S102: In response to the gaze focus coordinates falling into the interaction area of ​​the target virtual object, and the user gesture data matching the valid gesture template in the preset gesture dictionary, a spatial operation command corresponding to the valid gesture template is generated.

[0035] In response to the gaze focus coordinates falling within the interaction area of ​​the target virtual object, and the user gesture data matching a valid gesture template in a preset gesture dictionary, a spatial operation instruction corresponding to the valid gesture template is generated. Specifically, this includes: obtaining context information of the current interaction scenario, wherein the context information includes the attribute information of the virtual object and the user's historical operation records in the current session; based on the context information and the user gesture data, determining the probability distribution of the user gesture data corresponding to multiple preset operation intentions through a preset intent recognition model; determining the current gesture interaction intention based on the target operation intention with the highest probability exceeding a preset threshold, according to the probability distribution; and in response to the current gesture interaction intention being consistent with the interaction logic currently supported by the target virtual object, generating a spatial operation instruction corresponding to the valid gesture template and the current gesture interaction intention.

[0036] In headless interactive systems, the traditional method of simply mapping user gestures to predefined operation commands has significant limitations. Natural gestures are inherently ambiguous and context-dependent; the same physical gesture can represent drastically different user intentions in different scenarios, for different objects, or even at different stages of the same task flow. For example, a pinch gesture might indicate selecting an item in a virtual shopping scenario, scaling the model in a 3D modeling scenario, and simply canceling or closing a pop-up in a document browsing scenario. Rigid template matching based solely on gesture shape can lead to accidental triggering or operations that do not meet user expectations, undermining the accuracy of the interaction and user trust. Furthermore, user hand movements often include numerous unconscious, non-interactive, peripheral gestures or transitional movements. These movements may resemble valid gestures in form, but their kinematic characteristics (such as force, speed, and trajectory) and the context in which they occur (such as the user's gaze not being focused on the interactive object) indicate that they do not carry any operational intent. Simple template matching cannot filter out such accidental touches, causing frequent unexpected system responses and severely disrupting the user experience.

[0037] In one embodiment of this specification, after determining that the user's gaze focus coordinates fall within the interaction area of ​​a target virtual object, and after initially determining that the user's gesture data matches a valid gesture template in a preset gesture dictionary in terms of shape, a spatial operation command corresponding to the valid gesture template is generated.

[0038] First, the context information of the current interaction scenario is obtained by querying the global state database and scene graph maintained by the system. The attribute information of virtual objects is directly read from the object's metadata. These attributes include at least the object type (e.g., button, slider, 3D model, text paragraph), the set of currently acceptable interactive operations (e.g., for a "button" object, "click" is acceptable; for a "3D model," "translation," "rotation," and "scaling" are acceptable), the object's bounding box size and orientation in 3D space, and potentially specific object states (e.g., on / off state, numerical range). The user's historical operation records in the current session can be extracted from the user interaction behavior log. This log records, in chronological order, all successful operation command sequences executed by the user during this round of continuous interaction and their target objects. Operation records within the window or related to the current target object in the most recent period are extracted for analyzing the user's operating habits and task flow context. For example, if the user's history shows that they have performed a rotation operation on the same model multiple times consecutively, the probability that the current gesture is another rotation intention may be increased.

[0039] The acquired contextual information is then combined with user gesture data from the current frame (and the most recent frames) for feature encoding and concatenation to form the input vector for the intent recognition model. Gesture data features include, but are not limited to: one-hot encoding or embedding vectors of gesture categories, 3D coordinates of hand keypoints, velocity and acceleration statistics of gesture movements, and geometric features of gesture trajectories. Contextual information is encoded as structured features, such as object type embedding vectors, mask vectors of available operation sets, and recurrent neural network hidden states of historical operation sequences. The pre-defined intent recognition model is typically a deep neural network classifier, whose architecture can be a multilayer perceptron or a more complex network capable of handling temporal features. The model is trained offline using a large amount of labeled data, with each training data point containing gesture features, contextual features, and real operation intent labels (such as "select," "delete," and "zoom in"). The model learns the probabilistic mapping relationship from fused features to various intents. During the inference phase, the constructed input vector is fed into the trained intent recognition model. The model performs forward propagation calculations and determines the probability distribution of the user's gesture data corresponding to multiple preset operation intents at its output layer. That is, it outputs a probability vector, where each element represents the confidence score of the current input belonging to a specific preset operation intent, and the sum of all scores is 1.

[0040] After obtaining the probability distribution, it is filtered according to a preset threshold. This threshold is an empirical value used to exclude inputs with generally low confidence, likely belonging to noise, or ambiguous situations. From all intents with probabilities exceeding the preset threshold, the current gesture interaction intent is determined by the target operation intent with the highest probability. For example, the model might output "select" with a probability of 0.7, "cancel" with a probability of 0.25, and other intent probabilities all below 0.1. With a preset threshold of 0.2, the system determines "select" as the current gesture interaction intent.

[0041] After determining the gesture intent, a consistency check is required. This involves querying the target virtual object's currently supported interaction logic, i.e., the set of acceptable operations defined in the object's attributes. It then determines whether the current gesture interaction intent is consistent with the target virtual object's currently supported interaction logic. Because the intent recognition model is trained on global data, it may output the intent with the highest probability for a particular object under certain context combinations, even if that object does not support it. For example, outputting the "edit" intent for a "purely display text" object. If the check passes, meaning the intent exists in the object's supported operation set, a spatial operation instruction corresponding to the valid gesture template and the current gesture interaction intent is generated. This instruction is a structured data packet containing at least the following fields: operation type (determined by the interaction intent), a unique identifier for the target virtual object, and the spatial parameters required for the operation. For example, for drag operations, a displacement vector calculated based on gesture data is required; for pinch-to-zoom, a scaling center point and scaling factor are required. The generation process is implemented through a predefined instruction template library. The corresponding template is selected based on the operation type, and the target object ID and spatial parameters calculated in real-time from the gesture data are filled into the template to form a specific instruction that can be directly executed by the rendering and logic modules. If the consistency check fails, it is determined that although the gesture matches the form and the model provides an interpretation of the intent, it conflicts with the object's state. Therefore, no operation instruction is generated, thus avoiding invalid or erroneous operations.

[0042] By dynamically integrating virtual object attributes and contextual information such as user operation history, a precise reference system is provided for decoding gesture semantics. This allows for accurate parsing of the same physical gesture in different contexts into the operation most suited to the user's current goal, greatly enhancing support for complex, multi-step interactive tasks and making the interaction process more aligned with natural human cognition and behavior patterns. Through the probability distribution output by the intent recognition model and a preset probability threshold mechanism, it can more effectively distinguish between intentional behavior and unintentional noise, significantly reducing the system's false trigger rate and improving the purity and reliability of the interaction. The intent recognition model in this technical solution is a trainable module with excellent learning and expansion capabilities. When new interactive scenarios or intents need to be supported, the model can be incrementally trained or fine-tuned by collecting new labeled data without reconstructing the core interaction logic judgment code. The structured definition of contextual information also makes it easier for the system to access new object types and attributes, improving the flexibility and adaptability of the entire interactive system and reserving space for the evolution of future interaction paradigms.

[0043] Before determining the probability distribution of the user gesture data corresponding to multiple preset operation intentions through a preset intent recognition model, the method further includes: parsing the motion feature parameters of the current gesture from the user gesture data, wherein the motion feature parameters include the average speed of the gesture movement, the peak acceleration, and the curvature change of the motion trajectory; comparing the motion feature parameters with a preset mis-touch feature threshold range; if all the motion feature parameters fall within the mis-touch feature threshold range, then the current gesture is determined to be a mis-touch operation, thereby terminating the generation of spatial operation instructions.

[0044] After obtaining the three-dimensional coordinate sequence of the hand skeletal joints, and before calling the preset intent recognition model for semantic analysis, the motion feature parameters of the current gesture are parsed from the user gesture data. Here, the current gesture refers to a candidate gesture instance composed of a series of consecutive frame coordinate sequences, which the system may have temporarily associated with a certain valid gesture category through preliminary template matching.

[0045] The three-dimensional motion trajectory of a pre-defined representative joint of the hand (e.g., the tip of the index finger or the geometric center of the hand) within the most recent short time window is calculated. The average velocity of the gesture is obtained by summing the magnitudes of displacements across all consecutive frames on the trajectory and then dividing by the total time. The peak acceleration is determined by calculating the rate of change of velocity (i.e., acceleration) between consecutive frames on the trajectory and finding the maximum absolute value. The curvature variation of the motion trajectory is quantified by analyzing the directional changes of a continuous set of points along the three-dimensional trajectory; for example, by calculating the sequence of turns between consecutive trajectory segments and statistically analyzing their standard deviation or maximum variation value to characterize the degree of curvature and the intensity of directional changes. These calculations produce a specific set of scalar values, constituting a set of characteristic parameters describing the dynamics of the gesture motion.

[0046] The motion characteristic parameter is compared with a preset range of accidental touch thresholds. The preset range of accidental touch thresholds is a set of numerical intervals defining reasonable lower and / or upper limits for each characteristic parameter. These threshold ranges are derived through offline analysis of a large number of labeled accidental and valid gesture data samples. For example, if the average speed is below a certain lower threshold V_min, it indicates that the gesture is too slow and lacks the necessary decisiveness; if the peak acceleration is below a certain lower threshold A_min, it indicates that the action lacks a clear intention to start or stop; if the trajectory curvature change is above a certain upper threshold C_max, it indicates that the motion trajectory is too tortuous and random, lacking directionality. The comparison operation is parallel, checking whether the parsed average speed, peak acceleration, and trajectory curvature change values ​​all fall within their respective accidental touch judgment intervals.

[0047] If all motion feature parameters fall within the false touch threshold range—that is, the average speed is below the lower limit of false touch, the peak acceleration is below the lower limit of false touch, and the trajectory curvature change is above the upper limit of false touch—then the gesture is determined to conform to the typical pattern of unintentional operation in terms of kinematic characteristics. Therefore, the current gesture is determined to be a false touch operation. Furthermore, the subsequent advanced processing flow for this gesture is immediately interrupted, and the generation of spatial operation instructions is terminated. In other words, the subsequent intent recognition model is not invoked, and the consistency check between intent and object logic is not performed. The gesture instance is directly discarded, its state remains unchanged, and no interactive response is generated. If the above comparison result is negative, that is, at least one motion feature parameter falls outside its false touch threshold range (e.g., the speed is fast enough), then the system considers the gesture to have the potential for intentional operation at the kinematic level, allowing it to pass through this filtering stage and continue to flow into the subsequent intent recognition model for more refined semantic analysis.

[0048] By jointly analyzing three key dynamic parameters—average velocity, peak acceleration, and trajectory curvature changes—the system significantly improves its ability to identify and intercept various unintentional, relaxed, and transitional actions, reducing the risk of accidental touches at the source. It can quickly filter out a large number of typical invalid gestures using easily calculable low-level motion features in a very short time, optimizing the overall computational load distribution of the system and helping to ensure the system's real-time performance in high-frame-rate, low-latency interactive scenarios. Simultaneously, it reduces constraints on user behavior, alleviating the psychological burden and physiological fatigue of users during interaction, further enhancing immersion and freedom of control.

[0049] Step S103: Based on the spatial operation instructions and the coordinates of the viewpoint focus, generate rendering instructions for the virtual control, execute the rendering instructions to render and display the virtual control at the spatial position associated with the viewpoint focus coordinates.

[0050] Based on the spatial operation instruction and the coordinates of the viewpoint focus, a rendering instruction for the virtual control is generated, specifically including: calling the corresponding virtual control template from the preset control template library according to the type of the spatial operation instruction; using the coordinates of the viewpoint focus as the spatial anchor point, calculating the rendering position and rendering posture of the virtual control template in three-dimensional space; and encapsulating the rendering position, the rendering posture, the identifier of the virtual control template, and the display attributes into the rendering instruction.

[0051] Traditional graphical user interfaces (GUIs) or early spatial UIs often fix controls in a preset position on the screen or in space. This forces users to move their eyes and hands to that fixed area before interacting, disrupting the continuity of the task flow and causing fatigue due to the large range of arm movements.

[0052] In one embodiment of this specification, the spatial operation instruction is first parsed to extract its type field. The type field can be an enumerated value or a specific string, explicitly identifying the requested operation category, such as "display selection menu," "pop up parameter adjustment panel," or "display tool options." Then, based on the type of the spatial operation instruction, the corresponding virtual control template is called from a preset control template library. This control template library is part of the system resources and stores pre-designed 3D models of interactive virtual controls corresponding to various operations, along with their metadata. Each template has a unique identifier and contains the control's geometric mesh data, material information, interactive hotspot definition, and default display attributes (such as initial size, color, and transparency). The calling process involves loading the complete data of the template from the library into the rendering preparation area in memory based on the operation instruction type, either by looking up a mapping table or by directly matching the identifier.

[0053] After obtaining the virtual control template, its specific presentation in the 3D world needs to be determined. The coordinates of the gaze focus are used as the spatial anchor point. The gaze focus coordinates are a 3D point in the world coordinate system, previously calculated using eye-tracking and collaborative fusion models, representing the precise point where the user's gaze falls. Next, the rendering position and rendering posture of the virtual control template in 3D space are calculated. The rendering position is calculated according to preset offset rules. For example, for a pop-up planar menu, the system might calculate a plane perpendicular to the user's gaze direction in the near space in front of the gaze focus, and place the menu's anchor point (such as the center point or upper left corner) on this plane, maintaining a preset, comfortable visual distance from the gaze focus to ensure the menu is fully visible and does not obstruct the core object the user is observing. When calculating the rendering posture, the rotation angle of the control needs to be determined to present it in the direction most conducive to user viewing and operation. In one example, the planar control is always made to face the user, meaning its normal direction is consistent with the vector pointing from the center of the control to the user's eye position (estimated from the head-mounted device position). For 3D controls, their initial pose may be determined based on the scene's gravity direction or the surface normal of the associated virtual object. All position and pose calculations are represented in world coordinates.

[0054] Finally, the rendering position, rendering pose, identifier of the virtual control template, and display attributes are encapsulated into a rendering instruction. A structured data object is created as the rendering instruction, containing at least the following fields: the identifier of the target virtual control template, used to let the graphics engine know which model to instantiate; the rendering position in 3D space (a 3D vector); the rendering pose in 3D space (usually represented by a quaternion or rotation matrix); and optional, temporarily overridden display attributes (such as specific transparency or highlighting required for this rendering). After encapsulation, this rendering instruction is sent to the graphics rendering pipeline queue. The execution of the rendering instruction is handled by an independent graphics rendering engine. Upon receiving the instruction, the rendering engine loads or references the corresponding control template geometry and materials based on the identifier, transforms and places it in world space according to the rendering position and pose specified in the instruction, applies the specified display attributes, and finally, through the head-mounted display system, renders and displays the virtual control in real time at its spatial position corresponding to the user's gaze focus coordinates. The user will then see an interactive interface fully related to the upcoming operation already in place next to or in front of the target they were just looking at, which can be directly operated using gestures.

[0055] Conventional fixed-position control designs force users to frequently switch their gaze and hand positions between the task focus and the control area. This cognitive and physical exertion accumulates in complex tasks, leading to fatigue and decreased efficiency. The technical solution in the embodiments of this specification enables controls to appear on demand and be precisely deployed. They are generated only when the user expresses a clear interaction intention and are directly presented near the user's current visual attention center. This eliminates the visual search and spatial positioning process for users to find controls, greatly reducing the physiological load and cognitive burden of interaction. By deeply binding the appearance of controls to the user's gaze and intention, virtual interface elements achieve a higher level of integration with the three-dimensional environment and user behavior flow, reducing the continuous occlusion and visual interference of interface elements on the virtual or real world scene.

[0056] After executing the rendering instruction to render and display the virtual control at the spatial location associated with the gaze focus coordinates, the method further includes: continuously monitoring the user's eye movement data to determine whether the gaze focus coordinates have left the effective response area of ​​the virtual control and exceeded a first time threshold; continuously monitoring the user's gesture data to determine whether no effective gesture related to the virtual control has been received within a second time threshold; if any of the above judgment conditions are met, a control hiding instruction is generated and rendering of the virtual control is stopped.

[0057] In one embodiment of this specification, user eye-tracking data is continuously monitored, real-time data streams from eye-tracking sensors are continuously received, and a previously calibrated eye-gesture fusion model is used to continuously calculate the real-time gaze focus coordinates of the current user in three-dimensional space. Simultaneously, the effective response area of ​​the currently displayed virtual control is acquired. The effective response area is typically a three-dimensional spatial range encompassing all interactive hotspots, formed by appropriately expanding the bounding box of the control's three-dimensional model. Within each processing cycle, the spatial relationship between the latest real-time gaze focus coordinates and the effective response area is calculated to determine if the user has moved out of the area. The criterion for moving out is that the gaze focus coordinates fall outside the three-dimensional area. Upon detecting a move out event, a timer is started for that event. It is continuously determined whether the move out state is sustained continuously and whether the duration exceeds a preset first time threshold. This first time threshold is a time buffer set to avoid misjudging the hidden state due to brief, unintentional gaze fluctuations or swiping. Only when the move out state continuously exceeds this threshold is it considered that the user's gaze has truly and stably moved away from the control.

[0058] Simultaneously, user gesture data is monitored. The system continuously receives processed user gesture data streams, which include hand posture, gesture type, and target object information. It determines whether any gestures are recognized as valid and explicitly directed at the current virtual control. If no valid gestures related to the virtual control are received within a second time threshold during a continuous monitoring period—that is, if the user does not attempt any recognized valid interaction with the control within a set time period—this condition is met. This second time threshold is designed to address situations where the user's gaze may not have left the control but they are temporarily hesitant or contemplating, preventing premature hiding.

[0059] The two monitoring processes described above operate independently, with an OR relationship. After each frame update, it checks if any of the above conditions are met. If the conditions are met—that is, the user's gaze has stably moved away from the control, or the user has not interacted with the control within a reasonable time—the hiding logic is triggered. A control hiding instruction is generated, containing at least a unique identifier for the control to be hidden and an identifier for the hiding action. The instruction is then sent to the graphics rendering pipeline. Upon receiving the instruction, the rendering engine stops rendering the virtual control. Stopping rendering may involve immediately removing the control's geometry from the scene, or initiating a brief fade-out animation before removal to ensure a natural visual transition. After the control is hidden, the system resources it occupies (such as instances in memory) can be reclaimed or placed in a standby state, and its interaction listeners are also turned off, thus completing the full lifecycle management of the dynamic control.

[0060] Compared to conventional virtual control management methods that require users to actively close them, such as by making a specific cancel gesture, clicking a close button, or displaying them in a fixed position for a long time, this technical solution achieves automatic control cleanup by intelligently sensing the natural decay of user attention and interaction intent. This eliminates the need for users to close the controls, making the interaction process smoother and more efficient. It allows users to focus their attention and operation steps entirely on the core task itself, rather than on interface management.

[0061] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: To address the issues of high latency and poor spatial positioning accuracy in collaborative interaction caused by asynchronous eye-tracking and gesture data acquisition and inconsistent coordinate systems in existing technologies, this method synchronously acquires and processes both signals, providing precise input with synchronized time and unified coordinates for subsequent fusion and judgment. High-precision eye tracking captures the coordinates of the gaze focus, identifying the three-dimensional spatial anchor points of the user's cognitive attention. Simultaneously, image sequence analysis extracts the three-dimensional coordinate sequence of hand skeletal joints, thus completely reconstructing the user's hand spatial posture and movement trajectory in data form. This integrates previously discrete and disparate physiological signals into a spatiotemporally consistent and computable interactive state. The vector, in response to the gaze focus coordinates falling within the interaction area of ​​the target virtual object and the user gesture data matching the valid gesture templates in the preset gesture dictionary, generates spatial operation instructions corresponding to the valid gesture templates. This overcomes the problems of unnatural interaction, disruption of immersion, and easy misoperation in ambiguous scenarios caused by traditional controllers. By using gaze focus as a precondition for interaction triggering and combining it with intuitive gestures, its beneficial effect is to realize an intuitive interaction logic of "seeing is doing," significantly improving the directness of interaction and the accuracy of intent expression, while eliminating physical controllers and enhancing user immersion. Finally, based on the spatial operation instructions and the gaze focus coordinates, a rendering instruction for the virtual control is generated. The rendering instruction is executed to render and display the virtual control at the spatial position associated with the gaze focus coordinates. This solves the problems of low interaction efficiency and fatigue caused by the need for users to actively search and move their arms to operate traditional graphical user interfaces or fixed spatial controls. It realizes the dynamic and spatially adaptive interaction interface, accurately projecting the control to the user's current visual attention center, significantly reducing the user's visual search and limb movement range, thereby reducing the physiological and cognitive load of interaction and improving operation efficiency and comfort.

[0062] This specification also provides an example of a user-friendly, interface-free interactive device based on eye-tracking gesture coordination, such as... Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0063] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0064] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0065] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A faceless interaction method based on eye-tracking gesture coordination, characterized in that, The method includes: The system acquires user eye-tracking data collected by the eye-tracking sensor of the head-mounted device and a sequence of user hand images collected by the image acquisition device of the head-mounted device to determine the user's gaze focus coordinates and user gesture data in three-dimensional space. The user gesture data includes a three-dimensional coordinate sequence of hand skeletal joints. In response to the gaze focus coordinates falling into the interaction area of ​​the target virtual object, and the user gesture data matching the valid gesture template in the preset gesture dictionary, a spatial operation command corresponding to the valid gesture template is generated; Based on the spatial operation instructions and the coordinates of the viewpoint focus, a rendering instruction for the virtual control is generated, and the rendering instruction is executed to render and display the virtual control at the spatial position associated with the viewpoint focus coordinates.

2. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 1, characterized in that, Determine the coordinates of the user's gaze focus and the user's gesture data in three-dimensional space, specifically including: The image acquisition device simultaneously acquires multispectral images of the user's hand in multiple different spectral bands; The multispectral image is input into a pre-trained deep learning gesture recognition model to extract the fusion features of the multispectral image in the spatial and spectral dimensions through the deep learning gesture recognition model. Based on the fusion features, the three-dimensional coordinates of the hand bone joints in the current frame image are output; A three-dimensional coordinate sequence of the hand bone joints is generated based on the three-dimensional coordinates of the hand bone joints output from multiple consecutive frames of the user's hand image sequence.

3. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 2, characterized in that, After extracting the fusion features of the multispectral image in the spatial and spectral dimensions using the deep learning gesture recognition model, the method further includes: Obtain the coordinates of historical hand bone joints corresponding to multiple consecutive frames preceding the current frame; Spatiotemporal correlation analysis is performed on the historical hand bone joint coordinates and the fusion features extracted from the current frame image to determine the analysis results; When the analysis result indicates that a joint point is occluded in the current frame image, the current coordinates of the occluded joint point are predicted based on the motion trajectory of the historical hand bone joint point coordinates, and the predicted joint point coordinates of the current frame corresponding to the occluded joint point are determined. The three-dimensional coordinates of the hand bone joints are corrected based on the predicted joint coordinates of the current frame.

4. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 1, characterized in that, In response to the gaze focus coordinates falling within the interaction area of ​​the target virtual object, and the user gesture data matching a valid gesture template in a preset gesture dictionary, a spatial operation command corresponding to the valid gesture template is generated, specifically including: Obtain context information of the current interaction scenario, wherein the context information includes the attribute information of the virtual object and the user's historical operation records in the current session; Based on the context information and the user gesture data, a preset intent recognition model is used to determine the probability distribution of the user gesture data corresponding to multiple preset operation intents; Based on the probability distribution, the current gesture interaction intent is determined by the target operation intent with the highest probability exceeding a preset threshold. In response to the current gesture interaction intent being consistent with the interaction logic currently supported by the target virtual object, a spatial operation instruction corresponding to the valid gesture template and the current gesture interaction intent is generated.

5. A faceless interaction method based on eye-tracking gesture coordination according to claim 4, characterized in that, Before determining the probability distribution of the user gesture data corresponding to multiple preset operation intentions through a preset intent recognition model, the method further includes: The motion feature parameters of the current gesture are parsed from the user gesture data, wherein the motion feature parameters include the average speed of the gesture movement, the peak acceleration, and the curvature change of the motion trajectory; The motion feature parameters are compared with a preset range of accidental touch feature thresholds; If all the motion feature parameters fall within the range of the accidental touch feature threshold, the current gesture is determined to be an accidental touch operation, and the generation of spatial operation instructions is terminated.

6. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 1, characterized in that, Based on the spatial operation instructions and the coordinates of the viewpoint focus, rendering instructions for the virtual control are generated, specifically including: Based on the type of the spatial operation instruction, the corresponding virtual control template is called from the preset control template library; Using the coordinates of the viewpoint as spatial anchor points, calculate the rendering position and rendering posture of the virtual control template in three-dimensional space; The rendering position, rendering posture, identifier of the virtual control template, and display attributes are encapsulated into the rendering instruction.

7. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 1, characterized in that, Acquiring user eye-tracking data from the eye-tracking sensor of the head-mounted device and a sequence of user hand images from the high-precision camera of the head-mounted device to determine the user's gaze focus coordinates and gesture data in three-dimensional space, specifically including: The eye-tracking sensor and the image acquisition device are synchronized by a hardware synchronization signal, and the same timestamp is set for each frame of user eye-tracking data and user hand image data acquired synchronously, so as to obtain eye-tracking data stream and hand data stream with timestamps. Based on the timestamp, eye-tracking data and hand data belonging to the same timestamp are paired to determine the paired data, and the paired data is input into a preset eye-tracking gesture fusion model. The eye-tracking gesture fusion model is used to align the paired data in time and calibrate the spatial coordinate system to determine the coordinates of the synchronized gaze focus and the synchronized user gesture data.

8. The interfaceless interaction method based on eye-tracking gesture coordination according to claim 1, characterized in that, After executing the rendering instructions to render and display the virtual control at the spatial location associated with the viewpoint focus coordinates, the method further includes: Continuously monitor the user's eye movement data to determine whether the coordinates of the gaze focus have deviated from the effective response area of ​​the virtual control and exceeded a first time threshold. Continuously monitor the user gesture data to determine whether no valid gesture related to the virtual control is received within the second time threshold. If any of the above conditions are met, a control hiding instruction is generated, and rendering of the virtual control is stopped.

9. A faceless interactive device based on eye-tracking gesture coordination, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-8.

Images