Eye-hand coordination ray virtual object touch positioning method and system

Abstract

This application provides a virtual object touch positioning method and system based on eye-hand coordinated rays, relating to the interdisciplinary field of artificial intelligence and educational technology. The method includes: acquiring eye-tracking data and hand-tracking data; generating an eye-tracking ray based on the eye-tracking data; generating a hand-tracking ray based on the hand-tracking data; determining candidate touch points based on the eye-tracking ray and hand-tracking ray; adjusting the starting position and / or direction of the hand-tracking ray based on the target virtual object region, so that the hand-tracking ray is pre-pointed to the target virtual object region; filtering out target objects from virtual objects in the neighborhood of candidate touch points based on the adjusted hand-tracking ray direction; determining the target contact position on the surface of the target object based on the eye-tracking ray and hand-tracking ray; and generating a tactile guidance signal based on the target contact position to guide the user to touch the target contact position.

Description

Technical Field

[0001] This invention relates to the field of virtual reality and human-computer interaction technology, specifically to a virtual object touch positioning method and system based on eye-hand coordination rays. Background Technology

[0002] In virtual reality environments, precise interaction between users and virtual objects is one of the core experiences. Traditional raycasting methods typically rely on a single hand ray or eye ray for target pointing. However, single-modality approaches have inherent limitations: hand raycasting lacks accuracy in locating small targets at long distances and is prone to fatigue over extended periods; while eye raycasting, although fast, suffers from inherent physiological tremors and blurred depth perception.

[0003] Existing eye-hand coordination methods often employ simple weighted averaging or fixed-rule fusion, failing to adapt to the complexity of the virtual scene. For example, in areas with dense objects or occlusion, the confidence levels of eye-tracking and hand-tracking rays dynamically change, often leading to target localization errors or repeated jumps due to fixed fusion strategies. Furthermore, existing methods lack proactive prediction of user intent, making it difficult to provide effective pre-positioning assistance before the user completes a precise operation, resulting in low interaction efficiency. In the final target locking stage, fixed filtering tolerances struggle to balance the conflicting needs of preventing accidental touches in dense areas and facilitating selection in open areas. Summary of the Invention

[0004] This invention provides a virtual object touch positioning method and system based on eye-hand fusion rays, which solves the problems of low positioning accuracy, lack of intent prediction and adaptive filtering capabilities in the prior art.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, embodiments of this application provide a virtual object touch positioning method using eye-hand coordinated rays, the method comprising: Acquire eye-tracking data and hand-tracking data; generate eye-tracking rays based on eye-tracking data; and generate hand-tracking rays based on hand-tracking data. Candidate touch points are determined based on eye-tracking rays and hand rays; these candidate touch points are the weighted nearest points of the eye-tracking rays and hand rays in three-dimensional space. Based on the target virtual object region, adjust the starting position and / or direction of the hand ray so that the hand ray is pre-pointed to the target virtual object region; the target virtual object region is the area where the predicted virtual object intended to be touched by the user is located. Based on the adjusted hand ray direction, the target object is selected from the virtual objects in the neighborhood of the candidate touch point; Based on eye-tracking rays and hand-tracking rays, determine the target contact position on the surface of the target object; A tactile guidance signal is generated based on the target contact location to guide the user to touch the target contact location.

[0006] Optionally, candidate touch points are determined, including: Obtain the spatial distribution density and occlusion status of objects in a virtual scene; Based on spatial distribution density and occlusion status, the eye-tracking decision weight and hand-tracking decision weight are dynamically adjusted; the eye-tracking decision weight is a coefficient characterizing the credibility of eye-tracking rays in localization decision-making; the hand-tracking decision weight is a coefficient characterizing the credibility of hand-tracking rays in localization decision-making. Based on the adjusted eye-tracking decision weights and hand-tracking decision weights, a weighted nearest point calculation is performed on the eye-tracking ray and the hand-tracking ray, and the calculation result is used as a candidate touch point. The weighted nearest point calculation is performed by weighted interpolation of the two perpendicular points on the common perpendicular line of the eye-tracking ray and the hand-tracking ray according to the eye-tracking decision weights and the hand-tracking decision weights, to obtain the spatial point position.

[0007] Optionally, the target virtual object region is determined, including: Based on the trajectory, speed and / or duration of the user's gaze movement, identify the user's gaze saccade pattern on virtual objects; the saccade pattern is the sequence of gaze points formed when the user's gaze jumps between multiple virtual objects and the corresponding temporal relationship, including the duration of gaze on each virtual object and the speed of jumping between gaze points. Based on the saccade pattern, determine the saccade termination area; The scanning termination area is identified as the target virtual object area.

[0008] Optionally, based on the saccade pattern, the saccade termination area is determined, including: Based on the gaze duration of each virtual object in the saccade mode, the gaze intent weight of each virtual object is determined; this gaze intent weight is positively correlated with the gaze duration. Based on the jump speed between adjacent gaze points in the saccade pattern, the saccade convergence phase is identified; this saccade convergence phase is the time interval in which the user's gaze continuously converges on a specific virtual object and the jump speed decreases. The virtual object to which the gaze point points during the saccade convergence phase is determined based on the weight of the gaze intention. The spatial region where the virtual object is located during the scan termination is defined as the scan termination region.

[0009] Optionally, the target object is selected from virtual objects in the neighborhood of the candidate touch point, including: Obtain the distribution density of virtual objects around the candidate touch point; The convergence tolerance is dynamically set based on the distribution density. This convergence tolerance is a spatial angle threshold or spatial distance threshold based on the candidate touch point, used to limit the allowable deviation range when screening target objects. This convergence tolerance is negatively correlated with the distribution density. Using the adjusted hand ray direction and based on the convergence tolerance, the virtual objects in the neighborhood of the candidate touch point are hit to determine the target object.

[0010] Optionally, using the adjusted hand ray direction and based on convergence tolerance, a hit determination is performed on virtual objects in the neighborhood of the candidate touch point to identify the target object, including: Based on the candidate touch points, determine the spatial neighborhood range of the candidate touch points; Obtain all virtual objects located within the spatial neighborhood and form a candidate object set; Calculate the spatial orientation vector of each virtual object in the candidate object set relative to the candidate touch point; Calculate the spatial angular deviation between each spatial direction vector and the adjusted hand ray direction; By comparing the spatial angle deviation with the convergence tolerance, virtual objects with a spatial angle deviation less than or equal to the convergence tolerance are selected. When only one virtual object is selected, that virtual object is identified as the target object. When there are multiple virtual objects selected, the virtual object with the smallest spatial angle deviation is determined as the target object.

[0011] Optionally, determining the target contact location on the surface of the target object includes: Based on the finger micro-movement posture data, the touch depth offset is calculated along the common perpendicular direction of the eye movement ray and the hand ray; On the surface of the target object, the touch depth offset is superimposed on the candidate touch point to obtain the target contact position.

[0012] Optionally, generating a haptic guidance signal based on the target contact location to guide the user to touch the target contact location includes: The deviation between the target contact position and the user's current hand position is mapped into a tactile guidance signal in real time; A directional vibration gradient is generated on the user's hand using a force feedback device. The direction of the vibration gradient is towards the target contact position, and the intensity of the vibration gradient is positively correlated with the magnitude of the deviation.

[0013] Optionally, the eye-tracking decision weights and hand-tracking decision weights are dynamically adjusted based on spatial distribution density and occlusion status, including: When the virtual object distribution density in the area where the candidate touch point is located is higher than a preset threshold, and there is occlusion, the eye-tracking decision weight is increased; or, When the confidence level of the gaze depth of eye-tracking rays is lower than a preset threshold, the weight of hand decision-making is increased.

[0014] Secondly, embodiments of this application provide a virtual object touch positioning system based on eye-hand coordination rays, the system comprising: The acquisition module is used to acquire eye-tracking data and hand-tracking data, generate eye-tracking rays based on eye-tracking data, and generate hand rays based on hand-tracking data. The first determining module is used to determine candidate touch points based on eye-tracking rays and hand rays; the candidate touch point is the weighted nearest point of the eye-tracking rays and hand rays in three-dimensional space; The adjustment module is used to adjust the starting position and / or direction of the hand ray based on the target virtual object area, so that the hand ray is pre-pointed to the target virtual object area; the target virtual object area is the area where the predicted virtual object intended to be touched by the user is located. The filtering module is used to filter out target objects from virtual objects in the neighborhood of candidate touch points based on the adjusted hand ray direction. The second determination module is used to determine the target contact position on the surface of the target object based on eye-tracking rays and hand rays; The guidance module is used to generate tactile guidance signals based on the target contact location to guide the user to touch the target contact location.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements all the steps of any of the methods in the first aspect described above.

[0016] Compared with the prior art, the present invention has the following beneficial effects.

[0017] By employing an intelligent fusion mechanism that integrates eye and hand movements, the system effectively solves the challenge of precise touch positioning in complex virtual reality scenarios. It acquires real-time spatial distribution density and occlusion status of objects in the scene, dynamically adjusts the decision weights of eye movements and hand movements, and calculates the weighted nearest point, overcoming the low positioning accuracy and susceptibility to jumps inherent in traditional fixed-weight fusion methods in dense or occluded areas. Based on eye movement temporal features, it predicts the user's intended touch area and actively drives hand rays for pre-positioning, transforming the user's coarse-grained visual intent into guided hand adjustments, significantly shortening target locking time. By dynamically setting the convergence tolerance based on the density of objects surrounding candidate points, it achieves adaptive adjustment of the selection granularity, effectively preventing the misselection of adjacent objects in dense areas and avoiding missed targets due to excessively small tolerances in sparse areas. Combined with the precise contact depth analysis along the common perpendicular direction of finger micro-movement postures, and the generation of directional tactile vibration gradients for closed-loop guidance, a complete and efficient technical closed loop is formed. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a virtual object touch positioning method based on eye-hand coordination rays provided in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart illustrating another virtual object touch positioning method using eye-hand coordination rays provided in an embodiment of the present invention.

[0020] Figure 3 This is a structural block diagram of a virtual object touch positioning system based on eye-hand coordination rays provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In this article, the character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can be understood as A or B.

[0023] The terms “first” and “second” in the specification and claims of this application are used to distinguish different objects, rather than to describe a specific order of objects.

[0024] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or device.

[0025] Furthermore, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0026] Example 1: Example 1 This invention provides a virtual object touch positioning method based on eye-hand coordination rays, such as... Figure 1 As shown, the method includes the following steps S101 to S106: S101. Acquire eye-tracking data and hand-tracking data, generate eye-tracking rays based on eye-tracking data, and generate hand rays based on hand-tracking data.

[0027] For example, eye-tracking data is collected by eye-tracking sensors built into the VR headset, which can obtain the gaze origin and gaze direction of the user's eyes, thereby generating an eye-tracking ray formed by fusing the gaze directions of the eyes.

[0028] Hand tracking data is collected by a handle or data glove, which can obtain the position and orientation of the hand in space, thereby generating a hand ray that originates from the hand's position and points towards the hand's orientation.

[0029] S102. Based on eye-tracking rays and hand rays, determine candidate touch points.

[0030] The aforementioned candidate touch points are the weighted nearest points of the eye-tracking ray and the hand ray in three-dimensional space.

[0031] The aforementioned weighted nearest point refers to the spatial position obtained by weighted interpolation of the points where the eye-tracking ray and the hand-tracking ray intersect each other, based on the eye-tracking decision weight and the hand-tracking decision weight, on the spatial common perpendicular line between the eye-tracking ray and the hand-tracking ray.

[0032] Specifically, in three-dimensional space, there exists a unique common perpendicular segment between two non-parallel and non-intersecting rays, which intersects each ray perpendicularly. Let the intersection of the common perpendicular with the eye-tracking ray be the first foot point P_eye, and the intersection with the hand-tracking ray be the second foot point P_hand. The common perpendicular segment is the line connecting P_eye and P_hand. The weighted nearest point P_candidate lies on this common perpendicular segment, and its position is jointly determined by the eye-tracking decision weight w_eye and the hand-tracking decision weight w_hand.

[0033] In one possible implementation, weighted nearest point The calculation formula is: in, For eye-tracking decision weights, The weighting is determined by hand movement decisions. When the weighting of eye movement decisions is higher, the weighted nearest point is closer to the first foot of the perpendicular. In other words, candidate touch points tend to be more likely to be associated with the location indicated by the eye-tracking ray; when hand decision-making is given higher weight, the weighted nearest point is closer to the second foot of the perpendicular. This means that candidate touch points tend to believe the location indicated by the hand ray. When the eye-tracking decision weight and the hand-tracking decision weight are equal, the weighted nearest point is the midpoint of the common perpendicular segment.

[0034] For example, if objects are sparsely distributed and unobstructed in the current scene, and the confidence level of the hand ray direction is high, the system adjusts the hand decision weight to 0.8 and the eye-tracking decision weight to 0.2. Then the weighted nearest point will be located at 80% of the position on the side of the perpendicular segment closer to the foot of the hand ray, so that the candidate touch points reflect the precise pointing intention of the hand more accurately.

[0035] In a special case, when the eye-tracking ray and the hand ray intersect at exactly one point in three-dimensional space, the first foot of the perpendicular... With the second foot of the perpendicular If the points coincide at this intersection point, and the length of the common perpendicular segment is zero, then regardless of how the weights are allocated, the weighted nearest point is the unique intersection point.

[0036] In another special case, when either the eye-tracking decision weight or the hand-tracking decision weight is zero, the weighted nearest point completely coincides with the perpendicular point corresponding to the non-zero weight, indicating that the system fully accepts the localization result of one of the rays.

[0037] The specific implementation of S102 includes S1021 to S1023: S1021. Obtain the spatial distribution density and occlusion status of objects in the virtual scene.

[0038] Specifically, the spatial distribution density and occlusion status of objects in a virtual scene are two quantitative indicators that reflect the complexity of the local environment where the candidate touch point is located, providing a basis for the dynamic adjustment of the eye-hand decision weights in the future.

[0039] Spatial distribution density characterizes the degree of clustering of interactive virtual objects within a local area centered on a candidate touch point.

[0040] In one possible implementation, the system uses the candidate touch point as the center and a preset statistical radius to define a spherical query region. It then uses a spatial indexing structure (such as an octree or BVH tree) to quickly retrieve interactive virtual objects within this spherical region, counting their number or calculating the ratio of the sum of their bounding box volumes to the total volume of the spherical region. The statistical result is used as a quantitative value of spatial distribution density. A higher density value indicates a more crowded region, making it more difficult to accurately select a single target.

[0041] Occlusion status indicates whether there is a front-to-back occlusion relationship between virtual objects within the area where the candidate touch point is located, starting from the user's current viewpoint.

[0042] In one possible implementation, the system emits multiple sampling rays from the user's viewpoint to the area near the candidate touch point, detecting virtual objects hit along the path of the sampling rays. If multiple objects hit the same sampling ray, and the distance between consecutive hit points exceeds a preset threshold, then the area is determined to be occluded. The occlusion status can be output as a binary identifier (occluded / unoccluded), or further quantified as the proportion of occluded objects to the total number of objects in the area.

[0043] The process of obtaining the spatial distribution density and occlusion state is executed in real time in each frame rendering loop, and calculations are performed only on the local neighborhood of the candidate touch point to ensure the real-time response performance of the system.

[0044] S1022. Based on the spatial distribution density and occlusion status, dynamically adjust the eye-tracking decision weights and hand-tracking decision weights.

[0045] Among them, eye-tracking decision weight is a coefficient that characterizes the credibility of eye-tracking rays in positioning decisions; hand-tracking decision weight is a coefficient that characterizes the credibility of hand-tracking rays in positioning decisions.

[0046] Optionally, the weight adjustment rules may be as follows: when the virtual object distribution density in the area where the candidate touch point is located is higher than a preset threshold and there is occlusion, increase the eye-tracking decision weight; when the gaze depth confidence of the eye-tracking ray is lower than a preset threshold, increase the hand-tracking decision weight.

[0047] S1023. Based on the adjusted eye-tracking decision weights and hand-tracking decision weights, perform weighted nearest point calculation on the eye-tracking ray and the hand-tracking ray, and use the calculation result as a candidate touch point.

[0048] The weighted nearest point calculation is performed by weighted interpolation of the two feet of the common perpendicular line between the eye-tracking ray and the hand-tracking ray, based on the eye-tracking decision weight and the hand-tracking decision weight, to obtain the spatial point position.

[0049] S103. Based on the target virtual object region, adjust the starting position and / or direction of the hand ray so that the hand ray is pre-pointed to the target virtual object region.

[0050] The aforementioned target virtual object area is the area where the predicted virtual object intended to be touched by the user is located.

[0051] In one possible implementation, after determining the target virtual object region, the system calculates the position of the geometric center point of the region, smoothly shifts the starting position of the hand ray from the actual spatial position of the user's hand to the vicinity of the geometric center point, and adjusts the direction of the hand ray to point from the shifted starting position to the geometric center point, so that the hand ray is pre-aligned with the target region before the user performs fine operations.

[0052] The offset process can use linear interpolation or spherical linear interpolation to make the adjustment of the hand ray visually smooth and natural, avoiding abrupt jumps. After the adjustment is completed, the user only needs to make a small finger movement to accurately lock onto the target object, significantly reducing the burden of operation.

[0053] The specific implementations of S103 include S1031 to S1033: S1031. Based on the trajectory, speed and / or duration of the user's gaze movement, identify the user's gaze pattern on the virtual object.

[0054] The aforementioned saccade pattern is the sequence of gaze points and the corresponding temporal relationship formed when the user's gaze jumps between multiple virtual objects, including the duration of gaze on each virtual object and the speed of jump between gaze points.

[0055] Specifically, the recognition process for the scanning mode is carried out in real time while the user is wearing the virtual reality device and browsing the virtual scene.

[0056] The eye-tracking sensor continuously collects the user's gaze point 3D coordinates and corresponding timestamps at a preset sampling frequency to form a raw gaze point stream. The system preprocesses the raw gaze point stream, including noise reduction and gaze point clustering, aggregating spatially close and temporally continuous sampling points into a gaze event.

[0057] As the user's gaze moves within the virtual scene, their gaze point will sequentially fall on different virtual object surfaces. The system uses ray collision detection to determine the virtual object marker that the current gaze point is pointing to in real time. When the gaze point jumps from the first virtual object to the second virtual object, the system records a saccade event. This saccade event includes the following information: the starting virtual object, the target virtual object, the time of leaving the starting virtual object, the time of arriving at the target virtual object, and the angle jump between the two virtual objects.

[0058] As users continue browsing, a series of consecutive saccade events and gaze events together constitute a saccade pattern.

[0059] The data structure of the above saccade pattern can be represented as a temporal sequence, where each element corresponds to a gaze pause segment or a saccade jump segment.

[0060] Among them, the gaze duration on each virtual object is the duration of the gaze event corresponding to that virtual object; the jump speed between gaze points is the ratio of the change in viewing angle between the centers of gaze points in two adjacent gaze events to the time taken for the saccade jump segment.

[0061] For example, suppose a user sequentially views three objects in a virtual desktop scene: a "virtual water glass," a "virtual notebook," and a "virtual pen." The aforementioned gazing pattern can be recorded as follows: after gazing at the "virtual water glass" for 0.3 seconds, the user jumps to the "virtual notebook" at an average angular velocity of 120 degrees per second; after gazing at the "virtual notebook" for 1.2 seconds, the user jumps to the "virtual pen" at an average angular velocity of 80 degrees per second; and after gazing at the "virtual pen" for 0.5 seconds, the user jumps to an empty area of ​​the desktop at an average angular velocity of 50 degrees per second. This sequence and its temporal relationship constitute the gazing pattern at the current moment.

[0062] Based on the aforementioned saccade pattern, the system can perform intent prediction to determine the saccade termination area by considering the duration of gaze on each virtual object and the jump speed between gaze points. The duration of gaze reflects the user's cognitive attention to each virtual object; the longer the gaze, the higher the likelihood that the object is the user's intended target. The jump speed reflects the user's visual search state; when the jump speed continuously decreases and the gaze point converges on a specific object, it indicates that the user's visual search process is nearing its end, and the intended target has been essentially locked.

[0063] The aforementioned saccade pattern can be a sequence of gaze point shifts with a temporal order, formed when a user's gaze is visually searching among a group of interactive virtual objects.

[0064] In one possible implementation, the saccade pattern is specifically represented as a list of gaze events ordered by time, wherein each gaze event includes at least: the identifier of the gaze target object, the gaze start time, the gaze end time, and the saccade jump duration and saccade jump angular velocity experienced from the previous gaze target to the current gaze target.

[0065] For example, in a typical visual search process, the user's gaze first sweeps over object A and lingers briefly for 0.2 seconds, then jumps to object B at a faster speed and lingers for 0.8 seconds, and then jumps to object C at a slower speed and lingers for more than 1.5 seconds. The saccade pattern formed by this process can be characterized as a convergence trend of gradually increasing gaze duration and gradually decreasing jump speed between gaze points. The area where object C is located at the endpoint of this convergence trend is the predicted target virtual object area.

[0066] S1032. Based on this scanning mode, determine the scanning termination area.

[0067] Specifically, firstly, based on the gaze duration corresponding to each virtual object in the saccade mode, the gaze intent weight of each virtual object is determined, and this weight is positively correlated with the gaze duration; then, based on the jump speed between adjacent gaze points in the saccade mode, the saccade convergence phase is identified, which is the time interval in which the user's gaze continuously converges towards a specific virtual object and the jump speed decreases; finally, based on the virtual object pointed to by the gaze point in the saccade convergence phase and the gaze intent weight, the saccade termination virtual object is determined, and the spatial region where the saccade termination virtual object is located is determined as the saccade termination region.

[0068] S1033. The area where the scan ends is determined as the target virtual object area.

[0069] After determining the target virtual object area, the starting point of the hand ray is actively adjusted to be near the preset anchor point of that area, and / or the direction of the hand ray is rotated to point to the center of that area, so that the hand ray is pre-pointed to the target area that the user intends to operate, thus completing the active pre-positioning.

[0070] S104. Based on the adjusted hand ray direction, filter out the target object from the virtual objects in the neighborhood of the candidate touch point.

[0071] The specific implementations of S104 include S1041 to S1043: S1041. Obtain the distribution density of virtual objects around the candidate touch point.

[0072] Specifically, the distribution density of virtual objects around the candidate touch point is a key indicator used to measure the degree of aggregation of interactive virtual objects in the local spatial area where the candidate touch point is located, providing a basis for the dynamic setting of subsequent convergence tolerance.

[0073] In one possible implementation, the system constructs a spherical detection area with the candidate touch point as the center and a preset density detection radius as the search range.

[0074] The spherical detection region is quickly queried using a spatial index structure to obtain all interactive virtual objects contained within it. The virtual object distribution density can be expressed as the absolute number of interactive virtual objects within the spherical region; a higher number indicates a denser distribution. Alternatively, it can be expressed as the ratio of the sum of the bounding box volumes of all virtual objects within the spherical region to the total volume of the spherical region (i.e., the spatial duty cycle); a higher duty cycle indicates a denser distribution.

[0075] In another possible implementation, the virtual object distribution density can be calculated using a kernel density estimation method. The system applies a distance-weighted kernel function to the center point of each virtual object in the surrounding space, centered on the candidate touch point. The kernel function values ​​of all objects are then summed to obtain the density estimate for that point. This method can more smoothly reflect continuous changes in spatial density, avoiding abrupt density changes caused by hard-boundary spherical regions.

[0076] Specifically, the calculation process of the kernel density estimation method is as follows: Let the spatial coordinates of the candidate touch point be x, and there be n virtual objects in the surrounding space, with the center point coordinates of each virtual object being x_i (i=1,2,…,n). Then, the kernel density estimate of the virtual object distribution density ρ(x) at the candidate touch point x is: Wherein, K(·) is the kernel function and h is the bandwidth parameter. The kernel function K(·) satisfies non-negativity, symmetry and normalization. Its function value decreases as the independent variable (i.e., the normalized distance from the candidate touch point to the center point of the virtual object) increases, thereby achieving a distance-weighted effect that assigns higher weights to nearby objects and lower weights to distant objects.

[0077] In one possible implementation, the kernel function is a Gaussian kernel function, whose expression is: ; Where ||u|| is the ratio of the Euclidean distance from the candidate touch point to the center point of the virtual object to the bandwidth h.

[0078] It should be understood that the Gaussian kernel function has an infinite support domain and good smoothness, and can produce continuously differentiable density estimation surfaces.

[0079] The bandwidth parameter h is a key parameter controlling the smoothness of density estimation. A larger h value results in smoother density estimation and reduced sensitivity to local details; a smaller h value makes the density estimation more sensitive to local changes and better reflects subtle differences in object distribution. In one possible implementation, the bandwidth h can be adaptively set based on the overall object density of the virtual scene, for example, by using the average nearest neighbor distance of all virtual objects in the scene as a baseline value for h.

[0080] Compared with hard boundary methods based on spherical region counting, kernel density estimation does not require a preset statistical radius, avoiding density abruptness caused by improper selection of the statistical radius. It can generate a spatially continuous and smooth density distribution field, and more accurately reflect the true degree of aggregation of objects at candidate touch points.

[0081] For example, in a virtual desktop scene, a candidate touch point is located near a group of closely arranged virtual icons. If the system retrieves 6 interactive virtual icons within a spherical detection area with a radius of 10 cm centered on the candidate touch point, the virtual object distribution density is determined to be 6 icons per detection area.

[0082] S1042. Dynamically set the convergence tolerance based on the distribution density.

[0083] The aforementioned convergence tolerance is a spatial angle threshold or spatial distance threshold based on the candidate touch point, used to limit the allowable deviation range when screening target objects, and the convergence tolerance is negatively correlated with the distribution density.

[0084] Specifically, the negative correlation between convergence tolerance and distribution density means that the higher the distribution density of virtual objects around a candidate touch point, the smaller the convergence tolerance value; conversely, the lower the distribution density, the larger the convergence tolerance value. The principle behind this design is that in densely populated areas, where the spacing between virtual objects is small and their directional differences are subtle, a smaller convergence tolerance is needed to improve filtering accuracy and prevent the misselection of adjacent non-target objects. In sparsely populated areas, where the spacing between virtual objects is large and their directional differences are significant, a larger convergence tolerance can be used to reduce the difficulty of selection and prevent users from missing any objects due to slight hand tremors.

[0085] In one possible implementation, the mapping relationship between convergence tolerance and distribution density is defined using a predefined monotonically decreasing function. Let the virtual object distribution density be ρ, and the convergence tolerance be θtolerance. Then, they satisfy θtolerance = f(ρ), where f is a monotonically decreasing function. This monotonically decreasing function can be a linear function, an exponential function, or a piecewise function.

[0086] For example, when using a linear mapping, the formula for calculating the convergence tolerance is: Wherein, θmax is the preset maximum convergence tolerance (the tolerance value corresponding to the density being zero), and θmin is the preset minimum convergence tolerance (the tolerance value corresponding to the density being the highest density ρmax). By adjusting the values ​​of θmax and θmin, the dynamic range of the tolerance can be flexibly controlled.

[0087] When a spatial distance threshold is used as the convergence tolerance, it also shows a negative correlation with the distribution density; that is, the higher the density, the smaller the allowable distance deviation. For example, the distance threshold... The relationship with density ρ can be expressed as: Where dmax and dmin are the preset maximum and minimum distance thresholds, respectively.

[0088] The aforementioned preset maximum and minimum distance thresholds are boundary parameters for the convergence tolerance when using spatial distance thresholds. They correspond to the extreme values ​​of allowable distance deviation when the virtual object distribution density approaches zero and the maximum density, respectively.

[0089] The aforementioned preset maximum distance threshold refers to the most lenient distance tolerance adopted by the system when the distribution density of virtual objects around the candidate touch point is extremely low (such as when there are only a few sparse objects in the scene). In one possible implementation, the maximum distance threshold can be set as a preset multiple of the diagonal length of the average bounding box of all interactive objects in the virtual scene, for example, twice.

[0090] The rationale for this setting is that in sparsely populated scenes, the distance between objects is relatively large. Even if there is a large deviation in the user's hand pointing, it is still highly likely to point near the target object. Using a larger distance tolerance can ensure that the user can easily lock onto the target and avoid failing to select the object due to too small a tolerance.

[0091] The aforementioned preset minimum distance threshold refers to the strictest distance tolerance adopted by the system when the density of virtual objects around the candidate touch point is extremely high (such as objects being tightly stacked).

[0092] In one possible implementation, the minimum distance threshold can be set to half the diagonal length of the bounding box of the smallest interactive object in the virtual scene. This is based on the fact that in densely populated scenes, the edge spacing between adjacent objects is extremely small; to effectively distinguish adjacent objects, the distance tolerance must be less than the minimum distance between objects. Binding the minimum distance threshold to the size of the smallest object in the scene ensures accurate differentiation of different objects even in high-density areas, preventing the accidental selection of adjacent non-target objects.

[0093] For example, if the average diagonal length of the bounding box of objects in the virtual scene is 5 cm and the diagonal length of the smallest bounding box is 1 cm, then the maximum distance threshold can be set to 10 cm and the minimum distance threshold can be set to 0.5 cm. When the density is zero, the convergence tolerance takes the maximum value of 10 cm; when the density reaches its maximum, the convergence tolerance takes the minimum value of 0.5 cm; when the density is in an intermediate state, the convergence tolerance is linearly or nonlinearly interpolated within the above range.

[0094] With the aforementioned preset parameters, the dynamic adjustment of the convergence tolerance can match the scale characteristics and object distribution of different virtual scenes without requiring manual configuration by the user, thus achieving "scene-adaptive" tolerance adjustment.

[0095] In practical applications, the system calculates the convergence tolerance value applicable to the current frame in real time based on the virtual object distribution density ρ obtained in S1041 through the above mapping function, and passes it to the hit determination process in S1043.

[0096] S1043. Using the adjusted hand ray direction and based on the convergence tolerance, perform hit determination on virtual objects in the neighborhood of candidate touch points to identify the target object.

[0097] The specific process for determining the hit rate mentioned above includes: Ⅰ: Determine the spatial neighborhood range of the candidate touch points based on the candidate touch points.

[0098] For example, the spatial neighborhood of a candidate touch point is a spherical region with the candidate touch point as the center and a preset radius.

[0099] In one possible implementation, the preset radius is dynamically determined based on the average size of objects in the virtual scene.

[0100] Specifically, the system pre-calculates the bounding box dimensions of all interactive objects in the virtual scene, calculates their average diagonal length, and sets the preset radius to a preset multiple of this average diagonal length, such as 1.5 times. This allows the spatial neighborhood to adapt to the object distribution characteristics of virtual scenes at different scales.

[0101] The preset radius mentioned above is a parameter that characterizes the effective search range around the candidate touch point. When the preset radius is too large, the neighborhood range may contain too many irrelevant objects, increasing the amount of filtering calculations and introducing interference; when the preset radius is too small, the actual target object that the user intends to touch may be missed.

[0102] In one alternative implementation, the preset radius is also linked to the convergence tolerance. That is, the smaller the convergence tolerance (the denser the objects), the smaller the preset radius is, in order to further reduce the search space and improve the filtering efficiency.

[0103] II: Obtain all virtual objects located within the spatial neighborhood and form a candidate object set.

[0104] In one possible implementation, the system performs fast spatial queries on objects in the virtual scene using a spatial index structure. Using a candidate touch point as the query center and a preset radius as the query range, the system retrieves all virtual objects that intersect with or are contained within the spherical region of the spatial index structure, and adds the search results to the candidate object set. For some virtual objects located within the neighborhood, if the distance from the center of their bounding box to the candidate touch point exceeds the preset radius, they can be excluded to reduce subsequent computational load.

[0105] III: Calculate the spatial direction vector of each virtual object in the candidate object set relative to the candidate touch point.

[0106] In one possible implementation, for each virtual object in the candidate object set, the system first obtains the reference point position of that virtual object.

[0107] The reference point can be the center of the bounding box of the virtual object or the point on the surface of the virtual object that is closest to the candidate touch point. Then, a spatial vector is constructed with the candidate touch point as the starting point and the reference point as the ending point, and the spatial vector is normalized to obtain the spatial direction vector of the virtual object relative to the candidate touch point.

[0108] IV: Calculate the spatial angle deviation between each spatial direction vector and the adjusted hand ray direction.

[0109] In one possible implementation, the spatial angle deviation is obtained by calculating the angle between the spatial direction vector and the adjusted hand ray direction vector.

[0110] Specifically, let the spatial direction vector corresponding to a virtual object be a unit vector A, and the adjusted hand ray direction be a unit vector B. Then the spatial angle deviation θ corresponding to the virtual object is θ = arccos(A·B), where “·” represents the vector dot product operation.

[0111] The value of θ ranges from [0°, 180°]. The smaller the value of θ, the more consistent the direction of the virtual object is with the direction of the hand ray.

[0112] Optionally, to reduce computational complexity, the angle value can be substituted for the vector dot product A·B as a measure of spatial angle deviation during the comparison phase, instead of directly calculating the angle value. Since the cosine function is a monotonically decreasing function in the interval [0°, 180°], a larger dot product value corresponds to a smaller angle deviation. Correspondingly, the convergence tolerance can also be expressed as an angle cosine threshold, and the selection criteria for angle deviation are equivalently transformed into selection criteria where the dot product value is greater than or equal to the angle cosine threshold.

[0113] V: Compare the spatial angle deviation with the convergence tolerance to filter out virtual objects whose spatial angle deviation is less than or equal to the convergence tolerance.

[0114] In one possible implementation, the system iterates through all virtual objects in the candidate object set and compares the spatial angle deviation θ corresponding to each virtual object with the currently set convergence tolerance θ_tolerance. If θ ≤ θ_tolerance, the virtual object is marked as a candidate hit object and added to the hit object set; if θ > θ_tolerance, the virtual object is removed.

[0115] Specifically, when only one virtual object is selected, that virtual object is designated as the target object. When multiple virtual objects are selected, the virtual object with the smallest spatial angle deviation is designated as the target object. If the spatial angle deviations are the same, further arbitration can be performed based on the distance between the virtual object and the candidate touch point, and the virtual object closest to the candidate touch point is designated as the target object. If the number of selected virtual objects is zero (i.e., no virtual object's spatial angle deviation falls within the convergence tolerance range), the hit determination is considered a failure, no target object is returned, and the system can maintain the selected state of the previous frame or enter the waiting state.

[0116] S105. Based on eye-tracking rays and hand rays, determine the target contact position on the surface of the target object.

[0117] In one possible implementation, the touch depth offset is calculated along the common perpendicular direction of the eye movement ray and the hand ray based on the finger micro-movement posture data; on the surface of the target object, the touch depth offset is superimposed with the candidate touch point as a reference to obtain the target contact position.

[0118] S106. Generate a tactile guidance signal based on the target contact position to guide the user to touch the target contact position.

[0119] In one possible implementation, the deviation between the target contact position and the user's current hand position is mapped in real time as a tactile guidance signal; a directional vibration gradient is generated on the user's hand through a force feedback device, the direction of the vibration gradient is towards the target contact position, and the intensity of the vibration gradient is positively correlated with the magnitude of the deviation.

[0120] Example 2 The embodiments described above use virtual reality / augmented reality three-dimensional spatial scenes as examples. The following, in conjunction with the accompanying drawings, provides a detailed description of another eye-hand coordinated ray virtual object touch positioning method on a right-angle screen device.

[0121] In scenarios involving right-angle screen devices (such as foldable phones, tablets, and other touchscreen terminals), users perform touch operations on the screen plane using their fingers. Simultaneously, the device's front-facing camera or built-in eye-tracking module captures the user's gaze direction. Unlike three-dimensional virtual space scenarios, the positioning problem in right-angle screen scenarios degenerates into collaborative positioning on a two-dimensional plane: the eye-tracking ray originates from the user's eye position, passes through three-dimensional space, and projects onto the screen plane to form a gaze projection point. The finger touch position is a touch coordinate point on the screen plane. The system collaboratively fuses the gaze projection point and the touch coordinate point to determine the target interactive element that the user actually intends to touch and its precise touch position.

[0122] Specifically, in this embodiment, the method includes the following steps: S201. Acquire eye-tracking data and touch input data, determine the gaze projection point based on the eye-tracking data, and determine the touch coordinate point based on the touch input data.

[0123] The system uses a front-facing camera or a dedicated eye-tracking sensor to capture the user's gaze direction, extending the gaze from the eye position along the gaze direction to the intersection with the screen plane. This intersection is the gaze projection point. Simultaneously, the system captures the touch coordinates of the user's fingers through the touchscreen. The gaze projection point reflects the point on the screen where the user's visual intention lands, while the touch coordinates reflect the point on the screen where the user's operational intention lands.

[0124] S202. Based on the gaze projection point and the touch coordinate point, determine the candidate touch point.

[0125] The aforementioned candidate touch points are the weighted center points of the gaze projection point and the touch coordinate point on the screen plane.

[0126] Unlike the weighted nearest point of two skew rays in three-dimensional space, the system performs weighted fusion of the gaze projection point and the touch coordinate point on the two-dimensional screen plane.

[0127] Specifically, the system acquires the distribution density and occlusion status of interactive elements on the screen plane, and dynamically adjusts the gaze decision weight and touch decision weight based on the distribution density and occlusion status. The gaze decision weight represents the credibility of the gaze projection point in the positioning decision, and the touch decision weight represents the credibility of the touch coordinate point in the positioning decision.

[0128] The dynamic adjustment rules include: when the distribution density of interactive elements in the area where the candidate touch point is located is higher than a preset threshold and there is visual occlusion, the touch decision weight is increased because the finger touch position is more reliable at this time.

[0129] When visual occlusion occurs (e.g., a finger blocks part of the interactive elements in the user's line of sight), the gaze decision weight is increased because the gaze projection point is more reliable in pointing to the user's true intention. When the tracking confidence of the gaze projection point is lower than a preset threshold, for example, when the user's gaze deviates from the screen or the quality of eye-tracking data decreases.

[0130] Based on the adjusted gaze decision weights and touch decision weights, a weighted center point is calculated for the gaze projection point and touch coordinate point, and this weighted center point is used as a candidate touch point.

[0131] The aforementioned weighted center point is a two-dimensional coordinate position on the screen plane obtained by weighted interpolation of the gaze projection point and the touch coordinate point based on the gaze decision weight and the touch decision weight.

[0132] S203. Based on the target interactive element area, correct the position of the candidate touch point so that the candidate touch point is biased towards the target interactive element area.

[0133] The aforementioned target interactive element area is the area on the screen plane where the interactive element that the user intends to touch is located.

[0134] By analyzing users' eye-tracking behavior, we can predict their touch intentions and identify the target interactive element area.

[0135] Specifically, the system identifies the user's gaze pattern on interactive elements based on the trajectory, speed, and / or duration of the user's gaze point moving on the screen plane. This gaze pattern is the sequence of gaze points formed when the user's gaze jumps between multiple interactive elements and the corresponding temporal relationship, including the duration of gaze on each interactive element and the speed of jumping between gaze points.

[0136] Based on the saccade pattern, the system determines the saccade termination area.

[0137] The specific process includes: determining the gaze intent weight of each interactive element based on the gaze duration corresponding to each interactive element in the saccade mode, where the gaze intent weight is positively correlated with the gaze duration; identifying the saccade convergence phase based on the jump speed between adjacent gaze points in the saccade mode, where the saccade convergence phase is the temporal interval in which the user's gaze continuously converges towards a specific interactive element and the jump speed decreases; determining the saccade termination interactive element based on the interactive element pointed to by the gaze point and the gaze intent weight in the saccade convergence phase; and determining the area on the screen plane where the saccade termination interactive element is located as the saccade termination area.

[0138] The scanning termination area is determined as the target interactive element area, and the position of the candidate touch point is adjusted accordingly, so that the candidate touch point is shifted towards the target interactive element area.

[0139] S204. Based on the corrected candidate touch points, select the target interactive element from the interactive elements in their neighborhood.

[0140] Centered on the corrected candidate touch points, the target interactive elements that the user intends to touch are filtered out within their neighborhood.

[0141] Specifically, the distribution density of interactive elements around the candidate touch point is obtained, and the convergence tolerance is dynamically set according to the distribution density.

[0142] The aforementioned convergence tolerance is a planar distance threshold based on candidate touch points, used to limit the allowable deviation range when filtering target interactive elements. The convergence tolerance is negatively correlated with the distribution density; that is, the denser the interactive elements, the smaller the tolerance and the more accurate the filtering; the sparser the interactive elements, the larger the tolerance and the more lenient the filtering.

[0143] Using candidate touch points as a reference, determine their neighborhood range on the screen plane; obtain all interactive elements within this neighborhood range to form a candidate element set; calculate the planar distance between the center point of each interactive element in the candidate element set and the candidate touch point; compare the planar distance with the convergence tolerance, and filter out interactive elements whose planar distance is less than or equal to the convergence tolerance. When only one interactive element is selected, it is determined as the target interactive element; when multiple interactive elements are selected, the interactive element with the smallest planar distance is determined as the target interactive element.

[0144] S205. Based on the gaze projection point and the touch coordinate point, determine the target touch position within the target interactive element.

[0145] After identifying the target interactive element, the precise touch position of the user's finger within the target interactive element is further determined.

[0146] Specifically, the system calculates the touch position correction amount based on the direction and magnitude of the deviation between the gaze projection point and the touch coordinate point. This touch position correction amount reflects the mapping of the offset of the gaze projection point relative to the touch coordinate point within the target interactive element.

[0147] Within the target interactive element, the target touch position is obtained by overlaying the touch position correction amount based on the candidate touch points.

[0148] S206. Generate tactile feedback signals and / or visual feedback signals based on the target touch position to guide the user to touch the target touch position.

[0149] After determining the target touch location, a feedback signal is generated to guide the user's finger to slide to that location.

[0150] Specifically, the system maps the planar deviation between the target touch position and the current touch coordinate point into tactile feedback signals and / or visual feedback signals in real time.

[0151] For haptic feedback, the system generates directional vibration feedback on the touchscreen using a built-in linear motor. The vibration intensity is positively correlated with the magnitude of the planar deviation; a larger planar deviation results in a stronger vibration to provide stronger guidance, while a smaller planar deviation results in a weaker vibration to avoid excessive interference. For visual feedback, the system renders dynamic indicator animations (such as elastic lines, directional arrows, or pulsed halos) between the touch coordinate point and the target touch location. The animation direction points towards the target touch location, guiding the user's finger to slide towards it.

[0152] Users naturally slide their fingers to the target touch position based on the direction guided by haptic and / or visual feedback to complete precise touch.

[0153] The specific implementation of each step in the above embodiments is consistent with the core logic of the corresponding steps in the above three-dimensional space embodiments. The difference lies in the degradation of the geometric model from three-dimensional space to a two-dimensional screen plane: the weighted nearest point in three-dimensional space degenerates into the weighted center point on the two-dimensional plane, the angular deviation in three-dimensional space degenerates into the distance on the two-dimensional plane, and the three-dimensional force feedback vibration gradient degenerates into two-dimensional tactile feedback and visual feedback. Those skilled in the art will understand that this embodiment belongs to the natural degradation application of the technical solution of this application on a right-angle screen device, and still falls within the protection scope of this application.

[0154] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0155] Example 3 This invention provides a virtual object touch positioning system based on eye-hand coordination rays, such as... Figure 3 As shown, the system includes: The acquisition module 301 is used to acquire eye-tracking data and hand-tracking data, generate eye-tracking rays based on eye-tracking data, and generate hand rays based on hand-tracking data; The first determining module 302 is used to determine candidate touch points based on eye-tracking rays and hand rays; the candidate touch points are the weighted nearest points of eye-tracking rays and hand rays in three-dimensional space; The adjustment module 303 is used to adjust the starting position and / or direction of the hand ray based on the target virtual object area, so that the hand ray is pre-pointed to the target virtual object area; the target virtual object area is the area where the predicted virtual object intended to be touched by the user is located. The filtering module 304 is used to filter out target objects from virtual objects in the neighborhood of candidate touch points based on the adjusted hand ray direction. The second determining module 305 is used to determine the target contact position on the surface of the target object based on eye-tracking rays and hand rays. The guidance module 306 is used to generate a tactile guidance signal based on the target contact position to guide the user to touch the target contact position.

[0156] It should be noted that the implementation of the acquisition module 301 can be found in the detailed description of step S101 in the above method embodiment; the implementation of the first determination module 302 can be found in the detailed description of step S102; the implementation of the adjustment module 303 can be found in the detailed description of step S103; the implementation of the filtering module 304 can be found in the detailed description of step S104; the implementation of the second determination module 305 can be found in the detailed description of step S105; and the implementation of the guidance module 306 can be found in the detailed description of step S106.

[0157] Example 4: This invention also provides a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium can be any medium capable of storing program code, such as a read-only memory, random access memory, magnetic disk, optical disk, or solid-state drive.

[0158] When the computer program is executed by the processor, it implements all the steps of the image content parsing and problem-solving hint generation method described in any of the above method embodiments. The processor may be a central processing unit, a graphics processing unit, a neural network processor, or other processing unit with instruction execution capability.

[0159] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the constituent units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or a complete or partial classification of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0164] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A virtual object touch positioning method based on eye-hand coordination rays, characterized in that, The method includes: Acquire eye-tracking data and hand-tracking data; generate eye-tracking rays based on the eye-tracking data; and generate hand-tracking rays based on the hand-tracking data. Based on the eye-tracking ray and the hand-tracking ray, candidate touch points are determined; the candidate touch points are the weighted closest points of the eye-tracking ray and the hand-tracking ray in three-dimensional space. Based on the target virtual object region, the starting position and / or direction of the hand ray are adjusted so that the hand ray is pre-pointed to the target virtual object region; the target virtual object region is the region where the predicted virtual object intended to be touched by the user is located. Based on the adjusted hand ray direction, the target object is selected from the virtual objects in the neighborhood of the candidate touch point; Based on the eye-tracking rays and the hand-tracking rays, the target contact position on the surface of the target object is determined; A tactile guidance signal is generated based on the target contact location to guide the user to touch the target contact location.

2. The method according to claim 1, characterized in that, The process of determining candidate touch points includes: Obtain the spatial distribution density and occlusion status of objects in a virtual scene; Based on the spatial distribution density and occlusion status, the eye-tracking decision weight and hand-tracking decision weight are dynamically adjusted; the eye-tracking decision weight is a coefficient characterizing the credibility of eye-tracking rays in positioning decisions; the hand-tracking decision weight is a coefficient characterizing the credibility of hand-tracking rays in positioning decisions. Based on the adjusted eye-tracking decision weights and hand-tracking decision weights, a weighted nearest point calculation is performed on the eye-tracking ray and the hand-tracking ray, and the calculation result is used as the candidate touch point. The weighted nearest point calculation is to obtain the spatial point position by weighted interpolation of the two perpendicular points on the common perpendicular line of the eye-tracking ray and the hand-tracking ray according to the eye-tracking decision weights and the hand-tracking decision weights.

3. The method according to claim 1, characterized in that, Determining the target virtual object region includes: Based on the trajectory, speed and / or duration of the user's gaze movement, the user's gaze pattern on virtual objects is identified; the gaze pattern is the sequence of gaze points formed when the user's gaze jumps between multiple virtual objects and the corresponding temporal relationship, including the duration of gaze on each virtual object and the speed of jumping between gaze points. Based on the scanning pattern, determine the scanning termination area; The scanning termination area is determined as the target virtual object area.

4. The method according to claim 3, characterized in that, The step of determining the saccade termination region based on the saccade pattern includes: Based on the gaze duration corresponding to each virtual object in the saccade mode, the gaze intent weight of each virtual object is determined; the gaze intent weight is positively correlated with the gaze duration. Based on the jump speed between adjacent gaze points in the saccade pattern, the saccade convergence phase is identified; the saccade convergence phase is the time interval in which the user's gaze continuously converges on a specific virtual object and the jump speed decreases. Based on the virtual object to which the gaze point points during the saccade convergence phase and the gaze intention weight, determine the virtual object from which the saccade terminates. The spatial region where the virtual object where the saccade ends is located is defined as the saccade end region.

5. The method according to claim 1, characterized in that, The step of filtering the target object from the virtual objects in the neighborhood of the candidate touch point includes: Obtain the distribution density of virtual objects around the candidate touch point; The convergence tolerance is dynamically set based on the distribution density, wherein the convergence tolerance is a spatial angle threshold or spatial distance threshold based on the candidate touch point, used to limit the allowable deviation range when screening target objects; the convergence tolerance is negatively correlated with the distribution density; Using the adjusted hand ray direction and based on the convergence tolerance, the virtual objects in the neighborhood of the candidate touch point are hit to determine the target object.

6. The method according to claim 5, characterized in that, The step of using the adjusted hand ray direction, based on the convergence tolerance, to determine the target object by hitting virtual objects in the neighborhood of the candidate touch point, includes: Based on the candidate touch points, determine the spatial neighborhood range of the candidate touch points; All virtual objects located within the spatial neighborhood are obtained to form a candidate object set; Calculate the spatial orientation vector of each virtual object in the candidate object set relative to the candidate touch point; Calculate the spatial angular deviation between each of the aforementioned spatial direction vectors and the adjusted hand ray direction; The spatial angle deviation is compared with the convergence tolerance, and virtual objects whose spatial angle deviation is less than or equal to the convergence tolerance are selected. When the number of selected virtual objects is one, the virtual object is determined as the target object; When there are multiple virtual objects selected, the virtual object with the smallest spatial angle deviation is determined as the target object.

7. The method according to claim 1, characterized in that, Determining the target contact position on the surface of the target object includes: Based on the finger micro-movement posture data, the touch depth offset is calculated along the common perpendicular direction of the eye movement ray and the hand ray; On the surface of the target object, the target contact position is obtained by superimposing the touch depth offset on the candidate touch point.

8. The method according to claim 1, characterized in that, Generating a tactile guidance signal based on the target contact location to guide the user to touch the target contact location includes: The deviation between the target contact position and the user's current hand position is mapped into a tactile guidance signal in real time; A directional vibration gradient is generated on the user's hand using a force feedback device. The direction of the vibration gradient points towards the target contact position, and the intensity of the vibration gradient is positively correlated with the magnitude of the deviation.

9. The method according to claim 2, characterized in that, Based on the spatial distribution density and occlusion status, the eye-tracking decision weights and hand-tracking decision weights are dynamically adjusted, including: When the virtual object distribution density in the area where the candidate touch point is located is higher than a preset threshold and there is occlusion, the eye-tracking decision weight is increased; or, when the gaze depth confidence of the eye-tracking ray is lower than a preset threshold, the hand decision weight is increased.

10. A virtual object touch positioning system based on eye-hand coordination rays, characterized in that, The system includes: The acquisition module is used to acquire eye-tracking data and hand-tracking data, generate eye-tracking rays based on the eye-tracking data, and generate hand rays based on the hand-tracking data. The first determining module is used to determine candidate touch points based on the eye-tracking ray and the hand-tracking ray; the candidate touch point is the weighted nearest point of the eye-tracking ray and the hand-tracking ray in three-dimensional space; An adjustment module is used to adjust the starting position and / or direction of the hand ray based on the target virtual object region, so that the hand ray is pre-pointed to the target virtual object region; the target virtual object region is the region where the predicted virtual object intended to be touched by the user is located; A filtering module is used to filter out target objects from virtual objects in the neighborhood of the candidate touch point based on the adjusted hand ray direction. The second determining module is used to determine the target contact position on the surface of the target object based on the eye movement ray and the hand ray; The guidance module is used to generate a tactile guidance signal based on the target contact position to guide the user to touch the target contact position.

Images