Method, device and system for preventing fingertip kneading mold penetration in VR gesture tracking and application

Abstract

The invention relates to a method, a device and a system for preventing fingertip pinching mold penetration in VR gesture tracking and application. The method comprises the following steps: acquiring real-time three-dimensional space coordinates of a thumb fingertip joint point and an index finger fingertip joint point, and calculating a distance d between the thumb fingertip joint point and the index finger fingertip joint point; establishing a multi-level threshold group including an approach threshold and a contact threshold, establishing an anti-mold-penetrating preparation mode and an anti-mold-penetrating constraint mode, and triggering a corresponding mode based on the relationship between d and the multi-level threshold group; after the anti-mold-penetration preparation mode is triggered, the three-dimensional space coordinate positions of the thumb fingertip joint point and the index finger fingertip joint point are corrected; driving a virtual hand model to render based on the corrected fingertip position and other uncorrected hand joint point data; the device and the system are implemented based on the method, and are applied to the correction of kneading mold penetration in a VR man-machine gesture interaction process. According to the method, the interaction naturalness and immersion are remarkably improved, stable system performance is ensured, the problem of physical authenticity of micro-interaction is solved, and coherence, smoothness and naturalness of the whole hand posture are ensured.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and in particular to a method, apparatus, system, and application for preventing fingertip pinching from clipping through VR gesture tracking. Background Technology

[0002] In VR / AR systems, natural gesture interaction is key to enhancing immersion. Currently, vision-based gesture tracking technology has become the mainstream solution. This technology typically captures images of the user's hand through optical sensors on a head-mounted device and uses computer vision algorithms to identify the three-dimensional spatial coordinates and posture of multiple joints of the hand in real time. This joint data forms a virtual skeleton, which drives the movement of a pre-made virtual hand model with detailed skin and skeletal structure, thereby mapping the user's real hand movements into the virtual environment.

[0003] Although the above technologies can achieve a smooth mapping of the overall hand posture, they have an inherent "clipping" defect when dealing with the high-frequency and critical interaction action of fine fingertip pinching. That is, when the user perceives that the tips of the thumb and index finger have touched or pinched, the corresponding virtual fingertip models may overlap in the rendered screen, which seriously undermines the visual realism and physical credibility of the interaction. The occurrence of "clipping" is due to the fact that the user's perception of "contact" is subjective and continuous, while the system relies on discrete and noisy sensor data. The fingertip coordinates output by the algorithm itself have errors and are difficult to accurately correspond to the zero-distance contact point in the physical world. At the same time, in pursuit of rendering efficiency, virtual hand models often use simplified colliders, and even the fingertip area does not have independent collision constraints. When pinching quickly, discrete physical detection frames may "skip" the moment of actual contact. Furthermore, traditional general-purpose rigid body physics engines have huge computational overhead, and their response latency and iterative solution methods are difficult to meet the high-frequency, real-time, and lightweight anti-penetration requirements of fingertip pinching. Under the premise of maintaining a high rendering frame rate, directly applying complex physical detection often leads to performance bottlenecks or slow response.

[0004] Existing technologies used to mitigate clipping issues have limited effectiveness. For example, methods that forcibly separate the model upon detecting clipping result in a sudden, abrupt visual change, a jarring experience, and fail to prevent clipping altogether. Similarly, pre-enlarging the size of the fingertip collider, while triggering contact detection earlier, reduces interaction precision, making the virtual hand appear large or imprecise, which is unacceptable, especially in scenarios requiring delicate manipulation. While replacing the tracking device with a more advanced one could be an option, this would significantly increase system cost and complexity, and would not fundamentally address the deficiencies in the algorithmic mapping logic. Summary of the Invention

[0005] This invention solves the problems existing in the prior art and provides a method, device, system and application for preventing fingertip pinching clipping in VR gesture tracking. It is efficient and has low computational overhead. By monitoring the distance between fingertips in real time and dynamically constraining the virtual model, it ensures the visual correctness when fingertips make contact, thereby improving the immersion and accuracy of the interaction.

[0006] The technical solution adopted in this invention is a method to prevent fingertip pinching from clipping through VR gesture tracking, which obtains the real-time three-dimensional spatial coordinates of the thumb fingertip joint and the index finger fingertip joint and calculates the distance d between them; Establish a multi-level threshold group, including proximity threshold and contact threshold; establish an anti-penetration preparatory mode and an anti-penetration constraint mode; and trigger the corresponding mode based on the relationship between d and the multi-level threshold group. After triggering the anti-molding preparation mode, the three-dimensional spatial coordinates of the thumb tip joint and the index finger tip joint are corrected. Based on the corrected fingertip position and other uncorrected hand joint data, the virtual hand model is driven to render.

[0007] Preferably, when it is determined that the anti-drilling preparation mode has been entered, a smoothing function value based on d is set, wherein the smoothing function value is between the contact threshold and d, and smoothly decays towards the contact threshold over time.

[0008] Preferably, the position correction includes symmetric correction and asymmetric correction.

[0009] Preferably, symmetrical correction involves moving the two fingertips by equal displacements in opposite directions along the line connecting them; asymmetrical correction involves assigning different displacements to the two fingertips based on a preset weight ratio or the master-slave relationship of the current hand movement.

[0010] Preferably, when it is determined that the anti-drilling constraint mode has been entered, the minimum constraint distance is set to the contact threshold or physical tolerance value.

[0011] Preferably, based on the recognition results of fingertip movement speed, acceleration, and / or user interaction intent in historical frames, at least one threshold in the multi-level threshold group or the smoothing function value is dynamically adjusted.

[0012] Preferably, the multi-level threshold group further includes a separation threshold, which releases the anti-mold-breaking constraint mode when the three-dimensional spatial distance between the thumb fingertip joint and the index finger fingertip joint is greater than the separation threshold.

[0013] A device for preventing fingertip pinching clipping during VR gesture tracking includes: The data acquisition and calculation module is used to collect the real-time three-dimensional positions of the thumb and index finger joints and calculate the spatial distance between them. The anti-penetration module includes an anti-penetration preparatory mode and an anti-penetration constraint mode, which trigger the corresponding mode based on the relationship between spatial distance and multi-level threshold groups; The position correction module is used to correct the three-dimensional spatial coordinates of the thumb and index finger joints after the anti-drilling preparation mode is triggered. The data output module is used to output the position-corrected thumb and index finger joint data, as well as other uncorrected hand joint data, to drive the virtual hand model.

[0014] A system for preventing fingertip pinching clipping during VR gesture tracking includes a processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method for preventing fingertip pinching from clipping through VR gesture tracking according to the instructions in the program code.

[0015] An application of the method for preventing fingertip pinching clipping in VR gesture tracking is used to correct pinching clipping during VR human-computer gesture interaction.

[0016] This invention relates to a method, apparatus, system, and application for preventing fingertip clipping during VR gesture tracking. The method involves acquiring the real-time three-dimensional spatial coordinates of the thumb and index finger joints and calculating the distance *d* between them; establishing a multi-level threshold group, including an approach threshold and a contact threshold; establishing an anti-clipping preparatory mode and an anti-clipping constraint mode; triggering the corresponding mode based on the relationship between *d* and the multi-level threshold group; after triggering the anti-clipping preparatory mode, correcting the position of the three-dimensional spatial coordinates of the thumb and index finger joints; driving the rendering of a virtual hand model based on the corrected fingertip position and other uncorrected hand joint data; and implementing the apparatus and system based on this method, which are applied to correcting pinch clipping during VR human-computer gesture interaction.

[0017] The beneficial effects of this invention are as follows: (1) Constructing a multi-level threshold group can predict potential clipping risks in advance and apply constraints progressively, overcoming the visual abruptness problem caused by traditional binarization judgment, making the transition from free tracking to anti-clipping constraints smooth and timely, avoiding clipping in advance without the user's awareness, and significantly improving the naturalness and immersion of the interaction. (2) Linear constraints and position corrections are applied directly to the core thumb and index finger joints to avoid complex collision volume calculations and iterative solutions. Real-time anti-penetration protection is achieved with extremely low data processing and computational overhead, ensuring stable system performance in high frame rate rendering scenarios and solving the problem of physical realism of micro-interactions in VR / AR. (3) By precisely constraining and correcting key fingertip nodes locally, and then reintegrating them into the complete virtual hand skeleton driving pipeline for rendering, it ensures that the fingertip contact area does not experience model penetration that violates physical laws, and also ensures the continuity, smoothness and naturalness of the entire hand posture by maintaining the natural motion data of other joints. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating a specific implementation of the present invention; Figure 3 This is a block diagram of the device structure of the present invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0020] This invention relates to a method for preventing fingertip pinching from clipping through VR gesture tracking, the method comprising the following steps: (1) Obtain the real-time three-dimensional spatial coordinates of the thumb tip joint and the index finger tip joint and calculate the distance d between them; (2) Establish a multi-level threshold group, including proximity threshold and contact threshold, establish anti-penetration preparatory mode and anti-penetration constraint mode, and trigger the corresponding mode based on the relationship between d and the multi-level threshold group; (3) After triggering the anti-mold preparation mode, the three-dimensional spatial coordinates of the thumb fingertip joint and the index finger fingertip joint are corrected. (4) Based on the corrected fingertip position and other uncorrected hand joint data, drive the virtual hand model to render.

[0021] The method will be described below with reference to specific implementation methods.

[0022] (1) Obtain the real-time three-dimensional spatial coordinates of the thumb tip joint and the index finger tip joint and calculate the distance d between them; In this invention, the system continuously acquires the three-dimensional spatial coordinates of the thumb and index finger joints output by the gesture tracking algorithm, and calculates the Euclidean distance d between these two points as a basic indicator for judging the pinching state; this is a conventional technique in the field, and those skilled in the art can perform the calculation according to their needs.

[0023] (2) Establish a multi-level threshold group, including proximity threshold and contact threshold, establish anti-penetration preparatory mode and anti-penetration constraint mode, and trigger the corresponding mode based on the relationship between d and the multi-level threshold group; Obviously, we define a threshold T. approach and contact threshold Tcontact When the distance d is less than T approach When the fingertip is determined to be in a "near pinch" state, the anti-mold-breaking preparation mode is activated. When the distance d further decreases to less than or equal to T... contact When the fingertip is determined to have "touched," the anti-mold-breaking constraint is forcibly activated; in practical applications, T contact It is usually set to slightly greater than 0 to accommodate the model volume, T contact <T approach .

[0024] (3) After triggering the anti-mold preparation mode, the three-dimensional spatial coordinates of the thumb fingertip joint and the index finger fingertip joint are corrected. In other words, this method begins to dynamically correct the fingertip position based on distance constraints as soon as a pinching trend is detected; When the anti-drilling constraint mode is entered, the minimum constraint distance is set to the contact threshold or physical tolerance value.

[0025] The position correction includes symmetric correction and asymmetric correction.

[0026] Symmetric correction involves moving the two fingertips by equal displacements in opposite directions along the line connecting them; asymmetric correction assigns different displacements to the two fingertips based on a preset weighting ratio or the master-slave relationship of the current hand movement; after correction, the Euclidean distance between the two fingertip joints is at least d. min This is the minimum distance constraint, and the initial value is generally T. contact This ensures that the model does not penetrate.

[0027] Furthermore, when it is determined that the anti-denturing preparation mode has been entered, a smoothing function value based on d is set. The smoothing function value is between the contact threshold and d, and it smoothly decays towards the contact threshold over time to avoid visual abrupt changes.

[0028] In fact, in practical applications, the widespread use of big data models has also led to the following: Based on the recognition results of fingertip movement speed, acceleration and / or user interaction intent in historical frames, dynamically adjust at least one threshold in the multi-level threshold group or adjust the smoothing function value; In this invention, key parameters of the anti-mold-breaking system can be dynamically adjusted by real-time analysis of the user's fingertip movement trends and interaction intentions. Specifically, by analyzing the movement data of the thumb and index finger joints in historical frames, including but not limited to relative speed and acceleration, and combining this with the recognition of the current interaction scenario or intention, such as determining the user's intention to grab or trigger a rapid gesture, the T-shaped adjustment system can be adjusted online. approach T contact d min Or one or more of the smoothing parameters; For example, when high-speed kneading is detected, T should be appropriately increased. approach Furthermore, the response speed of the smoothing function should be increased, and anti-mold-breaking preprocessing should be initiated in advance to avoid mold breaking due to high-speed movement; for example, when low-speed fine movements are required, T should be appropriately reduced. contact This reduces the response speed of the smoothing function, allowing the virtual fingertip to be "closer" to the real position, improving the accuracy of operation and the realism of tactile feedback; In the specific implementation process, an index containing preset parameter configurations for different interaction scenarios can be set, and a lightweight classifier can be used to determine the most likely interaction mode in real time, thereby switching to the corresponding parameter configuration; motion data analysis can be achieved by caching historical frame coordinates and calculating differences, with extremely low computational overhead; This adjustment is controllable; it can be chosen whether to apply it and its weight can be adjusted. It is mainly used for optimizing corrections.

[0029] In practical applications, the multi-level threshold group also includes a separation threshold. When the three-dimensional spatial distance between the thumb fingertip joint and the index finger fingertip joint is greater than the separation threshold, the anti-mold-breaking constraint mode is released and normal gesture tracking mapping is restored.

[0030] (4) Based on the corrected fingertip position and other uncorrected hand joint data, drive the virtual hand model to render; Specifically, using the corrected thumb and index finger joint positions, combined with the uncorrected joint data of other hand joints, the entire virtual hand's skeletal and skinned model is re-rendered. This step ensures a natural overall hand posture while the fingertip area meets anti-clipping constraints.

[0031] The present invention also relates to a device for preventing fingertip pinching clipping during VR gesture tracking, comprising: The data acquisition and calculation module is used to collect the real-time three-dimensional positions of the thumb and index finger joints and calculate the spatial distance between them. The anti-penetration module includes an anti-penetration preparatory mode and an anti-penetration constraint mode, which trigger the corresponding mode based on the relationship between spatial distance and multi-level threshold groups; The position correction module is used to correct the three-dimensional spatial coordinates of the thumb and index finger joints after the anti-drilling preparation mode is triggered. The data output module is used to output the position-corrected thumb and index finger joint data, as well as other uncorrected hand joint data, to drive the virtual hand model.

[0032] The present invention also relates to a system for preventing fingertip pinching from clipping through VR gesture tracking, comprising a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method for preventing fingertip pinching from clipping through VR gesture tracking according to the instructions in the program code.

[0033] This invention also relates to the application of the method for preventing fingertip pinching clipping during VR gesture tracking, which is used to correct pinching clipping during VR human-computer gesture interaction.

[0034] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0035] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0036] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0037] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0038] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0039] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for preventing fingertip pinching from clipping through the frame during VR gesture tracking, characterized in that: Obtain the real-time three-dimensional spatial coordinates of the thumb tip joint and the index finger tip joint, and calculate the distance d between them; Establish a multi-level threshold group, including proximity threshold and contact threshold; establish an anti-penetration preparatory mode and an anti-penetration constraint mode; and trigger the corresponding mode based on the relationship between d and the multi-level threshold group. After triggering the anti-molding preparation mode, the three-dimensional spatial coordinates of the thumb tip joint and the index finger tip joint are corrected. Based on the corrected fingertip position and other uncorrected hand joint data, the virtual hand model is driven to render.

2. The method for preventing fingertip pinching from clipping through in VR gesture tracking according to claim 1, characterized in that: When it is determined that the anti-drilling preparation mode has been entered, a smoothing function value based on d is set. The smoothing function value is between the contact threshold and d, and it smoothly decays towards the contact threshold over time.

3. The method for preventing fingertip pinching through clipping in VR gesture tracking according to claim 2, characterized in that: The position correction includes symmetric correction and asymmetric correction.

4. The method for preventing fingertip pinching from clipping through in VR gesture tracking according to claim 3, characterized in that: Symmetrical correction involves moving the two fingertips by equal displacements in opposite directions along the line connecting them; asymmetric correction assigns different displacements to the two fingertips based on a preset weight ratio or the master-slave relationship of the current hand movement.

5. A method for preventing fingertip pinching from clipping through in VR gesture tracking according to claim 2, characterized in that: When the anti-drilling constraint mode is entered, the minimum constraint distance is set to the contact threshold or physical tolerance value.

6. The method for preventing fingertip pinching from clipping through in VR gesture tracking according to claim 2, characterized in that: Based on the recognition results of fingertip movement speed, acceleration, and / or user interaction intent from historical frames, at least one threshold in the multi-level threshold group or the smoothing function value is dynamically adjusted.

7. The method for preventing fingertip pinching from clipping through in VR gesture tracking according to claim 1, characterized in that: The multi-level threshold group also includes a separation threshold, which releases the anti-mold-breaking constraint mode when the three-dimensional spatial distance between the thumb fingertip joint and the index finger fingertip joint is greater than the separation threshold.

8. A device for preventing fingertip pinching from clipping through the frame during VR gesture tracking, characterized in that: include: The data acquisition and calculation module is used to collect the real-time three-dimensional positions of the thumb and index finger joints and calculate the spatial distance between them. The anti-penetration module includes an anti-penetration preparatory mode and an anti-penetration constraint mode, which trigger the corresponding mode based on the relationship between spatial distance and multi-level threshold groups; The position correction module is used to correct the three-dimensional spatial coordinates of the thumb and index finger joints after the anti-drilling preparation mode is triggered. The data output module is used to output the position-corrected thumb and index finger joint data, as well as other uncorrected hand joint data, to drive the virtual hand model.

9. A system for preventing fingertip pinching and clipping during VR gesture tracking, characterized in that: Including the processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute, according to the instructions in the program code, the method for preventing fingertip pinching through the mold in VR gesture tracking as described in any one of claims 1 to 7.

10. The application of the method for preventing fingertip pinching clipping in VR gesture tracking as described in any one of claims 1 to 7, characterized in that: It is applied to the correction of pinching and clipping during VR human-computer gesture interaction.

Images