A method and system for three-dimensional gesture manipulation of virtual models in an augmented reality environment

Abstract

This invention discloses a method and system for manipulating 3D virtual models in an augmented reality environment, belonging to the field of augmented reality human-computer interaction. The invention includes a data acquisition and processing module, a virtual hand module, a collision calculation module, and a gesture intent recognition module. Based on the physical characteristics of object grasping and the augmented reality environment, this invention constructs "grasping pairs" conditions and builds a grasping intent recognition algorithm based on these "grasping pairs." The grasping intent recognition algorithm and the "grasping pair" conditions determine the grasping state, eliminating the need for contact calculations based on multiple contact points to determine whether a grasp has been completed. This makes grasping intent judgment more flexible, closer to real 3D gesture manipulation, more adaptable to complex gesture interaction scenarios, and more consistent with the user's intuitive interaction experience. Furthermore, if multiple pairs of "grasping pairs" exist, the contact points constituting multiple grasping pairs will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency, and immersion of gesture interaction intent recognition.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality human-computer interaction, and particularly relates to a method for manipulating virtual models with gestures in an augmented reality environment. Background Technology

[0002] Augmented Reality (AR) is a technology that overlays virtual information onto the real environment, achieving a fusion of virtual and real information. In AR environments, because information is presented in a stereoscopic, three-dimensional manner, traditional interaction methods based on additional input devices such as keyboards and mice are no longer applicable, as these methods hinder a seamless interactive experience. Therefore, natural interaction methods, including gestures, voice, body language, and eye tracking, are more frequently researched and applied in AR. Among these, gesture interaction has a greater advantage than other interaction methods in directly interacting with 3D models, enabling multi-degree-of-freedom manipulation of 3D models. For example, in AR assembly systems, gesture interaction can provide a natural and intuitive user interface for manipulating virtual parts or tooling fixtures during virtual assembly.

[0003] Currently, gesture interaction solutions fall into two categories: two-dimensional (2D) and three-dimensional (3D). Two-dimensional gesture interaction solutions are generally designed for AR systems on mobile devices such as smartphones and tablets, allowing for simple interactions with models through 2D (planar) gestures, such as dragging virtual objects. However, since AR information is three-dimensional, 2D gesture interaction methods are not intuitive or accurate enough. Compared to 2D gesture interaction, 3D gesture interaction methods support user interaction with virtual objects in three-dimensional space, which is more in line with human intuition and experience. For example, Chinese Patent Publication No. CN110221690B discloses a gesture interaction method based on AR scenes, providing a new gesture interaction approach that can accurately present the occlusion relationship between the hand and the virtual object, as well as the "contact" between the hand and the virtual object, enabling more interactive actions between the hand and the virtual object, and achieving a more realistic interactive experience between the user and virtual objects in AR scenes. However, current 3D gesture interaction technology solutions have some shortcomings and problems. Most of them rely on fixed gesture recognition results to drive the movement or rotation of virtual objects, and cannot support natural interaction between the hands and virtual objects, such as grasping. Moreover, the user's adjustment of the virtual object's pose during actual operation is not natural enough, and it is difficult to accurately place the virtual object in the position expected by the user. Summary of the Invention

[0004] The main objective of this invention is to provide a method and system for manipulating virtual models with three-dimensional gestures in an augmented reality environment. In an augmented reality environment, the system can efficiently and accurately identify the user's natural grasping intentions toward the three-dimensional virtual model, support the user to move and rotate virtual objects with natural gestures, improve the robustness of the three-dimensional gesture interaction process, make the gesture interaction experience more intuitive and natural, thereby enhancing the virtual effect of augmented reality on three-dimensional gesture manipulation and improving the user's sense of immersion.

[0005] The objective of this invention is achieved through the following technical solution.

[0006] This invention discloses a method for manipulating a virtual model in three dimensions using gestures in an augmented reality environment, comprising the following steps:

[0007] Step 1: Acquire images of both hands in the current frame, and determine the position and pose data of key hand nodes relative to the AR device based on a convolutional neural network (CNN) hand pose estimation algorithm. The CNN algorithm consists of two CNNs for 2D to 3D hand pose estimation. The first CNN is trained to locate the hands and estimate the 2D position of the hand center in the image. Then, a localized image of the hand's location, along with the corresponding input depth value, is used to generate a normalized cropped image. This image is passed to the second CNN to regress the relative 3D hand joint positions in real time.

[0008] Step 2: At the key nodes of both hands identified in the current frame, overlay virtual hand models, and determine the position and posture of the virtual hand models according to the position and posture of the key nodes, so as to realize the mapping of real hands in virtual space.

[0009] The virtual hand model consists of several virtual joint models, each of which is a cylinder, approximately simulating the finger joints of a real hand. There are topological relationships between the virtual hand joint models; that is, higher-level virtual joint models contain lower-level virtual joint models, and movement of a higher-level virtual joint model will cause movement of the lower-level virtual joint models. The virtual hand model is represented by the following parameterization:

[0010]

[0011] Where Jointti is the i-th virtual joint model, pi is the position of the virtual joint model, represented by a set of vectors xi, yi and zi in the augmented reality environment coordinate system, ei is the pose of the virtual joint model, represented by a set of vectors wi, ri and li in the augmented reality environment coordinate system, Size is the parameter of the virtual joint model, li represents the length of the cylinder, di represents the diameter of the cylinder, children represents the child joint models driven by this virtual joint model, and Jk represents the k-th virtual joint model.

[0012] Each virtual joint model in the virtual hand model will correspond to a key node in the gesture tracking and recognition step one. The position and pose data of each recognized key hand node will be used to update the position and pose of the virtual joint model in the current frame. The position and pose of the virtual hand model are determined based on the position and pose of the key nodes, realizing the mapping of real hands in virtual space, as shown in the following formula:

[0013]

[0014] r i =R z (w i )R y (r i )R x (l i (5)

[0015] Where pi is the position vector of the i-th virtual joint model, ri is the rotation matrix of the virtual joint model, and the transformation relationship between this rotation matrix and Euler angles is shown in the equation, where Rz(wi) represents the rotation around the z-axis by wi degrees; T is the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model is located. Pi is the position vector of the key node corresponding to the i-th virtual joint model, and ri is the rotation matrix of the key node corresponding to the i-th virtual joint model.

[0016] Step 3: Based on the collision detection algorithm, calculate in real time in each frame whether there is contact or collision between the virtual hand model and other virtual models to be manipulated.

[0017] Step 4: Construct "grasping pairs" based on the physical characteristics of real-world object grasping and the augmented reality environment, and build a grasping intent recognition algorithm based on these "grasping pairs". If the collision detection algorithm described in Step 3 detects contact between the hands and other virtual models, the grasping intent recognition algorithm calculates whether a "grasping pair" can be formed between multiple contact points of the hands and the manipulated model to determine whether a grasping situation exists between the hands and the virtual model. A "grasping pair" consists of two contact points. If there is more than one "grasping pair", the grasped virtual model is determined to be in a grasping state. There is no need to calculate the contact based on multiple contact points to determine whether a grasping is complete, making the grasping intent judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. At the same time, if there are multiple pairs of "grasping pairs", the contact points that make up the multiple grasping pairs will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency, and immersion of gesture interaction intent recognition.

[0018] The physical process of grasping a real object is characterized by applying the basic laws of Newtonian rigid body mechanics. It determines whether the object can be grasped based on whether it is in force balance and the friction between the contact surfaces of the virtual hand model and the manipulated model. The principle behind this is to analyze the force state of the object using a simplified Coulomb friction model.

[0019] The aforementioned "grip pair" consists of two contact points of a virtual hand model that meets certain conditions and the model being gripped. The conditions for the "grip pair" are as follows: the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α; in this case, the two contact points will form a stable grip pair g(a,b). The fixed angle α is the friction angle.

[0020] The grasping intent recognition algorithm is established based on the "grasping pair" condition, iteratively determining whether all current contact points can form a "grasping pair" with another contact point. For any two contact points a and b between the virtual hand and the virtual object in one iteration, if the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α, then the two contact points will form a stable grasping pair g(a,b). This fixed angle α is the friction angle, meaning the grasping pair g(a,b) should satisfy...

[0021]

[0022] Where, n a and n b Let l be the normal vector for contact points a and b, which is the normal vector of the cylindrical surface of the joint virtual model at the contact point; ab Let α be the line connecting contact points a and b; α is the friction angle, the value of which needs to be set through testing for the specific manipulated model to ensure stable and natural gripping of the virtual component.

[0023] Step 5: Construct a gripping center acquisition method based on the "grip pair" conditions established in Step 4 to obtain the gripping center. If the gripping intent recognition algorithm in Step 4 determines that the virtual model is in a gripping state, the manipulation intent recognition algorithm calculates the virtual force or torque applied by both hands to the virtual model based on the displacement and posture changes of the gripping center on the manipulated model, and uses the virtual force or torque to drive the movement or rotation of the virtual model. By adopting the manipulation intent recognition algorithm and adding the gripping center judgment condition, all contact points participate in the manipulation intent recognition process, making manipulation intent recognition more flexible and improving its robustness.

[0024] The grasping center is the center point representing the movement of the entire hand. The entire hand is regarded as a complete rigid body, and the position, posture, and velocity of the grasping center represent the motion parameters of the entire virtual hand.

[0025] The method for determining the gripping center is as follows: The position and number of "grip pairs" are determined based on the "grip pair" conditions established in step four. The "grip pair" is considered a unified rigid body, and its position and orientation are represented by the gripping center. If a "grip pair" exists, the gripping center is the center of the line connecting the contact points that make up the "grip pair." The position and orientation of the gripping center are calculated as follows:

[0026]

[0027]

[0028] Where Pc represents the gripping center position, p1 and p2 represent the positions of the contact points that make up the "grip pair", and wc, rc, and lc represent the three Euler angle parameters of the gripping center, respectively. and This represents the unit vector pointing to the x, y, and z axes in the current coordinate system.

[0029] If there are multiple "grip pairs", the length of the line connecting the contact points of the multiple "grip pairs" will be used to determine the "grip pair" with the longest line length as the main gripp pair, and the gripping center will be constructed according to the formulas (7) and (8).

[0030] Step 5.1: Determine whether the "grip pair" meets the "grip pair" cancellation condition. If the cancellation condition is met, it is considered that the user has put down the manipulated virtual model, and the subsequent steps will not be executed. The position and pose of the virtual model will not be updated in the next frame. If the condition is not met, proceed to step 5.2.

[0031] The calculation of the "grip pair" cancellation condition is as follows:

[0032]

[0033] in, The distance between the two contact points that make up the "grip pair" in the current i-th frame. Let k be the distance between the two contact points that make up the "grip pair" in the (i-1)th frame, and k is a fixed value. That is, if the two contact points that make up the "grip pair" move further apart between two frames and the distance meets a certain threshold, then the grip is considered canceled.

[0034] Step 5.2: Calculate the virtual force or torque applied by both hands to the virtual model based on the manipulation intent recognition algorithm, and continue to step 5.3. The manipulation intent recognition algorithm calculates the virtual force or torque applied by both hands to the virtual model in the current frame based on the pose change trend of the grasping center, and calculates the movement and rotation parameters of the virtual model based on the virtual force or torque. These parameters include the direction and distance of movement, and the direction and angle of rotation. By employing the manipulation intent recognition algorithm and adding a "grasping center" condition judgment, all contact points participate in the manipulation intent recognition process, making manipulation intent recognition more flexible and improving its robustness.

[0035] The aforementioned maneuvering intent recognition algorithm is constructed based on a virtual linear and torsional spring-dampers model. The calculation formula for the maneuvering intent recognition algorithm is shown below.

[0036]

[0037]

[0038] Formula (11) is the formula for calculating virtual force, f vf The virtual control force is represented by formula (12), which is the formula for calculating the virtual torque, τ. vf This represents the virtual manipulation torque. The posture of both hands contacting the center point in the current i-th frame is represented as... In the (i+1)th frame, the gesture of both hands touching the center point is represented as (qi+1l, qi+1o). Let be the three-dimensional position of the hand in the i-th frame. A quaternion describing the orientation of the hand; and Let K be the linear velocity and angular velocity of the manipulated virtual model in frame i. sl (K so ) and K Dl (K Do ) represents the coefficients of the linear and torsional spring damping models. These are adjusted by K. sl (K so ) and K Dl (K Do The coefficient enables stable and smooth dynamic movement of virtual components, conforming to the user's intuitive interactive experience.

[0039] Step 5.3: Based on the virtual force or torque calculated by the manipulation intent recognition algorithm described in Step 5.2, the displacement and rotation changes of the virtual model are calculated using rigid body dynamics. The position and orientation of the manipulated virtual model are updated in the current frame based on the displacement and rotation changes, and the virtual model is rendered based on the new position and orientation.

[0040] The formula for calculating the change in displacement is as follows:

[0041]

[0042] Where Si represents the displacement of the manipulated virtual model in the current i-th frame, vi represents the velocity of the manipulated virtual model in the current i-th frame, Δt represents the time difference between the current i-th frame and the next (i+1)-th frame, and f vf The virtual manipulation force identified by the manipulation intent recognition algorithm is denoted by m, which represents the mass of the manipulated virtual model. ΔT i The displacement matrix represents the virtual model, and Z, Y, and X represent the coordinate system in the augmented reality environment.

[0043] The formula for calculating the change in rotation is as follows:

[0044]

[0045] ΔR i =R z (θ iz )R y (θ iy )R x (θ ix (16)

[0046] Where, θ i τ represents the rotation angle of the manipulated virtual model in the current i-th frame. vf The virtual manipulation force identified by the manipulation intent recognition algorithm is Δt, which represents the time difference between the current i-th frame and the next (i+1)-th frame, J, which represents the moment of inertia of the manipulated virtual model, and ΔR. i The rotation matrix θ represents the virtual model. iz θ iy and θ ix These represent the rotation angles at θ. i The components around the x, y, and z axes of the augmented reality environment coordinate system.

[0047] Step Six: Repeat steps one through five. Based on the virtual hand model, grasping intention recognition method, and manipulation intention recognition method, perform three-dimensional gesture manipulation in an augmented reality environment. This efficiently and accurately identifies the user's natural grasping intention towards the three-dimensional virtual model, supports the user to move and rotate virtual objects with natural gestures, improves the robustness of the three-dimensional gesture interaction process, makes the gesture interaction experience more intuitive and natural, thereby enhancing the virtual effect of three-dimensional gesture manipulation in an augmented reality environment and improving the user's immersion.

[0048] The present invention also discloses a gesture interaction system for implementing the aforementioned method for manipulating a virtual model in a three-dimensional gesture environment in an augmented reality environment. The gesture interaction system includes a data acquisition and processing module, a virtual hand module, a collision calculation module, and a gesture intent recognition module.

[0049] The data acquisition and processing module is used to acquire the RGB image and depth image of the current frame, and based on the convolutional neural network hand pose estimation algorithm, obtain the position and pose information of the key hand nodes in the current frame according to the RGB image and depth image.

[0050] The virtual hand module is used to overlay virtual hand models and determine the position and posture of the virtual hand models based on the positions and postures of key nodes, thereby achieving the mapping of real hands in virtual space. Based on the positions and postures of key hand nodes acquired by the data acquisition and processing module, the position and posture of the virtual hand model are updated and maintained in real time within the AR device coordinate system.

[0051] The collision calculation module calculates in real time each frame whether the virtual hand model and other virtual models to be manipulated have come into contact or collided, based on a collision detection algorithm.

[0052] The gesture intent recognition module includes a grasping intent recognition submodule and a manipulation intent recognition submodule. The grasping intent recognition submodule constructs "grasping pairs" conditions based on the physical characteristics of grasping real objects and the augmented reality environment. Based on these "grasping pairs," a grasping intent recognition algorithm is built to identify whether the hands are grasping or releasing the virtual model. If the collision detection module detects contact between the hands and other virtual models, the grasping intent recognition algorithm calculates whether multiple contact points between the hands and the manipulated model can form "grasping pairs," determining whether a grasping situation exists between the hands and the virtual model. A "grasping pair" consists of two contact points. If more than one "grasping pair" exists, the grasped virtual model is determined to be in a grasping state. This eliminates the need for contact calculations based on multiple contact points to determine whether a grasp is complete, making grasping intent judgment more flexible, closer to real 3D gesture manipulation, more adaptable to complex gesture interaction scenarios, and more consistent with the user's intuitive interaction experience. Furthermore, if multiple pairs of "grasping pairs" exist, the contact points forming multiple pairs will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency, and immersiveness of gesture interaction intent recognition.

[0053] The manipulation intent recognition submodule is used to identify the user's manipulation intent towards the manipulated virtual model, including movement and rotation. After the grasping intent recognition submodule identifies a grasping intent, this manipulation intent recognition module is invoked. Based on the manipulation intent recognition algorithm, it identifies the intention to manipulate the virtual model and represents it as virtual driving force and torque. Then, based on the force or torque, it calculates the displacement and rotation changes of the manipulated virtual model, updates the motion state of the manipulated model, and achieves the driving of the manipulated model. Compared to methods that simply use the displacement of both hands to predict the model's displacement, the manipulation intent recognition method based on virtual force is more consistent with the physical motion process and can achieve more precise manipulation.

[0054] Beneficial effects:

[0055] 1. This invention discloses a method and system for manipulating virtual models with 3D gestures in an augmented reality environment. Based on the physical characteristics of grasping real objects and the augmented reality environment, a "grasping pair" condition is constructed. A grasping intent recognition algorithm is built based on this "grasping pair" to identify the user's grasping intent. If the collision detection algorithm detects contact between the hands and other virtual models, the grasping intent recognition algorithm calculates whether a "grasping pair" can be formed between multiple contact points between the hands and the manipulated model, determining whether a grasping situation exists between the hands and the virtual model. A "grasping pair" consists of two contact points. If more than one "grasping pair" exists, the grasped virtual model is determined to be in a grasping state. This eliminates the need for contact calculations based on multiple contact points to determine whether a grasping is complete, making the grasping intent judgment more flexible, closer to real 3D gesture manipulation, more adaptable to complex gesture interaction scenarios, and more consistent with the user's intuitive interaction experience. Furthermore, if multiple pairs of "grasping pairs" exist, the contact points forming multiple gripping pairs will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency, and immersion of gesture interaction intent recognition.

[0056] 2. The present invention discloses a method and system for manipulating virtual models in a three-dimensional gesture environment in augmented reality. It uses a manipulation intention recognition algorithm to identify the virtual force and torque applied by the user to the manipulated model, and then combines rigid body dynamics to drive the movement of the manipulated model, rather than simply predicting the displacement of the manipulated model based on the displacement of the user's hands. This makes the gesture interaction in augmented reality environment more accurate and more in line with the physical motion process, allowing the user to manipulate virtual objects more precisely during the interaction. Attached Figure Description

[0057] Figure 1 A diagram illustrating the grasping action.

[0058] Figure 2 This is a flowchart of a three-dimensional gesture manipulation method for virtual models in an augmented reality environment, as disclosed in this invention.

[0059] Figure 3 This is a system block diagram of a virtual model 3D gesture manipulation system in an augmented reality environment, as disclosed in this invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Example

[0062] This embodiment discloses a gesture interaction method in an augmented reality environment, which can be applied to AR terminal devices equipped with cameras, including mobile phones, tablets, AR glasses, or AR helmets. This invention will use AR glasses as an example to describe the gesture interaction method in detail.

[0063] This embodiment discloses a gesture interaction method in an augmented reality environment, referring to... Figure 2 As shown, the specific implementation steps are as follows:

[0064] Step 1: Acquire images of both hands in the current frame, and identify and track the position and pose data of key nodes of both hands relative to the AR device based on the convolutional neural network hand pose estimation algorithm.

[0065] Hands-free image acquisition utilizes a pair of cameras facing the same side on the AR terminal device. One camera is a color camera for acquiring color images, and the other is a depth camera for acquiring depth images. In this embodiment, an RGB camera and a TOF module configured in the AR glasses are used to acquire color and depth images. Based on the extrinsic parameters between the two cameras, each frame of the depth image is registered to the color image for subsequent processing. The extrinsic parameters refer to the displacement and rotation transformation relationship between the camera coordinate systems of the depth and color cameras, with the optical center as the origin.

[0066] The convolutional neural network (CNN) two-handed pose estimation algorithm is used to identify and track the position and pose information of key hand nodes in real time. The algorithm uses two CNNs: one CNN locates the hand regions on an RGB image, crops the RGB image based on the localization result, normalizes it with the depth values ​​of the corresponding regions, and inputs it into the next CNN for real-time regression calculation and estimation of the position and pose information of the key hand nodes in the camera coordinate system. Finally, based on the pose transformation matrix between the camera coordinate system and the AR device coordinate system, the pose and pose results of the key hand nodes in the camera coordinate system are converted to the AR device coordinate system.

[0067] Step 2: Overlay virtual hand models at the key nodes of both hands identified in the current frame, and determine the position and pose of the virtual hand models based on the position and pose of the key nodes.

[0068] The virtual hand model consists of 19 virtual joint models, each a cylinder, approximating the fingers of a real hand. There are topological relationships between the virtual hand joint models; higher-level virtual joint models contain lower-level ones. When a higher-level virtual joint model moves, it influences the movement of lower-level models. For example, movement of the root joint of a finger will influence the movement of the middle and top joints. The parametric representation of this virtual hand model is shown below:

[0069]

[0070] Where Jointti is the i-th virtual joint model, pi is the position of the virtual joint model, represented by a set of vectors xi, yi and zi in the augmented reality environment coordinate system, ei is the pose of the virtual joint model, represented by a set of vectors wi, ri and li in the augmented reality environment coordinate system, Size is the parameter of the virtual joint model, li represents the length of the cylinder, di represents the diameter of the cylinder, children represents the child joint models driven by this virtual joint model, and Jk represents the k-th virtual joint model.

[0071] Each virtual joint model in the virtual hand model will correspond to a key node in the gesture tracking and recognition in step 1. The position and pose data of each recognized key hand node will be used to update the position and pose of the virtual joint model in the current frame, realizing the mapping of the virtual hand model, as shown below:

[0072]

[0073] r i =R z (w i )R y (r i )R x (l i (5)

[0074] Where pi is the position vector of the i-th virtual joint model, ri is the rotation matrix of the virtual joint model, and the transformation relationship between this rotation matrix and Euler angles is shown in the equation, where Rz(wi) represents the rotation around the z-axis by wi degrees; T is the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model is located. Pi is the position vector of the key node corresponding to the i-th virtual joint model, and ri is the rotation matrix of the key node corresponding to the i-th virtual joint model.

[0075] Step 3: Activate the collision detection algorithm to calculate in real time each frame whether there is contact or collision between the virtual hand model and other manipulable virtual models;

[0076] The collision detection algorithm used in this embodiment is the hierarchical bounding box algorithm. OBB (Oriented Bounding Box) bounding boxes are generated around the virtual joint model and other virtual models in the current AR environment. Based on these bounding boxes, collisions between the virtual joint models and manipulateable virtual models within the virtual hand model are calculated in real time. Since the virtual joint model is a regular convex polyhedron (cylindrical), a cylindrical bounding box with the same geometric parameters as its own is generated. Other virtual models will generate approximately convex polyhedral bounding boxes based on their shapes.

[0077] If a collision occurs between virtual joint models and virtual models, the specific location and penetration direction of the collision will be calculated based on the 3D GJK (Gilbert–Johnson–Keerthi, GJK) algorithm and EPA (Epanding Polytop Algorithm) algorithm, and step 4 will be executed.

[0078] Step 4: If contact between the hands and other virtual models is detected, the grasping intention recognition algorithm calculates whether multiple contact points between the hands and the model can form a "grasping pair". It determines whether there is a grasping situation between the hands and the virtual model. If a "grasping pair" exists, the grasped virtual model is determined to be in a grasping state and the process jumps to step 5.

[0079] The aforementioned "grip pair" consists of two contact points between a qualified virtual hand model and a grasped model, as shown in the attached figure. Figure 1 As shown. The "grip pair" condition is that for two contact points a and b between the virtual hand and the virtual object, the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α. In this case, the two contact points will form a stable grip pair g(a,b). This fixed angle α is the friction angle. Therefore, the judgment condition for grip pair g(a,b) is as follows:

[0080]

[0081] Where, n a and n b Let l be the normal vector for contact points a and b, which is the normal vector of the cylindrical surface of the joint virtual model at the contact point; ab Let α be the line connecting contact points a and b; α is the friction angle. In this embodiment, in order to ensure that the user can grasp a large virtual object in the AR environment, the friction angle is set to a large fixed value α = 75°.

[0082] The grasping intent recognition algorithm is constructed based on the "grasping pair" condition. It iterates through all current contact points and determines whether it can form a "grasping pair" with another contact point. When there is at least one "grasping pair" between the virtual hand and the virtual object, it is determined that the virtual object has been successfully grasped.

[0083] Step 5: If the virtual model is in a grasping state, the virtual force or torque generated by the manipulation of the virtual model is calculated based on the position and posture change trend of the grasping center of the hands on the manipulated virtual model, combined with the manipulation intention recognition algorithm, and the virtual force or torque is used to drive the movement or rotation of the virtual model.

[0084] The grasping center is the center point representing the movement of the entire hand. The entire hand is regarded as a complete rigid body, and the position, posture, and velocity of the grasping center represent the motion parameters of the entire virtual hand.

[0085] The method for obtaining the gripping center is as follows: The position and number of "grip pairs" are determined based on the "grip pair" conditions established in step four. The "grip pair" is considered a unified rigid body, and its position and orientation are represented by the gripping center. If a "grip pair" exists, the gripping center is the center of the line connecting the contact points that make up the "grip pair." The position and orientation of the gripping center are calculated as follows:

[0086]

[0087]

[0088] Where Pc represents the gripping center position, p1 and p2 represent the positions of the contact points that make up the "grip pair", and wc, rc, and lc represent the three Euler angle parameters of the gripping center, respectively. and This represents the unit vector pointing to the x, y, and z axes in the current coordinate system.

[0089] If there are multiple "grip pairs", the length of the line connecting the contact points of the multiple "grip pairs" will be used to determine the "grip pair" with the longest line length as the main gripp pair, and the gripping center will be constructed according to the formulas (8) and (9).

[0090] In this embodiment, specifically, step 5 may include the following:

[0091] Step 5.1: Determine whether the "grip" condition is met. If the condition is met, it is considered that the user has put down the manipulated virtual model, and the subsequent steps will not be executed. The position and pose of the virtual model will not be updated in subsequent frames. If the condition is not met, proceed to step 5.2.

[0092] The calculation of the "grip pair" cancellation condition is as follows:

[0093]

[0094] in, The distance between the two contact points that form the "grip pair" in the current frame (frame i). Let k be the distance between the two contact points that make up the "grip pair" in the (i-1)th frame, and k is a fixed value, set to 3mm in this example. That is, if the two contact points that make up the "grip pair" move further apart between two frames and the distance meets a certain threshold, then the grip is considered canceled.

[0095] Step 5.2: Obtain the current grasping center according to the grasping center acquisition method. Based on the intent recognition algorithm, calculate the virtual force or torque applied to the virtual model by the grasping center by both hands, and continue to execute step 5.3.

[0096] The aforementioned manipulation intent recognition algorithm calculates the virtual force or virtual torque applied by both hands to the virtual model in the current frame based on the pose change trend of the grasping center, and calculates the parameters of the virtual model's movement and rotation based on the virtual force or virtual torque, including the direction and distance of movement and the direction and angle of rotation.

[0097] The aforementioned manipulation intent recognition algorithm is based on a virtual linear and torsional spring-damped model. The calculation formula for this algorithm is shown below:

[0098]

[0099]

[0100] Formula (11) is the formula for calculating virtual force, and formula (12) is the formula for calculating virtual torque. The posture of both hands contacting the center point in the i-th frame (current frame) is represented as follows: In the (i+1)th frame, the gesture of both hands touching the center point is represented as (qi+1l, qi+1o). Let be the three-dimensional position of the hand in the i-th frame. A quaternion describing the orientation of the hand; and Let K be the linear velocity and angular velocity of the virtual component in the i-th frame. sl (K so ) and K Dl (K Do ) represents the coefficients of the linear and torsional spring damping models, which are adjusted through experience and specific test experiments to achieve stable and smooth dynamic motion of the virtual component.

[0101] Step 5.3: Based on the virtual force or torque calculated by the manipulation intent recognition algorithm described in Step 5.2, the displacement and rotation changes of the virtual model are calculated using rigid body dynamics. The position and orientation of the manipulated virtual model are updated in the current frame based on the displacement and rotation changes, and the virtual model is rendered according to the new position and orientation. Step 5.1 is then continued in the next frame.

[0102] The formula for calculating the change in displacement is as follows:

[0103]

[0104] Where Si represents the displacement of the manipulated virtual model in the current i-th frame, vi represents the velocity of the manipulated virtual model in the current i-th frame, Δt represents the time difference between the current i-th frame and the next (i+1)-th frame, and f vf The virtual manipulation force identified by the manipulation intent recognition algorithm is denoted by m, which represents the mass of the manipulated virtual model and is set to a uniform value of 10 in this example. ΔT i The displacement matrix represents the virtual model, and Z, Y, and X represent the coordinate system in the augmented reality environment.

[0105] The formula for calculating the change in rotation is as follows:

[0106]

[0107] ΔR i =R z (θ iz )R y (θ iy )R x (θ ix (16)

[0108] Where, θ i τ represents the rotation angle of the manipulated virtual model in the current i-th frame. vf The virtual manipulation force identified by the manipulation intent recognition algorithm is Δt, which represents the time difference between the current i-th frame and the next (i+1)-th frame, J, which represents the moment of inertia of the manipulated virtual model, and ΔR. i The rotation matrix θ represents the virtual model. iz θ iy and θ ix These represent the rotation angles at θ. i The components around the x, y, and z axes of the augmented reality environment coordinate system.

[0109] Step 6: Repeat steps 1 to 5. Based on the virtual hand model, grasping intention recognition method, and manipulation intention recognition method, perform three-dimensional gesture manipulation in the augmented reality environment. This efficiently and accurately identifies the user's natural grasping intention towards the three-dimensional virtual model, supports the user to move and rotate virtual objects with natural gestures, improves the robustness of the three-dimensional gesture interaction process, makes the gesture interaction experience more intuitive and natural, thereby enhancing the virtual effect of augmented reality on three-dimensional gesture manipulation and improving the user's immersion.

[0110] Further, refer to Figure 3 As shown, this embodiment also provides a system for gesture interaction in an augmented reality environment. The system includes a data acquisition module, a virtual hand module, a collision calculation model, and a gesture interaction intent recognition module.

[0111] The data acquisition and processing module is used to acquire the RGB and depth images of the current frame and obtain the position and pose data of key hand nodes based on the convolutional neural network hand pose estimation algorithm. This module calls the depth and RGB cameras to acquire depth and RGB images, and applies a convolutional neural network to estimate the position and pose information of key hand nodes in the current frame based on the RGB and depth images, then transmits this information to the virtual hand module.

[0112] The virtual hand module updates the position and posture of each virtual joint model that makes up the virtual hand in the AR device coordinate system based on the position and posture of the key hand nodes in the current frame obtained by the data processing module.

[0113] The collision calculation module is used to detect whether there is contact between the virtual hand model and the virtual model to be manipulated in the AR environment. It maintains the OBB bounding box of the virtual hand joint model and the AR model in each frame, and calculates the collision and contact between the virtual hand model and other models in the AR environment. After contact occurs, it calculates the position and direction of the contact according to the GJK algorithm.

[0114] The gesture interaction intent recognition module includes a grasping intent recognition submodule and a manipulation intent recognition submodule. The grasping intent recognition submodule constructs "grasping pairs" conditions based on the physical characteristics of grasping real objects and the augmented reality environment, and builds a grasping intent recognition algorithm based on the "grasping pairs" to identify whether the hands are grasping or releasing the virtual model. If the collision detection module detects that the hands are in contact with other virtual models to be manipulated, the grasping intent recognition algorithm calculates whether a "grasping pair" can be formed between multiple contact points of the hands and the manipulated model to determine whether there is a grasping situation between the hands and the virtual model. A "grasping pair" consists of two contact points. If there is more than one "grasping pair", the grasped virtual model is determined to be in a grasping state. It is not necessary to calculate the contact based on multiple contact points to determine whether the grasping is completed, making the grasping intent judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. At the same time, if there are multiple pairs of "grasping pairs", the contact points of multiple grasping pairs will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency and immersion of gesture interaction intent recognition.

[0115] The manipulation intent recognition submodule is used to identify the user's manipulation intent towards the manipulated virtual model, including movement and rotation. After the grasping intent recognition submodule detects a grasping intent, this manipulation intent recognition module is invoked. Based on the manipulation intent recognition algorithm, it identifies the intention to manipulate the virtual model and represents it as virtual driving force and torque. Then, based on the force or torque, it calculates the displacement and rotation changes of the manipulated virtual model, updates the motion state of the manipulated model, and achieves the driving of the manipulated model. Compared to methods that simply use the displacement of both hands to predict the model's displacement, the manipulation intent recognition method based on virtual force is more consistent with the physical motion process and can achieve more precise manipulation.

[0116] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for manipulating a virtual model in three dimensions using gestures in an augmented reality environment, characterized in that: Includes the following steps, Step 1: Acquire images of both hands in the current frame, and determine the position and pose data of key nodes of both hands relative to the AR device based on the convolutional neural network (CNN) hand pose estimation algorithm. The CNN algorithm is a 2D to 3D hand pose estimation algorithm composed of two CNNs. The first CNN is trained to realize hand localization and estimate the 2D position of the hand center in the image. Then, the localized image of the hand position, together with the corresponding input depth value, is used to generate a normalized cropped image, which is passed to the second CNN to regress the relative 3D hand joint position in real time. Step 2: At the key nodes of both hands identified in the current frame, overlay a virtual hand model, and determine the position and posture of the virtual hand model according to the position and posture of the key nodes to realize the mapping of real hands in virtual space; Step 3: Based on the collision detection algorithm, calculate in real time in each frame whether there is contact or collision between the virtual hand model and other virtual models to be manipulated; Step 4: Construct "grasping pairs" conditions based on the physical characteristics of grasping real objects and the augmented reality environment. Build a grasping intent recognition algorithm based on these "grasping pairs." If the collision detection algorithm described in Step 3 detects contact between the hands and other virtual models, the grasping intent recognition algorithm calculates whether "grasping pairs" can be formed between multiple contact points between the hands and the manipulated model. It then determines whether a grasping situation exists between the hands and the virtual model. A "grasping pair" consists of two contact points. If more than one "grasping pair" exists, the grasped virtual model is determined to be in a grasping state. The physical process of grasping a real object is characterized by applying the basic laws of Newtonian rigid body mechanics. It judges whether the object can be grasped based on whether it is in force balance and the friction between the contact surfaces of the virtual hand model and the manipulated model. Its implementation principle is to analyze the force state of the object using a simplified Coulomb friction model. The "grip pair" consists of two contact points of a virtual hand model that meets certain conditions and the model being gripped; the conditions for the "grip pair" are as follows: the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle. Then the two contact points will form a stable gripping pair g(a,b); the fixed angle That is, the friction angle; The aforementioned grasping intent recognition algorithm is based on the "grasping pair" condition, iteratively determining whether all current contact points can form a "grasping pair" with another contact point; for any two contact points a and b between the virtual hand and the virtual object in one iteration, the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle. Then the two contact points will form a stable gripping pair g(a,b); the fixed angle This is the friction angle, meaning the gripping joint g(a,b) should satisfy... in, and Let be the normal vectors of contact points a and b, which are the normal vectors of the cylindrical surface of the joint virtual model at the contact points; Let be the vector of the line connecting contact points a and b; The friction angle is the angle of friction. The value of the friction angle needs to be set through testing for the specific manipulated model to ensure stable and natural gripping of the virtual component. Step 5: Construct a gripping center acquisition method based on the "grip pair" conditions established in Step 4 to obtain the gripping center, which is the center of the line connecting the contact points that make up the "grip pair". If the virtual model is determined to be in a gripping state based on the gripping intention recognition algorithm in Step 4, calculate the virtual force or torque applied by both hands to the virtual model based on the displacement and posture changes of the gripping center on the manipulated model according to the manipulation intention recognition algorithm, and drive the movement or rotation of the virtual model with the virtual force or torque. The manipulation intention recognition algorithm calculates the virtual force or torque applied by both hands to the virtual model in the current frame based on the pose change trend of the gripping center, and calculates the parameters of the movement and rotation of the virtual model based on the virtual force or torque. The parameters of movement and rotation include the direction and distance of movement and the direction and angle of rotation. By adopting the manipulation intention recognition algorithm and adding the gripping center judgment condition, all contact points participate in the manipulation intention recognition process, making the manipulation intention recognition more flexible and improving the robustness of the manipulation intention recognition. Step Six: Repeat steps one through five to perform three-dimensional hand gesture manipulation in an augmented reality environment based on the virtual hand model, grasping intention recognition method, and manipulation intention recognition method described above.

2. The method for manipulating a virtual model in a three-dimensional gesture environment in an augmented reality environment as described in claim 1, characterized in that: The virtual hand model consists of several virtual joint models, each of which is a cylinder and approximates the finger joints of a real hand; the virtual hand model is represented by the following parameterization: Where Jointti is the i-th virtual joint model. The position vector of the virtual joint model is a set of vectors in the augmented reality environment coordinate system. , and express, The pose of the virtual joint model is determined by a set of Euler angle parameters in the augmented reality environment coordinate system. , and This indicates that Size is a parameter of the virtual joint model. Indicates the length of the cylinder. This represents the diameter of the cylinder, and `children` represents the child joint models that this virtual joint model drives. and These represent the k-th and m-th virtual joint models, respectively. Each virtual joint model in the virtual hand model will correspond to a key node in the gesture tracking and recognition step one. The position and pose data of each recognized key hand node will be used to update the position and pose of the virtual joint model in the current frame. The position and pose of the virtual hand model are determined based on the position and pose of the key nodes, realizing the mapping of real hands in virtual space, as shown in the following formula: in, Let be the position vector of the i-th virtual joint model. Let be the rotation matrix of the i-th virtual joint model. This rotation matrix has a transformation relationship with Euler angles as shown in equation (5), where Indicates rotation around the z-axis Spend, Indicates rotation around the y-axis Spend, Indicates rotation around the x-axis Degree; T is the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model is located; Let be the position vector of the key node corresponding to the i-th virtual joint model. Let be the rotation matrix of the key node corresponding to the i-th virtual joint model.

3. The method for manipulating a virtual model in a three-dimensional gesture environment in an augmented reality environment as described in claim 1, characterized in that: The grasping center is the center point representing the movement of the entire hand. The entire hand is regarded as a complete rigid body, and the position, posture, and velocity of the grasping center represent the motion parameters of the entire virtual hand. The method for determining the gripping center is as follows: Based on the "grip pair" constructed in step four, determine the position and number of the "grip pair"; treat the "grip pair" as a unified rigid body, and use the gripping center to represent the position and orientation of this rigid body; if a "grip pair" exists, the position and orientation of the gripping center are calculated as follows: in, This indicates that the center position should be captured. and This indicates the location of the contact points that make up the "grip pair". , and These represent the three Euler angle parameters of the center of the capture. , and This represents the unit vector pointing to the x, y, and z axes in the current coordinate system; If there are multiple "grip pairs", the length of the line connecting the contact points of the multiple "grip pairs" will be used to determine the "grip pair" with the longest line length as the main gripp pair, and the gripping center will be constructed according to the formulas (8) and (9). Step 5.1: Determine whether the "grip pair" meets the "grip pair" cancellation condition. If the cancellation condition is met, it is considered that the user has put down the manipulated virtual model, and the subsequent steps will not be executed. The position and pose of the virtual model will not be updated in the next frame. If the condition is not met, proceed to step 5.

2. The calculation of the "grip pair" cancellation condition is as follows: in, The distance between the two contact points that make up the "grip pair" in the current t-th frame. The distance between the two contact points that form the "grip pair" in frame t-1. It is a fixed value; that is, when the two contact points that make up the "grip pair" move away from each other between two frames and the degree of distance meets a certain threshold, the grab is considered canceled. Step 5.2: Calculate the virtual force or torque applied by both hands to the virtual model according to the manipulation intention recognition algorithm, and continue to execute step 5.3; After adopting the manipulation intention recognition algorithm and adding the "grabbing center" condition judgment, all contact points participate in the manipulation intention recognition process, making manipulation intention recognition more flexible and improving the robustness of manipulation intention recognition. The aforementioned manipulation intent recognition algorithm is constructed based on a virtual linear and torsional spring-dampers model; the calculation formula for the manipulation intent recognition algorithm is shown below; Formula (11) is the formula for calculating virtual force. This represents the virtual control force. Formula (12) is the formula for calculating the virtual torque. Represents the virtual manipulation torque; where the posture of both hands contacting the center point in the current t-th frame is represented as In frame t+1, the posture of both hands touching the center point is represented as ( , ), Let be the three-dimensional position of the hand in frame t. A quaternion describing the orientation of the hand; and Let be the linear velocity and angular velocity of the manipulated virtual model in frame t; , and , The coefficients are for the linear and torsional spring damping models; adjustments are made for specific manipulated virtual models. , and , The coefficient enables stable and smooth dynamic movement of virtual components, conforming to the user's intuitive interactive experience; Step 5.3: Based on the virtual force or torque calculated by the manipulation intent recognition algorithm described in Step 5.2, combine rigid body dynamics to calculate the displacement and rotation changes of the virtual model; and update the position and orientation of the manipulated virtual model in the current frame based on the displacement and rotation changes, and render the virtual model based on the new position and orientation. The formula for calculating the change in displacement is as follows: in, This indicates that the virtual model is being manipulated to shift in the current t-th frame. This represents the speed at which the virtual model is manipulated in the current t-th frame. This represents the time difference between the current t-th frame and the next t+1-th frame. The virtual manipulation force identified by the manipulation intent recognition algorithm, where m represents the mass of the manipulated virtual model; The displacement matrix represents the virtual model, and Z, Y, and X represent the coordinate system in the augmented reality environment; The formula for calculating the change in rotation is as follows: in, This indicates the rotation angle of the virtual model being manipulated in the current t-th frame. The virtual maneuvering torque identified by the aforementioned maneuvering intent recognition algorithm. This represents the time difference between the current t-th frame and the next t+1-th frame. This represents the moment of inertia of the manipulated virtual model. The rotation matrix represents the virtual model. , and These represent the rotation angles at... The components around the x, y, and z axes of the augmented reality environment coordinate system.

4. A gesture interaction system for implementing a three-dimensional gesture manipulation method for a virtual model in an augmented reality environment as described in claim 1, 2, or 3, characterized in that: It includes a data acquisition and processing module, a virtual hand module, a collision calculation module, and a gesture intention recognition module; The data acquisition and processing module is used to acquire the RGB image and depth image of the current frame, and based on the convolutional neural network hand pose estimation algorithm, obtain the position and pose information of the key hand nodes in the current frame according to the RGB image and depth image. The virtual hand module is used to overlay virtual hand models and determine the position and posture of the virtual hand models based on the key node positions and postures, thereby realizing the mapping of real hands in virtual space; based on the position and posture of the key hand nodes obtained by the data acquisition and processing module, the position and posture of the virtual hand model are updated and maintained in real time in the AR device coordinate system. The collision calculation module calculates in real time each frame whether the virtual hand model and other virtual models to be manipulated are in contact or collide, based on a collision detection algorithm. The gesture intent recognition module includes a grasping intent recognition submodule and a manipulation intent recognition submodule; The grasping intent recognition submodule constructs "grasping pairs" conditions based on the physical characteristics of grasping real objects and the augmented reality environment. Based on these "grasping pairs," a grasping intent recognition algorithm is built to identify whether the hands are grasping or releasing the virtual model. If the collision detection module detects contact between the hands and other virtual models, the grasping intent recognition algorithm calculates whether a "grasping pair" can be formed between multiple contact points of the hands and the manipulated model to determine if a grasping situation exists between the hands and the virtual model. A "grasping pair" consists of two contact points. If more than one "grasping pair" exists, the grasped virtual model is determined to be in a grasping state. This eliminates the need for contact calculations based on multiple contact points to determine whether a grasp is complete, making grasping intent judgment more flexible, closer to real 3D gesture manipulation, more adaptable to complex gesture interaction scenarios, and more consistent with the user's intuitive interaction experience. Furthermore, if multiple pairs of "grasping pairs" exist, the contact points forming multiple "grasping pairs" will participate in the interaction intent recognition, improving the robustness, flexibility, efficiency, and immersiveness of gesture interaction intent recognition. The manipulation intent recognition submodule is used to recognize the user's manipulation intent toward the manipulated virtual model, and the manipulation intent includes movement and rotation; After the grasping intention recognition submodule recognizes the grasping intention, the manipulation intention recognition module is called. Based on the manipulation intention recognition algorithm, the intention to manipulate the virtual model is identified and represented by the results of virtual driving force and torque. Then, based on the force or torque, the displacement change and rotation change of the manipulated virtual model are calculated, the motion state of the manipulated model is updated, and the driving of the manipulated model is realized. Compared to methods that simply use the displacement of both hands to predict the displacement of the model, the manipulation intention recognition method based on virtual force is more in line with the physical motion process and can achieve more precise manipulation.

Images