A virtual hand interacting with a virtual object and a motion generation method thereof

Abstract

The present invention belongs to a generation method, and specifically relates to a virtual hand that interacts with a virtual object and a method for generating its motion. It includes the following steps: Step 1: Environment and posture tracking; Step 2: Determining the posture of the fingers of the left and right hands; Step 3: Rendering of both hands. Compared with the prior art, the present invention has the following beneficial effects: It can avoid the current low hand tracking accuracy and occlusion problems, and generate hand postures that match the moving virtual objects at a low cost for virtual environment hand rendering, thereby improving the user's immersion and fun in the virtual environment. The invention only needs to track the posture of the wrist and the virtual object, which is low-cost and easy to use. The posture of the virtual hand is only related to the shape of the virtual object, and has nothing to do with the shape of the entity. The same entity can control different virtual objects, and the user does not need to use an entity that is consistent with the shape of the virtual object, thereby reducing the cost of entity manufacturing. The method of the present invention can be used in real time without delay.

Description

Technical Field

[0001] The present invention belongs to a generation method, and in particular relates to a virtual hand interacting with a virtual object and a motion generation method thereof. Background Art

[0002] Virtual reality (VR) is commonly used in education, training, healthcare, and other industries, where virtual hands are often needed to control the movement of virtual objects to enhance user immersion, learning efficiency, and interaction efficiency. Currently, finger motion trajectories of virtual hands are typically captured using optical tracking, data glove, or image-based methods to control the finger motion trajectory of the virtual hand. However, optical tracking-based methods (such as OptiTrack) are expensive and require dozens of optical tracking devices to be fixed on the finger surface, requiring a large space and presenting occlusion issues. Data glove-based methods (such as Noitom Hi5) are also expensive and require users to wear gloves, making calibration cumbersome. Image-based finger motion tracking methods (such as LeapMotion) have low accuracy and severe occlusion issues.

[0003] In virtual reality applications, physical devices are usually used to provide force feedback for corresponding virtual objects to improve interaction efficiency. However, when the user's hands interact with the entity, occlusion is easily caused, making image-based tracking methods unusable.

[0004] Specifically, patent document A method for generating natural grasping movements of a virtual hand based on grasping taxonomy, CN110991237 A, discloses a generation method comprising the following steps: 1) segmenting and fitting a three-dimensional object model based on a segmentation and fitting algorithm of a superquadric surface model, and selecting graspable object components for subsequent movement planning; 2) combining grasping taxonomy with grasping movement planning to construct a mapping relationship between grasped objects and standard grasping types; 3) searching for stable grasping movement candidates in the virtual hand pose space based on a simulated annealing heuristic search algorithm combined with a grasping quality metric; and 4) introducing a grasping posture similarity distance to guide the planning process to generate grasping movements that are consistent with natural human grasping postures.

[0005] The drawback of this method is that it can only generate grasping postures but cannot generate continuous movements. In addition, this method has a delay and cannot generate hand postures in real time.

[0006] The public paper ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation discloses the following technical solution: using a neural network, by inputting the hand pose of the previous frame and the spatial representation of the hand-object history frame and future frames, the hand pose of the next frame is predicted, thereby realizing the pose estimation of the virtual hand interacting with the virtual object.

[0007] The technical solution described in this document has the drawback that it needs to utilize data of future frames and has a delay. Summary of the Invention

[0008] The purpose of the present invention is to address the defects of the prior art and provide a virtual hand that interacts with a virtual object and a motion generation method thereof.

[0009] The specific technical solution adopted by the present invention is as follows: a virtual hand interacting with a virtual object and a method for generating its motion, which includes the following steps:

[0010] Step 1: Environment and posture tracking;

[0011] Step 2: Determine the finger positions of the left and right hands;

[0012] Step 3: Rendering with both hands.

[0013] In the above-mentioned method for generating a virtual hand and motion thereof for interacting with a virtual object, the first step includes collecting environmental data and collecting data of the left and right wrists.

[0014] A virtual hand interacting with a virtual object and a motion generation method thereof as described above, wherein the environmental data collection includes the pose data collection of entities in a real environment and the data collection of virtual objects in a virtual environment, and the virtual objects in the virtual environment can be pre-set or specified or created by the user,

[0015] The shape information of the virtual object is represented by a point cloud, including the position of the point cloud and the normal direction of the point cloud along the surface of the object. The point cloud is obtained by uniformly sampling the surface of the object mesh.

[0016] Use tracking devices to obtain the user's wrist position and the position of the entity in the real world; the position of the virtual object is controlled by the position of the corresponding entity in the real environment;

[0017] The pose of the virtual wrist is controlled by the pose of the user's wrist in the real world;

[0018] The poses of the wrist and the entity can be obtained through optical tracking, and the obtained entity poses are used to control the corresponding virtual objects in real time.

[0019] As described above, a virtual hand that interacts with a virtual object and a method for generating its motion, wherein the step one includes coordinate transformation, setting the position of the wrist in the wrist coordinate system to (0,0,0), and transforming the collected data to the wrist coordinate system; the wrist coordinate system is a three-dimensional right-handed Cartesian coordinate system, and the Z-axis direction is perpendicular to the palm of the hand.

[0020] As described above, in the method for generating a virtual hand and motion thereof for interacting with a virtual object, in step 2, the finger postures of the left hand and the right hand are determined respectively.

[0021] In the above-mentioned method for generating a virtual hand and motion thereof for interacting with a virtual object, in step 2, the posture of the fingers of the right hand is determined by the following steps:

[0022] Inputting the motion trajectory and shape of the virtual object in the wrist coordinate system into the first neural network to generate rough coordinate values of the finger joints of the virtual hand that conform to the motion of the virtual object in the current frame;

[0023] The motion trajectory of the virtual object is the posture of the center of the virtual object in the wrist coordinate system of the current frame and the previous three frames; the posture includes position and rotation; the position is 3 degrees of freedom, including the coordinate values of the x, y, and z axes; the rotation is 6 degrees of freedom, and the rotation matrix is calculated based on the 3-degree-of-freedom Euler angle value using the pytorch3d.transforms.euler_angles_to_matrix function of the pytorch3d package in Python, and then the pytorch3d.transforms.matrix_to_rotation_6d function is used to convert the rotation matrix into a 6-degree-of-freedom rotation. The 3-degree-of-freedom Euler angle is rotated in the order of the coordinate axes YZX.

[0024] The virtual object shape information is the value of the point cloud and point cloud normal of the virtual object in the current frame and the previous three frames in the wrist coordinate system of the corresponding frame.

[0025] The first neural network includes an encoder, a residual network, and a decoder;

[0026] The encoder comprises four fully connected layers;

[0027] The residual network includes four residual blocks, each of which includes an activation layer ELU and a fully connected layer;

[0028] The decoder comprises a fully connected layer;

[0029] The first neural network is used to regress and predict the hand joint position from the wrist posture and object information.

[0030] The rough pose of the virtual hand's fingers in the current frame in the wrist coordinate system, the pose of the virtual object in the current frame, the shape of the virtual object in the current frame, and the optimized pose of the virtual hand's fingers in the previous frame are input into the second neural network to obtain the optimized fine pose of the virtual hand's fingers in the wrist coordinate system of the current frame.

[0031] The second neural network consists of three parts: encoder, residual network and decoder.

[0032] The second neural network encoder includes six fully connected layers; the residual network includes four residual blocks, each residual block includes an activation layer ELU and a fully connected layer;

[0033] The decoder comprises a fully connected layer;

[0034] The hand joint coordinate values output by the second neural network are converted from the wrist coordinate system to the world coordinate system.

[0035] As described above, a virtual hand that interacts with a virtual object and a method for generating its motion, wherein, in the step 2, the posture of the left hand is determined by the following steps: first, the left hand is mirrored to become the right hand, secondly, the posture of the mirrored left hand is determined using the method for determining the posture of the right hand, and finally, the result is mirrored again to obtain the posture of the left hand.

[0036] Compared with the prior art, the present invention has the following beneficial effects: it can avoid the current low hand tracking accuracy and occlusion problems, and can generate hand postures that match the moving virtual objects at a low cost for virtual environment hand rendering, thereby improving the user's immersion and fun in the virtual environment. The invention only needs to track the posture of the wrist and the virtual object, which is low-cost and easy to use. The posture of the virtual hand is only related to the shape of the virtual object and has nothing to do with the shape of the entity. The same entity can control different virtual objects. The user does not need to use an entity that is consistent with the shape of the virtual object, which reduces the cost of entity manufacturing. The method of the present invention can be used in real time without delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the wrist coordinate system

[0038] Figure 2 This is a diagram of the wrist and finger joints

[0039] Figure 3 This is the first neural network structure diagram

[0040] Figure 4 is the hand joint position prediction result of the first neural network

[0041] Figure 5 This is the result of virtual reality hand rendering

[0042] Figure 6 This is the second neural network structure diagram

[0043] Figure 7 is the second neural network optimization result

[0044] Figure 8 Virtual hand generation flow chart

[0045] Figure 9 This is the flow chart for generating the right virtual hand

[0046] Figure 10 This is the flow chart for generating the left virtual hand

[0047] Figure 11 User diagram

[0048] Figure 12 This is a schematic diagram of the predicted hand joint pose

[0049] Figure 13 This is a screenshot of the virtual hand rendered in the virtual environment.

[0050] Figure 14 Is the device logic diagram DETAILED DESCRIPTION

[0051] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0052] In this invention, we use a tracker to track the user's real wrist and the physical posture motion trajectory of the real environment, and input the real wrist posture and physical posture motion trajectory as well as the shape information of the virtual object into the neural network to generate a precise finger motion trajectory for controlling the corresponding virtual object.

[0053] 1) Obtain virtual objects and their shape information;

[0054] The virtual objects in the virtual environment may be pre-set or specified or created by the user.

[0055] The shape information of the virtual object is represented by a point cloud, including the position of the point cloud and the normal direction of the point cloud along the surface of the object, wherein the point cloud is obtained by uniformly sampling the surface of the object mesh.

[0056] 2) Use a tracking device to obtain the user's wrist pose and the pose of the entity in the real world;

[0057] The pose of the virtual object is controlled by the pose of the corresponding entity in the real environment;

[0058] The pose of the virtual wrist is controlled by the pose of the user's wrist in the real world;

[0059] The position of the wrist and the entity can be obtained by optical tracking, and the obtained entity position is used to control the corresponding virtual object in real time, wherein the shape of the virtual object and the entity can be consistent or inconsistent. Figure 5 As shown, the position of the entity is tracked in the real environment and is used to control the position of the virtual water cup in the virtual environment.

[0060] 3) Convert the virtual wrist position, virtual object position and virtual object shape information from the world coordinate system to the wrist coordinate system.

[0061] The origin of the wrist coordinate system is the position of the wrist, and the position of the wrist in the wrist coordinate system is (0,0,0). That is, at different times, the position of the wrist in the spatial coordinate system changes, but the value in the wrist coordinate system is always (0,0,0).

[0062] The wrist coordinate system is a three-dimensional right-handed Cartesian coordinate system, such as Figure 1 As shown in , the z axis is along the palm direction. The wrist position is as Figure 2 shown.

[0063] 4) Inputting the virtual object motion trajectory and virtual object shape in the wrist coordinate system into the first neural network to generate rough coordinate values of the finger joints of the virtual hand that conform to the motion of the virtual object in the current frame;

[0064] The motion trajectory of the virtual object is the posture of the center of the virtual object in the wrist coordinate system of the current frame and the previous three frames; the posture includes position and rotation; the position is 3 degrees of freedom, including the coordinate values of the x, y, and z axes; the rotation is 6 degrees of freedom, and the rotation matrix is calculated according to the 3-degree-of-freedom Euler angle value using the pytorch3d.transforms.euler_angles_to_matrix function of the pytorch3d package in Python, and then the pytorch3d.transforms.matrix_to_rotation_6d function is used to convert the rotation matrix into a 6-degree-of-freedom rotation, and the 3-degree-of-freedom Euler angle is rotated in the order of the coordinate axes YZX.

[0065] The virtual object shape information is the value of the point cloud and point cloud normal of the virtual object in the current frame and the previous three frames in the wrist coordinate system of the corresponding frame.

[0066] The first neural network includes an encoder, a residual network, and a decoder, such as Figure 3 shown.

[0067] The encoder comprises four fully connected layers;

[0068] The residual network includes four residual blocks, each of which includes an activation layer ELU and a fully connected layer;

[0069] The decoder comprises a fully connected layer;

[0070] For example, when the object point cloud samples 1000 points, the positions of the 1000 points in the wrist coordinate system of the t frame are expressed as The normal corresponding to the point in frame t is expressed as The position of the object center in frame t is expressed as The rotation of the object center in frame t is expressed as

[0071] The input of the network consists of four parts: the spatial position of the object point cloud in frames t, t-1, t-2, and t-3: Normals of the object point cloud at time t, t-1, t-2, and t-3: The position of the center of the object at time t, t-1, t-2, and t-3: The rotation of the object center at time t, t-1, t-2, and t-3:

[0072] The output of the network is the coordinate value of the hand joints other than the wrist in the wrist coordinate system of frame t

[0073] The four parts of input are encoded by the corresponding fully connected layers in the encoder, and then input into the residual network. Finally, the decoder is used to regress and predict the coordinates of the 21 joints except the wrist in frame t. The initial positions of the 21 joints are as follows: Figure 2 shown.

[0074] In the first neural network training phase, we use the predicted values and the true value The mean absolute error of supervised training.

[0075] Figure 4 It shows that through the first neural network, the current hand joint position can be predicted by regression from the wrist posture and object information. Figure 5 A virtual hand controlled by the joint positions of the virtual hand obtained by the first neural network is shown in a virtual environment.

[0076] 5) The rough pose of the virtual hand fingers in the current frame in the wrist coordinate system, the pose of the virtual object in the current frame, the shape of the virtual object in the current frame, and the optimized pose of the virtual hand fingers in the previous frame are input into the second neural network to obtain the optimized fine pose of the virtual hand fingers in the wrist coordinate system of the current frame.

[0077] The second neural network includes three parts: an encoder, a residual network, and a decoder. Figure 6 A diagram showing a second neural network.

[0078] The encoder comprises six fully connected layers;

[0079] The residual network includes four residual blocks, each of which includes an activation layer ELU and a fully connected layer;

[0080] The decoder comprises a fully connected layer;

[0081] For example, when the number of sampling points is 1000, the network input contains 6 parts: the spatial coordinates of the object point cloud in the t frame: Normal of the object point cloud in frame t: The pose of the object center in frame t: The distance between the virtual hand and the virtual object generated by the first neural network in frame t: Rough hand joint coordinates in frame t: The optimized hand joint coordinates of the second neural network output at frame t-1 are: Output is the optimized hand joint coordinate value of t frame

[0082] The six parts of the input in the second neural network are encoded by the encoder and then input into the residual network. Finally, the decoder is used to regress and predict the coordinates of the 21 optimized joints of the hand except the wrist at time t. In the training phase of the second neural network, we use the predicted values and the true value The mean absolute error of supervised training.

[0083] Among them, the optimized hand joint coordinate values output by the second neural network at the start time t-1 frame are For example Figure 2 Neutral posture shown.

[0084] Figure 7 The optimization results of the second neural network are shown. Before the second neural network, the pinky finger was already immersed in the virtual object. After the optimization, the pinky finger and the virtual object have good contact.

[0085] 6) Convert the hand joint coordinate values output by the second neural network from the wrist coordinate system to the world coordinate system.

[0086] 7) For the left-hand prediction, all inputs are mirrored to the right-hand representation according to the mirroring method and then calculated.

[0087] The mirroring method is as follows: for the coordinate value, the Z coordinate value is inverted; for the rotation value, the original rotation matrix is R_old, and the mirrored matrix is R_new, R_new=M*R_old*M, M=[[[1,0,0,0],[0,1,0,0],[0,0,-1,0]]].

[0088] After obtaining the output of the second neural network, the same mirroring method is used to obtain the coordinate values of the finger joints of the left hand.

[0089] The generation flow chart of virtual hands for interacting with virtual objects controlled by entities in virtual environments is shown in the following figure: Figure 8 As shown. First, the tracker is used to track the postures of the left and right hands and the entity. Virtual objects can be loaded from local storage, cloud storage, or created by users, etc. Then, the virtual object information and the right wrist posture and left wrist posture are processed and input into the first neural network respectively to obtain the estimated rough joint positions of both hands, which are then input into the second neural network together with the virtual object information to obtain the optimized hand joint positions. The joint position of the left hand needs to be mirrored. Finally, the hand joint positions of both hands control the rendering of the virtual hands in the virtual environment to obtain hand postures that match the moving virtual objects.

[0090] Figure 9 The specific process of generating the right virtual hand is demonstrated. First, the tracker obtains the right wrist, entity posture, and virtual object information. The virtual object posture is controlled by the entity posture. Then, the virtual object information is converted from the world coordinate system to the wrist coordinate system, and the point cloud coordinates, normals and other information mentioned above are extracted. The information of the previous three frames and the current moment are input into the first neural network to obtain the predicted joint posture of the virtual hand. Then, the joint posture, object information, and the output of the second neural network of the previous frame are input into the second neural network to obtain the optimized hand joint posture at the current moment. At this time, the posture is in the wrist coordinate system, and then it is converted to the world coordinate system to control the virtual hand in the virtual environment.

[0091] The hand joint pose is related to the shape of the virtual object, but not to the shape of the physical object. Therefore, when the same entity controls different virtual objects, the joint poses of the generated virtual hand will be different when the motion pose remains the same.

[0092] Figure 10The specific process of generating a virtual hand for the left hand is demonstrated. The process is basically the same as that for the right hand, but before inputting into the first neural network, the wrist information and object information need to be mirrored to obtain the information of the right hand. After obtaining the output of the second neural network, it needs to be mirrored again to obtain the joint information of the left hand.

[0093] Figure 11 Shows an example of user usage. The user interacts with the entity in a real environment.

[0094] Figure 12 Shows the joint positions of the virtual hand predicted by the neural network.

[0095] Figure 13 Shows the rendered virtual hand that the user sees in the virtual environment.

[0096] The present invention provides a device for generating a virtual hand for interacting with a virtual object controlled by a physical object in virtual reality. The device includes a tracking unit, a computing unit and a display unit, such as Figure 14 The tracking unit is used to track the position of the user's wrist and interactive entities in the real environment, and can use optical tracking and other methods; the computing unit is used to calculate the user's neural network and other virtual scenes, including a storage unit for storing virtual object information and operation instructions; the display unit is used to display the virtual environment. The three units can be located on the same device, such as an integrated head-mounted display, or on separate devices.

Claims

1. A virtual hand interacting with a virtual object and a motion generation method thereof, characterized in that: The steps include: Step 1: Environment and posture tracking; Step 2: Determine the finger positions of the left and right hands; Step 3: Render with both hands; The step 1 includes collecting environmental data and collecting data from left and right wrists; Environmental data collection includes the collection of physical posture data of real environments and the collection of data of virtual objects in virtual environments. The virtual objects in the virtual environments can be pre-set or specified or created by the user. The shape information of the virtual object is represented by a point cloud, including the position of the point cloud and the normal direction of the point cloud along the surface of the object. The point cloud is obtained by uniformly sampling the surface of the object mesh. Use tracking devices to obtain the user's wrist position and the position of the entity in the real world; the position of the virtual object is controlled by the position of the corresponding entity in the real environment; The pose of the virtual wrist is controlled by the pose of the user's wrist in the real world; The position of the wrist and the entity can be obtained through optical tracking, and the obtained entity position is used to control the corresponding virtual object in real time; The step 1 includes coordinate transformation, setting the position of the wrist in the wrist coordinate system to (0,0,0), and transforming the collected data into the wrist coordinate system; the wrist coordinate system is a three-dimensional right-handed Cartesian coordinate system, with the Z axis perpendicular to the palm; In the step 2, the finger positions of the left hand and the right hand are determined respectively; In step 2, the finger positions of the right hand are determined by the following steps: Inputting the motion trajectory and shape of the virtual object in the wrist coordinate system into the first neural network to generate rough coordinate values of the finger joints of the virtual hand that conform to the motion of the virtual object in the current frame; The motion trajectory of the virtual object is the position of the center of the virtual object in the wrist coordinate system of the current frame and the previous three frames; the position includes position and rotation; the position is 3 degrees of freedom, including the coordinate values of the x, y, and z axes; the rotation is 6 degrees of freedom, and the rotation matrix is calculated using the pytorch3d.transforms.euler_angles_to_matrix function of the pytorch3d package in Python according to the 3-degree-of-freedom Euler angle value, and then the pytorch3d.transforms.matrix_to_rotation_6d function is used to convert the rotation matrix into a 6-degree-of-freedom rotation. The 3-degree-of-freedom Euler angle is rotated in the order of the coordinate axes YZX. The virtual object shape information is the value of the point cloud and point cloud normal of the virtual object in the current frame and the previous three frames in the wrist coordinate system of the corresponding frame; The first neural network includes an encoder, a residual network, and a decoder; The encoder comprises four fully connected layers; The residual network includes four residual blocks, each of which includes an activation layer ELU and a fully connected layer; The decoder comprises a fully connected layer; The first neural network regresses and predicts the hand joint position based on the wrist posture and object information; Inputting the rough poses of the fingers of the virtual hand in the current frame in the wrist coordinate system, the pose of the virtual object in the current frame, the shape of the virtual object in the current frame, and the optimized poses of the fingers of the virtual hand in the previous frame into the second neural network to obtain the optimized fine poses of the fingers of the virtual hand in the wrist coordinate system of the current frame; The second neural network includes three parts: encoder, residual network and decoder; The second neural network encoder includes six fully connected layers; the residual network includes four residual blocks, each residual block includes an activation layer ELU and a fully connected layer; The decoder comprises a fully connected layer; The hand joint coordinate values output by the second neural network are converted from the wrist coordinate system to the world coordinate system.

2. A virtual hand for interacting with a virtual object and a method for generating motion thereof according to claim 1, characterized in that: In step 2, the posture of the left hand is determined by the following steps: first, the left hand is mirrored to form the right hand, secondly, the posture of the mirrored left hand is determined using the posture determination method of the right hand, and finally, the result is mirrored again to obtain the posture of the left hand.

Images