A hand action interaction rendering method and device, a storage medium and an electronic device

Abstract

The application provides a hand action interaction rendering method and device, a storage medium and electronic equipment, and applies to the technical field of visual rendering.The target hand model provided by the application is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchical structure, the geometric size of the second hand model is greater than that of the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interaction collision detection.Collision between the hand simplified model and objects in a virtual environment is detected to determine the mode in which the target hand model is located.In a free space mode, a skeleton pose is calculated by using motion capture data and is synchronized to hand detail model rendering.In a physical interaction mode, a more accurate pose is obtained according to vertex displacement of the hand simplified model and is synchronized to hand detail model rendering, so that penetration of the hand model into virtual objects is effectively prevented, and the authenticity and reliability of hand actions are improved.

Description

Technical Field

[0001] This invention relates to the field of visual rendering technology, and in particular to a method, apparatus, storage medium, and electronic device for interactive rendering of hand gestures. Background Technology

[0002] With the development of virtual reality and human-computer interaction, hand motion capture technology has been widely applied. Existing technologies mainly include inferring skeletal posture through sparse sensors and neural networks, extracting key hand points from video frames to achieve motion recognition, and using event cameras to improve the response speed of high-speed motion recognition.

[0003] However, existing technologies still suffer from insufficient smoothness, poor motion continuity, and noticeable jumpiness when performing high-speed and complex movements. In particular, "clipping" is a common phenomenon when hands interact with virtual objects, affecting visual realism and immersive experience.

[0004] Therefore, how to effectively eliminate clipping in physical hand interactions and improve the realism and reliability of hand movements has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a hand gesture interaction rendering method, apparatus, storage medium, and electronic device that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0006] A hand gesture interaction rendering method, comprising:

[0007] A target hand model is obtained, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand bone hierarchy structure, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a simplified hand model for physical interaction collision detection.

[0008] Detect whether the second hand model collides with an object in the virtual environment, and determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result;

[0009] When the target hand model is in the free space mode, the hand motion data captured by motion capture is used to calculate the first hand bone pose data, and the first hand bone pose data is synchronized to the first hand model for hand rendering.

[0010] When the target hand model is in the physical interaction mode, the second hand bone pose data is determined based on the vertex displacement of the second hand model, and the second hand bone pose data is synchronized to the first hand model for hand rendering.

[0011] A hand motion interaction rendering device includes: a target hand model acquisition unit, a drive discrimination unit, a free space drive unit, and a physical interaction drive unit.

[0012] The target hand model obtaining unit is used to obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand bone hierarchy structure, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a simplified hand model for physical interaction collision detection.

[0013] The driving discrimination unit is used to detect whether the second hand model collides with an object in the virtual environment, and to determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result.

[0014] The free space driving unit is used to calculate the first hand skeleton posture data using the hand motion data captured by motion capture when the target hand model is in the free space mode, and to synchronize the first hand skeleton posture data to the first hand model for hand rendering.

[0015] The physical interaction driving unit is used to determine the second hand skeleton posture data based on the vertex displacement of the second hand model when the target hand model is in the physical interaction mode, and to synchronize the second hand skeleton posture data to the first hand model for hand rendering.

[0016] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the hand gesture interactive rendering method.

[0017] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the hand motion interaction rendering method.

[0018] By means of the above technical solution, the present invention provides a hand motion interaction rendering method, device, storage medium, and electronic device, which obtains a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchy structure, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a detailed hand model used for hand rendering, and the second hand model is a simplified hand model used for physical interaction collision detection; it detects whether the second hand model collides with objects in the virtual environment, and determines whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection results; when the target hand model is in free space mode, it calculates the first hand skeleton posture data using the hand motion data captured by motion capture, and synchronizes the first hand skeleton posture data to the first hand model for hand rendering; when the target hand model is in physical interaction mode, it determines the second hand skeleton posture data based on the vertex displacement of the second hand model, and synchronizes the second hand skeleton posture data to the first hand model for hand rendering. This invention binds a high-precision visual detail model of the hand and a simplified hand model with a larger geometric size to the same hand skeleton level. The hand detail model is used for visual rendering, while the simplified hand model is used for collision detection and to distinguish between free space mode and physical interaction mode. In physical interaction mode, the bone posture is reversed based on the vertex displacement of the simplified hand model, which can effectively prevent clipping and thus improve the realism and reliability of hand movements.

[0019] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0021] Figure 1 The diagram illustrates a flowchart of one embodiment of the hand motion interaction rendering method provided in this invention.

[0022] Figure 2 The diagram shows a specific implementation of step S120 in the hand motion interaction rendering method provided in this embodiment of the invention.

[0023] Figure 3The diagram shows a specific implementation of step S130 in the hand motion interaction rendering method provided in this embodiment of the invention.

[0024] Figure 4 A schematic diagram of the hand motion interaction rendering control provided in an embodiment of the present invention is shown;

[0025] Figure 5 A schematic diagram of the structure of the hand motion interaction rendering device provided in an embodiment of the present invention is shown;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0027] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0028] With the rapid development of Virtual Reality (VR), Augmented Reality (AR), and human-computer interaction, hand motion capture technology has become a crucial foundation for achieving natural interactive experiences. Currently, various hand motion capture technologies exist on the market, primarily categorized as those based on Inertial Measurement Units (IMUs), computer vision and deep learning, and novel sensors such as event cameras. While each technology has its advantages in practical applications, current mainstream technologies generally suffer from the following shortcomings when the hand physically interacts with virtual objects: limited motion smoothness, especially prone to stuttering during high-speed and complex operations; weak continuity of motion sequences, often resulting in jumps or abrupt changes that affect immersion; in virtual scenes, when the hand contacts a virtual object, "clipping" (i.e., the hand model penetrating the virtual object) often occurs, severely compromising physical realism and visual consistency; and in complex interactive scenarios, the system often struggles to simultaneously guarantee high precision, low latency, and physically consistent dynamic interaction effects.

[0029] Based on this, the present invention provides a hand motion interaction rendering method, employing a "high-low" dual-model collaborative architecture. A high-precision, detailed model handles visual rendering to enhance the realism of the image, while a low-polygon simplified model is hidden within the physics engine for efficient collision detection and deformation simulation. Based on a state-adaptive hybrid driving mechanism, when physical interaction is detected, real-time collision and deformation data from the low-poly model are used to calculate the skeletal posture that satisfies human physiological constraints through inverse dynamics, effectively preventing the hand from penetrating virtual objects. In free movement, the positions of IK (Inverse Kinematics) control points are collected and combined with finger joint motion constraints to optimize the naturalness and smoothness of finger movements. Therefore, the present invention can improve the realism and immersion of hand-to-virtual object interaction while ensuring smooth movement and consistency of physical rules, thus significantly improving the realism and reliability of the hand motion interaction experience.

[0030] like Figure 1 The diagram shows a flowchart of one embodiment of the hand motion interaction rendering method provided by this invention. The method may include:

[0031] S100. Obtain the target hand model, wherein the target hand model is obtained by binding the first hand model and the second hand model to the same hand bone hierarchy structure. The geometric size of the second hand model is larger than that of the first hand model. The first hand model is a hand detail model used for hand rendering, and the second hand model is a simplified hand model used for physical interaction collision detection.

[0032] The target hand model refers to a 3D (Three-Dimensional) hand model used for hand movement simulation and interaction in a virtual environment. It consists of a first hand model and a second hand model, both bound to the same skeletal hierarchy. The target hand model is rendered using the first hand model, and the visibility of the second hand model is disabled.

[0033] The first hand model, also known as the high-polygon hand model, refers to a 3D hand model with a relatively large number of vertices and rich details. It is mainly used for visual rendering to show fine features such as skin and wrinkles, and to enhance the realism of the image.

[0034] The second hand model, also known as the low-face hand model, refers to a 3D hand model with a relatively small number of vertices and a simplified geometric structure. Its geometric size is slightly larger than that of the first hand model. It is used to hide behind the visual layer and is dedicated to the physics engine for collision detection and deformation simulation.

[0035] The hand skeletal hierarchy refers to the skeletal system used to drive the hand model, including the wrist joint, metacarpal joints, and finger joints, which defines the motion hierarchy and rotational constraints of the hand.

[0036] Specifically, in this embodiment of the invention, a standard hand skeletal hierarchy structure including the wrist joint, metacarpal joints, and finger joints can be pre-constructed, ensuring that the joint positions are in their default state and aligned with the hand model. Next, a first hand model with rich details for final visual rendering and a second hand model with simplified geometry and fewer vertices are provided. The overall size of the second hand model is slightly larger than the first hand model to reduce the risk of clipping. The first and second hand models are respectively bound to the pre-constructed same hand skeletal hierarchy structure. They share the same bone nodes but have independent skinning weight data. During runtime, only the first hand model is rendered, while the second hand model is hidden and used for collision detection and deformation simulation in the physics engine, thereby completing the construction of the target hand model.

[0037] The default state of joint positions provided in this embodiment of the invention refers to the initial posture maintained by each joint in the hand skeletal hierarchy when it is not driven by motion capture data and is not affected by any external force or animation. For example, the finger joints are in a naturally straight state, the rotation angle of each joint is zero, and the overall hand is in a relaxed and straight baseline posture. Maintaining the default state of each joint position ensures that the hand skeletal hierarchy is precisely aligned in space with the bound first and second hand models, such as the center point of the model-driven part, providing a unified initial reference coordinate system for deformation calculation and inverse solving in subsequent physical interactions.

[0038] S110. Detect whether the second hand model collides with an object in the virtual environment, and determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection results.

[0039] The virtual environment refers to a computer-generated 3D scene, which may include interactive objects, spaces, and virtual objects, and the hand model realizes actions and interactions in this environment.

[0040] The free space mode refers to the state when the hand model does not collide with virtual objects, and the natural hand movement is driven by motion capture data.

[0041] Among them, the physical interaction mode refers to the state when the second hand model collides with the virtual object, and the skeletal posture is reverse-corrected based on the physical data generated by the collision of the second hand model.

[0042] Specifically, embodiments of the present invention can utilize a rendering engine or a physics engine to perform real-time collision detection between the second hand model and other objects in the virtual environment. The vertex distances between the second hand model and all relevant virtual objects in the virtual scene are calculated. When some vertex positions overlap or the distance is below a set threshold, a collision is determined to have occurred. Based on the collision detection results, if the second hand model does not overlap with any virtual object's vertex, the target hand model is in free space mode; conversely, if contact is detected at any vertex, the target hand model switches to physical interaction mode, providing a basis for subsequent action-driven and physical simulation switching.

[0043] S120. When the target hand model is in free space mode, use the hand motion data captured by motion capture to calculate the first hand bone pose data, and synchronize the first hand bone pose data to the first hand model for hand rendering.

[0044] Hand motion data refers to hand movement information acquired through motion capture devices (such as optical cameras and data gloves), including data such as joint position and posture.

[0045] The first hand skeleton pose data refers to the hand skeleton joint pose calculated from motion capture data in free space mode, which is used to render the first hand model.

[0046] Hand rendering refers to applying the calculated skeletal pose data to the first hand model to achieve visual representation and animation effects of the hand.

[0047] Specifically, in embodiments of the present invention, when the target hand model is in free space mode, inverse dynamics calculations can be performed using motion capture data of the hand to calculate the first hand skeleton pose data, and the first hand skeleton pose data can be synchronized to the first hand model for hand rendering.

[0048] Inverse dynamics calculation refers to an algorithm that, based on a given target point (such as the fingertip position), reverses the angles and postures of each skeletal joint to make hand movements natural and in line with physiological constraints.

[0049] Furthermore, in this embodiment of the invention, when the target hand model is in free space mode, the position data of the target joints in the world coordinate system can be obtained, transformed into a local coordinate system centered on the palm, and then physiological motion range constraints can be applied to correct their positions. Based on the corrected joint positions, the finger motion plane is dynamically determined, and the middle finger joint located between the fingertip joint and the finger root joint is projected onto this plane to determine its target position. Based on the corrected joint positions and the target position of the middle finger joint, the first hand skeleton posture data is solved through inverse kinematics.

[0050] S130. When the target hand model is in physical interaction mode, determine the second hand bone pose data based on the vertex displacement of the second hand model, and synchronize the second hand bone pose data to the first hand model for hand rendering.

[0051] Vertex displacement refers to the change in the position of the surface vertices of the second hand model due to collision during physical interaction. By analyzing these changes, the skeletal movement can be inferred.

[0052] The second hand skeleton pose data refers to the skeleton joint pose calculated based on the physical deformation of the vertices of the second hand model in the physical interaction mode. This pose is used to render the first hand model and achieve physically consistent hand animation.

[0053] Specifically, when the target hand model is in physical interaction mode, the bone pose data of the second hand is obtained by inversely solving the bone rotation data of each joint based on the vertex displacement of the second hand model, and the bone pose data of the second hand is synchronized to the first hand model for hand rendering.

[0054] Among them, skeletal rotation data refers to the rotation axis and rotation angle of each joint, which describes the current posture of the skeletal system.

[0055] Furthermore, in embodiments of the present invention, when the target hand model is in physical interaction mode, the dominant joints that have the greatest impact on its deformation are determined based on the displacement vectors and skin weights of the target vertices that collide in the second hand model. Then, each dominant joint is solved in reverse order from fingertip to finger root to obtain the bone rotation data of each joint. These bone pose data are summarized as the second hand bone pose data and synchronized to the first hand model to complete the hand rendering.

[0056] This invention provides a hand motion interaction rendering method, which includes: obtaining a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand skeletal hierarchy, the second hand model having a larger geometric size than the first hand model, the first hand model being a detailed hand model for hand rendering, and the second hand model being a simplified hand model for physical interaction collision detection; detecting whether the second hand model collides with an object in the virtual environment, and determining whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result; when the target hand model is in free space mode, calculating the first hand skeletal pose data using motion-captured hand motion data, and synchronizing the first hand skeletal pose data to the first hand model for hand rendering; when the target hand model is in physical interaction mode, determining the second hand skeletal pose data based on the vertex displacement of the second hand model, and synchronizing the second hand skeletal pose data to the first hand model for hand rendering. This invention binds a high-precision visual detail model of the hand and a simplified hand model with a larger geometric size to the same hand skeleton level. The hand detail model is used for visual rendering, while the simplified hand model is used for collision detection and to distinguish between free space mode and physical interaction mode. In physical interaction mode, the bone posture is reversed based on the vertex displacement of the simplified hand model, which can effectively prevent clipping and thus improve the realism and reliability of hand movements.

[0057] Optional, based on Figure 1 The method shown is as follows: Figure 2 The diagram shows a specific implementation of step S120 in the hand motion interaction rendering method provided in this embodiment of the invention. Step S120 may specifically include:

[0058] S200: Obtain the position data of the first joint of the target joint in the hand motion data captured by motion capture.

[0059] The first joint position data refers to the raw three-dimensional coordinate data of the target joint of the hand in the world coordinate system, which is directly acquired by motion capture equipment (such as an optical camera or data gloves). The target joint can include the wrist joint, fingertip joint, middle finger joint, and finger root joint.

[0060] Specifically, embodiments of the present invention can receive raw spatial data in real time from markers in the optical motion capture system or sensors built into the data glove. By processing this raw data, the three-dimensional coordinates of the target joints, including the wrist joint, fingertip joints, middle finger joints, and root finger joints, in the world coordinate system are directly obtained, i.e., the position data of the first joint point.

[0061] S210. Transform the position data of the first joint point from the world coordinate system to the local coordinate system centered on the palm to obtain the position data of the second joint point.

[0062] The second joint position data refers to the three-dimensional coordinate data after the first joint position data is transformed from the world coordinate system to a local coordinate system with the palm coordinate system as the origin. It is used to unify the reference system through coordinate transformation, eliminate the influence of the overall movement and rotation of the palm on the relative movement of the fingers, and simplify subsequent inverse kinematics calculations.

[0063] Specifically, in this embodiment of the invention, a local coordinate system can be constructed with the wrist joint position as the origin and the spatial orientation of the palm. Then, the world coordinates of all joint points (including fingertips, fingers, etc.) are transformed into this local coordinate system with the palm as the reference through a coordinate transformation matrix (including rotation and translation). After the transformation, the position of the finger joints no longer depends on the absolute placement of the hand in the virtual world, but only represents their relative position with respect to the palm, greatly simplifying subsequent calculations for the movement of the fingers themselves.

[0064] S220. Perform physiological motion range verification on the second joint position data, constrain the position of the target joint within a spherical motion domain centered on the root joint and with the maximum length of the phalanx as the radius, and obtain the corrected third joint position data.

[0065] The third joint position data refers to the corrected coordinate data obtained after verifying the physiological range of motion of the second joint position data.

[0066] In this embodiment of the invention, a spherical motion domain can be constructed with the finger root joint as the center and the maximum length of the finger bone when fully extended as the radius. If the joint point exceeds the domain, it is projected onto the boundary of the sphere to ensure that the finger movement conforms to the physiological limits of the human body.

[0067] Specifically, to prevent non-physiological overextension or bending of the fingers, this embodiment of the invention restricts the range of motion of the fingertip target point. A spherical motion domain is constructed with the base of the finger joint as the center and the maximum length of the finger bone when fully extended as the radius. Next, the position data of the second joint point of the fingertip joint is detected to be within this sphere. If the point is within or on the surface of the sphere, it is directly used as the position data of the third joint point; if it exceeds the sphere's range, the point is pulled back onto the sphere's surface along the line connecting it to the center of the sphere (base of the finger joint). This corrected position is the position data of the third joint point.

[0068] S230: Based on the position data of the third joint point and combined with the preset polar vector of the back of the hand, dynamically calculate the plane of finger movement.

[0069] The finger movement plane is dynamically determined based on the line connecting the root and tip joints of the finger, combined with a preset polar vector in the direction of the back of the hand. The plane normal vector of the finger movement plane can be obtained by normalizing the cross product of the vector from the tip joint to the root joint and the polar vector, ensuring that the plane simultaneously includes the direction of finger movement and the orientation of the back of the hand, which is used to constrain the natural movement trajectory of the finger when bending.

[0070] Specifically, in this embodiment of the invention, to ensure that the fingers naturally lie within a reasonable plane when bent (avoiding twisting), a dedicated motion plane is dynamically calculated for each finger. Using the third joint point position data of the fingertip and finger root joints, which have been physiologically validated, a fingertip-finger root vector is obtained. This vector is then cross-producted with a pre-defined polar vector pointing towards the back of the hand in the local coordinate system of the palm, and the result is normalized to obtain a unit normal vector perpendicular to the target plane. The plane defined by this normal vector encompasses both the finger's pointing direction and the back of the hand, thus determining the natural motion trajectory plane of the bent finger.

[0071] S240. Project the position of the middle joint, which is located between the fingertip joint and the finger root joint, onto the finger movement plane to obtain the position data of the fourth joint point of the middle joint on the finger movement plane.

[0072] The middle joint refers to the intermediate joint located between the root joint and the tip joint of the finger. It is a key transition node in the inverse kinematics solution of the finger. Its position can be determined by projecting it onto the finger motion plane, thus avoiding joint twisting or anti-physiological movement.

[0073] Among them, the fourth joint position data refers to the corrected coordinate data after the original position of the middle finger joint is vertically projected onto the finger movement plane. The middle finger joint can be constrained within the dynamically generated finger movement plane through the vector projection formula, ensuring that all joints are coplanar when the finger is bent as a whole, presenting a natural deformation.

[0074] Specifically, in this embodiment of the invention, after determining the finger movement plane, the middle joint is constrained to that plane to determine its correct bending position. The position vector of the middle joint relative to the root joint is projected onto the finger movement plane calculated in the previous step. This projection process uses mathematical calculations to find a point, which is the vertical projection point of the middle joint on the target plane. This vertical projection point is the fourth joint point position data, used to ensure that when the finger bends towards the fingertip joint at the target point, the middle joint naturally falls within the correct movement plane.

[0075] S250. Based on the position data of the third joint and the fourth joint, the posture data of the first hand skeleton is calculated by inverse kinematics.

[0076] Specifically, in this embodiment of the invention, the fixed position data of the root and tip joints in the third joint position data, and the projected position data of each middle joint in the fourth joint position data, can be used to calculate the angle of rotation (i.e., skeletal pose data) required by the root and middle joints in order for the end of the skeletal chain (fingertip) to reach the specified position through an inverse kinematics solver. The first hand skeletal pose data containing a set of joint rotation values ​​is output for the first hand model to perform hand rendering, so that it presents a pose that is both consistent with motion capture input and natural and without distortion.

[0077] To facilitate understanding of the hand model driving principle in free space mode provided in this embodiment of the invention, an example is given here: When the second hand model does not collide with any virtual object, the target hand model enters free space mode. In this mode, hand movements are entirely driven by motion capture data to ensure high-fidelity reproduction of the user's real hand movements. By receiving raw input data from an optical motion capture camera or a data glove sensor, the position of the palm (wrist joint) and the positions of key joints of each finger (finger root, middle finger, fingertip) in the world coordinate system are obtained. Subsequently, all joint point data are uniformly transformed into a local coordinate system with the palm coordinates as the origin. This step eliminates the influence of overall palm movement and rotation, simplifying the problem to the processing of relative finger movements, thus facilitating subsequent calculations.

[0078] To overcome the unnatural finger twisting that may result from direct actuation, this invention introduces a motion plane-based optimization algorithm. This algorithm dynamically calculates a unique motion plane for each finger to ensure its bending trajectory conforms to physiological principles. This plane is formed by the finger root joint (…). ) pointing to the fingertip joint ( The vector of ) and a pre-defined polar vector representing the direction of the back of the hand ( Together, they determine whether to ensure that the defined plane simultaneously includes the direction of the finger. — (Connect the lines) and the direction of the back of the hand ( Its plane normal vector. The calculation formula is:

[0079] ;

[0080] After determining the plane of motion, the middle finger joint ( The constraint is applied to this plane. Through projection calculations, we obtain... Correction position on the target plane This ensures that the three joints ( ) when the finger is bent , , The joints always lie in the same natural plane, thus avoiding joint twisting. The calculation formula is:

[0081] ;

[0082] Simultaneously, strict physiological constraints are imposed on the range of motion of the fingertip joints (IK target points) to prevent displacement exceeding physiological limits. This constraint is achieved through a mechanism centered on the finger root joint (…). With the center of the ball as the maximum straight length of the finger bone ( This is achieved through a spherical motion domain with a radius of . For the target point The constraint rules are defined by the following mathematical formula:

[0083] ;

[0084] Where C represents the center of the spherical motion domain, i.e., in the formula... The position, C can also be represented as This formula indicates that if the target point is within the sphere, it is applied directly; if it is outside the sphere, it is pulled back onto the sphere along the direction from the center of the sphere to that point, and the corrected point is denoted as... .

[0085] This invention transforms the original motion capture data into ergonomically accurate skeletal posture data through coordinate system transformation, physiological range of motion verification, dynamic finger motion plane calculation, and joint projection. This provides high-quality input for inverse kinematics solving, thereby ensuring that the rendering result of the first hand model in free space mode can simultaneously achieve high-fidelity motion reproduction and natural and smooth visual performance.

[0086] Optional, based on Figure 1 The method shown is as follows: Figure 3 The diagram shows a specific implementation of step S130 in the hand motion interaction rendering method provided in this embodiment of the invention. Step S130 may specifically include:

[0087] S300: Obtain the target displacement vectors of each target vertex that collided in the second hand model.

[0088] In this context, target vertices refer to the specific vertices in the second hand model that collide with objects in the virtual environment. The physics engine detects that the positions of these vertices change due to the collisions, and these changed positions are the key data points for model deformation, serving as the input basis for subsequent skeletal pose adjustments.

[0089] The target displacement vector refers to the displacement difference vector of each collided target vertex from its position in the previous frame to its current position.

[0090] Specifically, in this embodiment of the invention, the collision between the second hand model and objects in the virtual environment can be detected in real time by a physics engine, capturing the set of all vertices that come into contact. For each colliding vertex, its current position after physical simulation (target position) is obtained, and combined with the position of the vertex determined by the bone pose of the previous frame (previous frame position), the target displacement vector of the vertex is calculated by vector difference.

[0091] S310. Using the skin weight data of the target displacement vector of the target vertex, map out the dominant joint that has the greatest impact on the deformation of the target vertex.

[0092] In this context, the dominant joint refers to the skeletal joint that has the greatest impact on a particular vertex in the skinning weight allocation. Each vertex is usually affected by the weights of multiple joints, but the dominant joint is the main driving force for the displacement of that vertex. By identifying the dominant joint, the vertex deformation problem can be decomposed into the rotational adjustment of individual joints.

[0093] Specifically, embodiments of the present invention can utilize the skinning weight information of the second hand model to map each target vertex that has collided and whose displacement vector has been calculated to the skeletal joint with the highest weight, i.e., the dominant joint. For example, embodiments of the present invention can search for the joint weight data corresponding to the target vertex, determine the joint with the highest weight value as the dominant joint, and include the vertex in the controlled vertex set of that joint. Through this mapping, the complex vertex deformation problem is decomposed into multiple single-joint driven problems, facilitating subsequent independent pose adjustment of each dominant joint.

[0094] S320. Following the order from fingertip joints to finger root joints, reverse skeletal posture processing is performed on each dominant joint in sequence to obtain the second hand skeletal posture data.

[0095] Among them, inverse skeletal pose processing is a method that reverse-engineers the rotation and pose changes of skeletal joints by analyzing the deformation of model vertices (such as vertex displacement caused by collisions).

[0096] According to the embodiments of the present invention, the joint rotation axis and rotation angle can be calculated based on the target displacement of the collision vertex and its geometric relationship with the dominant joint, so that the vertex deformation effect required by the physics engine can be reproduced after the bone posture is adjusted, ensuring that the hand model deforms naturally and without clipping.

[0097] Specifically, embodiments of the present invention can process each dominant joint in reverse order from the fingertips (end joints) to the wrist (root joints) according to the hierarchical structure of the hand skeleton. For each dominant joint, based on the target displacement vector and lever arm vector of the set of vertices it controls, the optimal rotation axis and rotation angle of the joint are solved by minimizing the difference between the predicted motion direction and the actual vertex displacement direction. Subsequently, the calculated rotation matrix is ​​applied to update the current bone pose of the joint, and the vertex positions controlled by the joint are updated synchronously. This process recursively covers all affected joints, and finally obtains second hand skeleton pose data that conforms to physical collision constraints, which is used to drive the natural rendering of the first hand model.

[0098] In this embodiment of the invention, when a collision is detected between the second hand model and an object in the virtual environment, the target displacement vector of the collision vertex is obtained and accurately mapped to the corresponding dominant joint using skin weights. This effectively decomposes the complex vertex deformation problem into a single joint rotation adjustment problem. Subsequently, the bone rotation data is calculated in reverse order from fingertip to finger root, ensuring that the rotation of each joint accurately reflects the deformation caused by the collision. This obtains the second hand bone posture data under the physical interaction mode. This not only achieves a high degree of matching between the hand bone posture and the physical collision deformation, improving the naturalness and realism of the hand movements, but also effectively avoids model clipping, enhancing the interactive experience and rendering refinement of the virtual hand.

[0099] Optionally, in the above Figure 3 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S320 may specifically include:

[0100] For the dominant joint currently being processed: Based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding lever arm vectors, the optimal rotation axis of the dominant joint is obtained; Using the average lever arm vector, average displacement vector, and optimal rotation axis of the dominant joint, the joint rotation angle of the dominant joint is obtained; The posture of the dominant joint in the second hand model is adjusted according to the joint rotation angle to obtain the target posture data of the dominant joint; The target posture data of each dominant joint are summarized to obtain the posture data of the second hand skeleton.

[0101] Optionally, in this embodiment of the invention, after adjusting the posture of the dominant joint in the second hand model according to the joint rotation angle and obtaining the target posture data of the dominant joint, the positions of all vertices in the second hand model mapped to the dominant joint are updated based on the target posture data. Based on the updated vertex positions of the second hand model, the next dominant joint to be processed is determined in the order from the fingertip joint to the finger root joint. The process then returns to the step of obtaining the optimal rotation axis of the dominant joint based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding lever arm vectors.

[0102] Specifically, in this embodiment of the invention, for the currently processed dominant joint, all vertices mapped to the joint are first collected. These vertices have target positions after a collision. By calculating the displacement vectors of these vertices and their lever arm vectors relative to the joint center, an optimization algorithm is used to find a unit vector as the rotation axis, minimizing the difference between the predicted motion direction generated by this rotation axis and the actual motion direction of the vertex. This determines the optimal direction of joint rotation, i.e., the optimal rotation axis of the dominant joint. Next, the average lever arm and average displacement of all mapped vertices are calculated. By projecting the average displacement vector onto a plane perpendicular to the rotation axis, the effective tangential displacement is obtained. Based on the relationship between the lever arm length and the magnitude of the tangential displacement, combined with an adjustment parameter, the joint rotation angle of the dominant joint is derived to ensure that the joint rotation can reproduce the actual displacement changes of the vertices to the greatest extent. According to the determined optimal rotation axis and joint rotation angle, the current posture of the joint is updated, and the joint is rotated around the axis by the corresponding angle to obtain the target posture data of the joint. This posture represents the reasonable new position of the joint under physical collision constraints. Using the new joint pose, the positions of all vertices within the joint's control range are recalculated, updating the vertex coordinates of the second hand model to ensure consistency between vertex positions and joint poses, providing the latest vertex data for subsequent joint calculations. Each dominant joint is processed sequentially from fingertip to finger root. After processing each joint, the updated vertex positions are used to calculate the rotation axis and angle of the next joint, repeating this process until the pose adjustments for all affected joints are complete. The updated pose data for all dominant joints are merged to form complete skeletal pose information, which is output as the second hand skeletal pose data. This second hand skeletal pose data can be directly applied to the first hand model, achieving a realistic hand dynamic rendering effect with high physical collision accuracy and no clipping.

[0103] Optionally, embodiments of the present invention may use the average lever arm vector and the optimal rotation axis of the dominant joint to obtain the predicted motion direction unit vector of the dominant joint; use the average displacement vector and the optimal rotation axis of the dominant joint to obtain the effective tangential displacement of the dominant joint; and use the predicted motion direction unit vector, the effective tangential displacement, and the average lever arm vector to obtain the joint rotation angle of the dominant joint.

[0104] Specifically, in this embodiment of the invention, after obtaining the optimal rotation axis of the dominant joint, the average lever arm vector mapped to all vertices of the joint is first calculated, i.e., the average position vector of these vertices relative to the joint center. Then, a direction vector is obtained through the cross product operation of the rotation axis and the average lever arm vector. This direction represents the predicted motion direction of the vertices when the joint rotates. Next, this direction vector is normalized to obtain a normalized predicted motion direction unit vector. The average displacement vector, reflecting the actual movement trend of the vertices as a whole, is projected onto a plane perpendicular to the optimal rotation axis, and the component parallel to the rotation axis direction is removed, so that the remaining part is the tangential displacement that the joint rotation can effectively drive. This effective tangential displacement measures the actual vertex deformation component that can be produced by the joint rotating around the rotation axis. Based on the mechanical lever principle, the magnitude of the effective tangential displacement is compared with the average lever arm length. The joint rotation angle is directly proportional to the tangential displacement and inversely proportional to the lever arm length. Combining the directional relationship between the predicted motion direction unit vector and the tangential displacement (the rotation direction is determined by the sign), a reasonable rotation angle value is calculated, i.e., the joint rotation angle of the dominant joint. The joint rotation angle allows the motion generated by the controlled vertex group after joint rotation to be as close as possible to the displacement required by the physics engine, ensuring natural and accurate deformation.

[0105] This invention utilizes the average lever arm vector of the dominant joint and the optimal rotation axis to first obtain the predicted motion direction unit vector guided by joint rotation. This direction accurately reflects the driving direction of joint rotation on vertex motion. The average displacement vector of the dominant joint is projected onto a plane perpendicular to the optimal rotation axis to obtain effective tangential displacement. Invalid components parallel to the rotation axis are eliminated, ensuring that only the actual vertex movement that can be generated by joint rotation is considered. Combining the predicted motion direction unit vector, the effective tangential displacement, and the magnitude and direction relationship of the average lever arm vector, a reasonable joint rotation angle is calculated. This angle enables the positional change of the vertex group after joint rotation to match the deformation caused by physical collision to the greatest extent, thereby effectively improving the accuracy and naturalness of joint posture recovery, avoiding invalid rotation and deformation distortion, and significantly improving the realism and stability of the hand model in physical interaction mode.

[0106] To facilitate understanding of the hand model driving principle in free space mode provided in this embodiment of the invention, an example is given here: When the physics engine detects a collision between the second hand model and an object in the virtual environment, it automatically switches to physics interaction mode. At this time, the second hand model will come into contact with objects in the scene (including the hand itself), causing the vertices in the contact area to shift position. Based on the principle of motion skinning weights, the physical displacement of these local vertices is used to inversely solve the posture adjustment of the corresponding bones. This deformation is mainly caused by the rotation of local joints. After calculating the rotation data of the joints, this data is synchronized to the first hand model to achieve a dynamic, clipping-free hand rendering effect.

[0107] To simplify the calculations, based on the assumption that "hand vertices are mostly influenced by a single dominant joint," the overall problem is decomposed into the independent processing of each dominant joint. First, the target displacement vector of the colliding vertices in the second hand model is calculated. The set of target positions of the vertices after the collision is obtained through the physics engine. The old position of the corresponding vertex in the previous frame (Determined by the bone pose of the previous frame), calculate the displacement of each vertex:

[0108] ;

[0109] Where i represents the current vertex. The displacement of the current vertex. The target position of the current vertex. This represents the position of the current vertex in the previous frame.

[0110] Using skin weights, each collision vertex is mapped to the dominant joint J that has the greatest influence on it. Let C be the set of vertices dominated by this joint. i .

[0111] Next, for each dominant joint J, its rotation axis is calculated. This serves as the basis for the direction of joint rotation. For set C... i Each vertex v in i Calculate the lever arm vector from the joint center J to the vertex. According to physical laws, the predicted direction of motion caused by joint rotation should be consistent with the actual displacement direction of the vertex, that is, the predicted direction of motion is... The actual direction of motion is The optimal rotation axis is solved by minimizing the difference between the two. :

[0112] ;

[0113] in, This indicates the search for a value that minimizes the objective function. This indicates that vector a is a unit vector and is only used to calculate the direction of the vector.

[0114] Obtain the optimal rotation axis Next, the rotation angle Δθ of the joint is calculated to make the displacement of the controlled vertex group after joint rotation as close as possible to the simulated average displacement. First, the average lever arm vector r of the vertex set is calculated. avg and average displacement vector and will Projecting onto a plane perpendicular to the axis of rotation yields the effective tangential displacement. :

[0115] ;

[0116] The predicted unit vector of motion direction is:

[0117] ;

[0118] Based on the lever principle, the joint rotation angle is directly proportional to the magnitude of the tangential displacement and inversely proportional to the average lever arm length. The calculation formula is as follows:

[0119] ;

[0120] in, This is an adjustable scaling factor, typically close to 1, used to compensate for errors caused by model simplification; It is a sign function used to ensure the correct rotation direction. If the dot product is positive, sign is +1; otherwise, it is -1.

[0121] Subsequently, joint J is rotated around the parent space axis. Rotation angle Update the rotation matrix:

[0122] ;

[0123] in, For the updated rotation matrix, The rotation matrix before the update. For the axis Rotation The rotation matrix corresponding to radians.

[0124] Based on the new skeletal pose, the positions of all vertices affected by this joint in the second hand model are updated in real time, providing the latest data for the calculation of subsequent joints. Following the hierarchical structure of the hand skeleton, the above steps are repeated sequentially from the fingertip (end effector) to the wrist (root joint) to ensure that the positions of child joints are accurately updated before the parent joint is calculated, avoiding the accumulation of pose errors.

[0125] After processing all affected joints, second hand skeleton pose data matching the physical interaction is obtained. It is then synchronized to the first hand model to achieve realistic, clipping-free dynamic rendering of the hand.

[0126] This invention performs reverse skeletal pose processing on each dominant joint sequentially, from the fingertip joint to the base joint. By accurately solving for the optimal rotation axis and rotation angle of each joint based on the displacement vector and lever arm vector mapped to the vertices of the dominant joints, it can effectively reflect the influence of local vertex deformation on the skeletal pose. After each joint rotation adjustment, the position of the corresponding vertex of the second hand model is updated in a timely manner to ensure that subsequent joint calculations are based on the latest geometric state, avoiding accumulated errors and clipping issues. Finally, by summarizing the target pose data of all dominant joints, second hand skeletal pose data that highly matches the physical collision is obtained, making the restoration of hand skeletal pose more accurate and natural, significantly improving the realism and visual effect of the interaction between the virtual hand and environmental objects, and ensuring the stability and clipping-free performance of the first hand model rendering.

[0127] Optionally, in the above Figure 1 In addition to one or more corresponding embodiments, another optional embodiment provided by the present invention may further include:

[0128] When a mode switch is detected in the target hand model, the hand skeleton pose data calculated in free space mode and physical interaction mode are interpolated and mixed to enable the first hand model to be rendered smoothly during the mode transition period.

[0129] Specifically, to ensure visual continuity between free space mode and physical interaction mode, this embodiment of the invention interpolates the skeletal rotation data during the transition between the two states, avoiding abrupt jumps. Specifically, spherical linear interpolation (Slerp) is used to fuse the two sets of skeletal poses:

[0130] ;

[0131] in, This is the hand skeleton pose data after interpolation and mixing; The first hand skeleton pose data calculated in free space mode; The second hand skeleton posture data calculated in the physical interaction mode; It is a hybrid weight function that increases from 0 to 1 during the transition period. The transition time is usually set within 0.1 to 0.3 seconds to achieve a smooth and natural visual experience.

[0132] In this embodiment of the invention, when the target hand model is detected to be switching between free space mode and physical interaction mode, the hand skeleton posture data calculated separately in the two modes are interpolated and mixed to achieve a smooth transition of skeleton posture, avoiding visual jumps or unnatural phenomena caused by sudden changes in posture, thereby ensuring the rendering continuity and natural performance of the first hand model during the mode switching process.

[0133] To facilitate understanding of the overall control process of hand motion interaction rendering provided in the embodiments of the present invention, this section combines... Figure 4 Explanation: Figure 4 This is a flowchart illustrating the hand motion interaction rendering control provided in this embodiment of the invention. During system initialization: a complete hand skeleton hierarchy is created, and a high-precision hand model (high-poly) and a simplified hand model (low-poly) are loaded. The high-poly and low-poly models are bound to the same skeleton system for unified driving. The low-poly model is set as a physical collider, assigned physical properties (mass, friction, etc.), and loaded into the physics engine, but it does not participate in the final rendering. The initial system state can be set to "free space mode." During the real-time running loop (frame-by-frame processing) stage: raw data from the motion capture device is received and converted into initial pose data for the hand skeleton. The physics engine monitors the collision between the low-poly model and virtual environment objects in real time (based on collision intensity or depth), automatically determining whether the current state is "free space mode" or "physical interaction mode." During the mode splitting processing stage: for free space mode, the range of motion of the fingertips and key control points is limited according to ergonomics to avoid unreasonable movements. Based on the captured data, the natural hand skeleton pose is calculated, and the finger bending plane is dynamically adjusted to prevent distortion and deformation. The calculation results are simultaneously assigned to the high-poly and low-poly models for rendering and physics calculations. For the physics-based interaction mode, the physics engine calculates the target position changes of low-poly vertices based on the laws of mechanics such as collision and friction. Inverse dynamics solution: Vertex mapping: Mapping deformed vertices to their corresponding dominant joints. Joint-by-joint iteration: Processing each joint sequentially from fingertip to finger root: Dynamically determining the rotation axis, analyzing the joint rotation direction based on the current vertex movement. Estimating the rotation angle, combining vertex displacement and lever arm distance to determine the joint rotation amount. Updating the bone pose, adjusting the low-poly vertex position in real time to provide the latest data for the next joint calculation. After all joints are processed, the final bone pose is simultaneously assigned to both the high-poly and low-poly models. During the mode transition processing phase, when a mode switch is detected (such as switching back from physics-based interaction to free space), a brief smooth transition mechanism is activated. By interpolating and blending the bone poses of the two modes, a smooth pose transition is achieved in a very short time (a fraction of a second), avoiding abrupt changes in movement and ensuring the continuity and natural smoothness of the rendered visuals.

[0134] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0135] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0136] Corresponding to the above method embodiments, this invention also provides a hand motion interaction rendering device, the structure of which is as follows: Figure 5 As shown, it may include: a target hand model acquisition unit 10, a driving discrimination unit 20, a free space driving unit 30, and a physical interaction driving unit 40.

[0137] The target hand model acquisition unit 10 is used to obtain the target hand model, wherein the target hand model is obtained by binding the first hand model and the second hand model to the same hand bone hierarchy structure. The geometric size of the second hand model is larger than that of the first hand model. The first hand model is a hand detail model used for hand rendering, and the second hand model is a simplified hand model used for physical interaction collision detection.

[0138] The driving discrimination unit 20 is used to detect whether the second hand model collides with an object in the virtual environment, and to determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result.

[0139] The free space driving unit 30 is used to calculate the first hand skeleton pose data using the hand motion data captured by motion capture when the target hand model is in free space mode, and to synchronize the first hand skeleton pose data to the first hand model for hand rendering.

[0140] The physical interaction driving unit 40 is used to determine the second hand bone posture data based on the vertex displacement of the second hand model when the target hand model is in physical interaction mode, and to synchronize the second hand bone posture data to the first hand model for hand rendering.

[0141] Optionally, the free space drive unit 30 may include: a first joint position data acquisition subunit, a second joint position data acquisition subunit, a third joint position data acquisition subunit, a finger motion plane calculation subunit, a fourth joint position data acquisition subunit, and an inverse kinematics calculation subunit.

[0142] The first joint position data acquisition subunit is used to acquire the first joint position data of the target joint in the hand motion data captured by motion capture.

[0143] The second joint position data acquisition sub-unit is used to transform the first joint position data from the world coordinate system to a local coordinate system centered on the palm, and obtain the second joint position data.

[0144] The third joint position data acquisition subunit is used to verify the physiological range of motion of the second joint position data, constrain the position of the target joint within a spherical motion domain centered on the root joint and with the maximum length of the phalanx as the radius, and obtain the corrected third joint position data.

[0145] The finger motion plane calculation subunit is used to verify the physiological motion range of the second joint position data, constrain the position of the target joint within a spherical motion domain centered on the finger root joint and with the maximum length of the finger bone as the radius, and obtain the corrected third joint position data.

[0146] The fourth joint position data acquisition subunit is used to project the position of the middle joint, which is located between the fingertip joint and the finger root joint, onto the finger motion plane to obtain the fourth joint position data of the middle joint on the finger motion plane.

[0147] The inverse kinematics calculation subunit is used to project the position of the middle joint, which is located between the fingertip joint and the finger root joint, onto the finger motion plane to obtain the position data of the fourth joint point of the middle joint on the finger motion plane.

[0148] Optionally, the physical interaction driving unit 40 may include: a target displacement vector acquisition subunit, a dominant joint mapping subunit, and an inverse skeleton pose processing subunit.

[0149] The target displacement vector acquisition sub-unit is used to obtain the target displacement vectors of each target vertex that collides in the second hand model.

[0150] The dominant joint mapping subunit is used to map the dominant joint that has the greatest impact on the deformation of the target vertex using the skin weight data of the target displacement vector of the target vertex.

[0151] The reverse skeletal pose processing subunit is used to perform reverse skeletal pose processing on each dominant joint in the order from the fingertip joint to the finger root joint to obtain the second hand skeletal pose data.

[0152] Optionally, the inverse skeletal pose processing subunit can be used for the currently processed dominant joint: based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding lever arm vectors, to obtain the optimal rotation axis of the dominant joint; using the average lever arm vector, average displacement vector, and optimal rotation axis of the dominant joint, to obtain the joint rotation angle of the dominant joint; adjusting the pose of the dominant joint in the second hand model according to the joint rotation angle to obtain the target pose data of the dominant joint; and summarizing the target pose data of each dominant joint to obtain the second hand skeletal pose data.

[0153] Optionally, the inverse skeletal pose processing subunit can be used to obtain the predicted motion direction unit vector of the dominant joint using the average lever arm vector and the optimal rotation axis; to obtain the effective tangential displacement of the dominant joint using the average displacement vector and the optimal rotation axis; and to obtain the joint rotation angle of the dominant joint using the predicted motion direction unit vector, the effective tangential displacement, and the average lever arm vector.

[0154] Optionally, the inverse skeletal pose processing subunit can also be used to adjust the pose of the dominant joint in the second hand model according to the joint rotation angle, and after obtaining the target pose data of the dominant joint, update the position of all vertices in the second hand model mapped to the dominant joint based on the target pose data, so as to determine the next dominant joint to be processed in the order from the fingertip joint to the finger root joint based on the updated vertex position of the second hand model, so as to obtain the target pose data of the next dominant joint.

[0155] Optionally, the hand gesture interaction rendering device may also include a smoothing rendering unit.

[0156] The smooth rendering unit is used to interpolate and mix the hand skeleton pose data calculated in free space mode and physical interaction mode respectively when the target hand model is detected to switch modes, so that the first hand model can be smoothly rendered during the mode transition period.

[0157] This invention provides a hand motion interaction rendering device, which is used to: obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand skeleton hierarchy, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a detailed hand model for hand rendering, and the second hand model is a simplified hand model for physical interaction collision detection; detect whether the second hand model collides with an object in the virtual environment, and determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result; when the target hand model is in free space mode, calculate the first hand skeleton pose data using motion capture hand motion data, and synchronize the first hand skeleton pose data to the first hand model for hand rendering; when the target hand model is in physical interaction mode, determine the second hand skeleton pose data based on the vertex displacement of the second hand model, and synchronize the second hand skeleton pose data to the first hand model for hand rendering. This invention binds a high-precision visual detail model of the hand and a simplified hand model with a larger geometric size to the same hand skeleton level. The hand detail model is used for visual rendering, while the simplified hand model is used for collision detection and to distinguish between free space mode and physical interaction mode. In physical interaction mode, the bone posture is reversed based on the vertex displacement of the simplified hand model, which can effectively prevent clipping and thus improve the realism and reliability of hand movements.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0159] The hand motion interaction rendering device includes a processor and a memory. The target hand model acquisition unit 10, the drive discrimination unit 20, the free space drive unit 30, and the physical interaction drive unit 40 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0160] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured; by adjusting kernel parameters, collision detection between the simplified hand model and objects in the virtual environment is performed, determining the target hand model's current mode. In free-space mode, motion capture data is used to calculate the skeletal pose, which is then synchronized to the rendering of the detailed hand model. In physical interaction mode, the simplified hand model's vertex displacement is used to obtain a more precise pose, which is then synchronized to the rendering of the detailed hand model. This effectively prevents the hand model from penetrating virtual objects, improving the realism and reliability of hand movements.

[0161] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the hand gesture interactive rendering method.

[0162] This invention provides a processor for running a program, wherein the program executes the hand motion interaction rendering method during runtime.

[0163] like Figure 6 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned hand gesture interaction rendering method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0164] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program that initializes a hand gesture interactive rendering method step.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0167] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0170] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0171] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0172] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A hand gesture interaction rendering method, characterized in that, include: A target hand model is obtained, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand bone hierarchy structure, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a simplified hand model for physical interaction collision detection. Detect whether the second hand model collides with an object in the virtual environment, and determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result; When the target hand model is in the free space mode, the hand motion data captured by motion capture is used to calculate the first hand bone pose data, and the first hand bone pose data is synchronized to the first hand model for hand rendering. When the target hand model is in the physical interaction mode, obtain the target displacement vector of each target vertex that collides in the second hand model; Using the skin weight data of the target displacement vector of the target vertex, the dominant joint that has the greatest impact on the deformation of the target vertex is mapped; Following the order from the fingertip joint to the finger root joint, reverse skeletal pose processing is performed on each of the dominant joints in sequence to obtain the second hand skeletal pose data, and the second hand skeletal pose data is synchronized to the first hand model for hand rendering.

2. The method according to claim 1, characterized in that, The calculation of the first hand skeleton pose data using motion capture hand motion data includes: Obtain the position data of the first joint of the target joint in the hand motion data captured by motion capture; The first joint position data is transformed from the world coordinate system to a local coordinate system centered on the palm to obtain the second joint position data. The physiological range of motion of the second joint point position data is verified, and the position of the target joint is constrained within a spherical motion domain centered on the root joint and with the maximum length of the finger bone as the radius, to obtain the corrected third joint point position data. Based on the position data of the third joint point, and combined with the preset polar vector of the back of the hand, the finger motion plane is dynamically calculated. Project the position of the middle joint between the fingertip joint and the base joint onto the finger movement plane to obtain the fourth joint point position data of the middle joint on the finger movement plane; Based on the position data of the third joint and the position data of the fourth joint, the posture data of the first hand skeleton is calculated by inverse kinematics.

3. The method according to claim 1, characterized in that, The process involves sequentially performing reverse skeletal posture processing on each dominant joint from the fingertip joint to the finger root joint to obtain second hand skeletal posture data, including: For the dominant joint currently being processed: the optimal rotation axis of the dominant joint is obtained based on the displacement vectors mapped to all vertices of the dominant joint and their corresponding lever arm vectors; The joint rotation angle of the dominant joint is obtained by using the average lever arm vector, average displacement vector and the optimal rotation axis of the dominant joint. The attitude of the dominant joint in the second hand model is adjusted according to the joint rotation angle to obtain the target attitude data of the dominant joint. The target posture data of each of the dominant joints are summarized to obtain the second hand skeleton posture data.

4. The method according to claim 3, characterized in that, The step of obtaining the joint rotation angle of the dominant joint using the average lever arm vector, average displacement vector, and optimal rotation axis includes: Using the average lever arm vector of the dominant joint and the optimal rotation axis, the predicted motion direction unit vector of the dominant joint is obtained; The effective tangential displacement of the dominant joint is obtained by using the average displacement vector of the dominant joint and the optimal rotation axis. The joint rotation angle of the dominant joint is obtained by using the predicted motion direction unit vector, the effective tangential displacement, and the average lever arm vector.

5. The method according to claim 3, characterized in that, After adjusting the posture of the dominant joint in the second hand model according to the joint rotation angle to obtain the target posture data of the dominant joint, the method further includes: Update the positions of all vertices in the second hand model mapped to the dominant joint based on the target pose data. Based on the updated vertex positions of the second hand model, determine the next dominant joint to be processed in the order from fingertip joint to finger root joint. Return to the step of obtaining the optimal rotation axis of the dominant joint based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding lever arm vectors.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: When the target hand model is detected to be switching modes, the hand skeleton pose data calculated in the free space mode and the physical interaction mode are interpolated and mixed to make the first hand model render smoothly during the mode transition period.

7. A hand gesture interaction rendering device, characterized in that, include: The target hand model acquisition unit, driving discrimination unit, free space driving unit, and physical interaction driving unit are included. The target hand model obtaining unit is used to obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model to the same hand bone hierarchy structure, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a simplified hand model for physical interaction collision detection. The driving discrimination unit is used to detect whether the second hand model collides with an object in the virtual environment, and to determine whether the target hand model is currently in free space mode or physical interaction mode based on the collision detection result. The free space driving unit is used to calculate the first hand skeleton posture data using the hand motion data captured by motion capture when the target hand model is in the free space mode, and to synchronize the first hand skeleton posture data to the first hand model for hand rendering. The physical interaction driving unit is used to obtain the target displacement vectors of each target vertex that collides in the second hand model when the target hand model is in the physical interaction mode; and to map the dominant joint that has the greatest impact on the deformation of the target vertex using the skin weight data of the target displacement vectors of the target vertex. Following the order from the fingertip joint to the finger root joint, reverse skeletal pose processing is performed on each of the dominant joints in sequence to obtain the second hand skeletal pose data, and the second hand skeletal pose data is synchronized to the first hand model for hand rendering.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the hand motion interaction rendering method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the hand motion interaction rendering method as described in any one of claims 1 to 6.

Images