Hand action interaction rendering method and device, storage medium and electronic equipment

Abstract

The invention provides a hand action interaction rendering method and device, a storage medium and electronic equipment, and is applied to the technical field of visual rendering. A target hand model provided by the invention is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchical structure, the geometric size of the second hand model is greater than that of the first hand model, and the first hand model is a hand detail model for hand rendering. The second hand model is a hand simplified model used for physical interaction collision detection. And detecting collision with an object in the virtual environment through the simplified hand model, and judging the mode of the target hand model. In the free space mode, skeleton postures are calculated through motion capture data and synchronized to hand detail model rendering, and in the physical interaction mode, according to vertex displacement of a hand simplified model, more accurate postures are obtained and synchronized to hand detail model rendering, so that the hand model is effectively prevented from penetrating through a virtual object, and the user experience is improved. And the authenticity and reliability of hand actions are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual rendering, in particular to a hand motion interaction rendering method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the development of virtual reality and human-computer interaction, hand motion capture technology is widely used. The existing technology mainly includes calculating bone posture through sparse sensors and neural networks, realizing motion recognition through video frame extraction of hand key points, and improving high-speed motion recognition response speed by using event cameras.

[0003] However, the existing technology still has the problems of insufficient smoothness, poor motion continuity and obvious jumping sensation under high-speed complex motion. Especially when the hand interacts with virtual objects, the "model penetration" phenomenon generally occurs, affecting the visual reality and immersion experience.

[0004] Therefore, how to effectively eliminate the model penetration phenomenon in hand physical interaction and improve the authenticity and reliability of hand motion has become a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] In view of the above problems, the present application provides a hand motion interaction rendering method, device, storage medium and electronic device which overcome the above problems or at least partially solve the above problems, and the technical solutions are as follows:

[0006] A hand motion interaction rendering method comprises:

[0007] obtaining a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchy, the geometric size of the second hand model is greater than that of the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a hand simplified model for physical interaction collision detection;

[0008] detecting whether the second hand model collides with an object in a virtual environment, and judging whether the target hand model is currently in a free space mode or a physical interaction mode according to the collision detection result;

[0009] in the case that the target hand model is in the free space mode, calculating first hand skeleton posture data by using captured hand motion data, and synchronizing the first hand skeleton posture data to the first hand model for hand rendering;

[0010] In a case where the target hand model is in the physical interaction mode, second hand skeleton pose data is determined according to vertex displacement of the second hand model, and the second hand skeleton pose data is synchronized to the first hand model for hand rendering.

[0011] A hand motion interaction rendering apparatus, comprising: a target hand model obtaining unit, a driving discrimination unit, a free space driving unit, and a physical interaction driving unit.

[0012] The target hand model obtaining unit is configured to obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on a same hand skeleton hierarchy, the second hand model has a larger geometric size than the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interaction collision detection.

[0013] The driving discrimination unit is configured to detect whether the second hand model collides with an object in a virtual environment, and determine whether the target hand model is currently in a free space mode or a physical interaction mode according to a collision detection result.

[0014] The free space driving unit is configured to, in a case where the target hand model is in the free space mode, calculate first hand skeleton pose data by using captured hand motion data, and synchronize the first hand skeleton pose data to the first hand model for hand rendering.

[0015] The physical interaction driving unit is configured to, in a case where the target hand model is in the physical interaction mode, determine second hand skeleton pose data according to vertex displacement of the second hand model, and synchronize the second hand skeleton pose data to the first hand model for hand rendering.

[0016] A computer readable storage medium, having a program stored thereon, the program being executed by a processor to implement the hand motion interaction rendering method.

[0017] An electronic device, comprising at least one processor, and at least one memory connected with the processor through a bus; wherein the processor, the memory, and the bus complete mutual communication; the processor is configured to invoke program instructions in the memory to execute the hand motion interaction rendering method.

[0018] By the technical scheme, the hand action interactive rendering method, device, storage medium and electronic equipment are provided, the target hand model is obtained, wherein the target hand model is obtained by binding the first hand model and the second hand model on the same hand skeleton hierarchy, the geometric size of the second hand model is greater than that of the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interactive collision detection; whether the second hand model collides with an object in a virtual environment is detected, and whether the target hand model is currently in a free space mode or a physical interactive mode is determined according to a collision detection result; in the case that the target hand model is in the free space mode, hand action data captured by action capture is used to calculate first hand skeleton posture data, and the first hand skeleton posture data is synchronized to the first hand model for hand rendering; in the case that the target hand model is in the physical interactive mode, second hand skeleton posture data is determined according to vertex displacement of the second hand model, and the second hand skeleton posture data is synchronized to the first hand model for hand rendering. The hand detail model with high-precision visual performance and the hand simplified model with a larger geometric size are bound on the same hand skeleton hierarchy, the hand detail model is used for visual rendering, collision detection is performed by using the hand simplified model, and the free space mode and the physical interactive mode are distinguished, the skeleton posture is inversely corrected according to the vertex displacement of the hand simplified model in the physical interactive mode, the phenomenon of model penetration can be effectively prevented, and the authenticity and reliability of hand action are improved.

[0019] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to depict only preferred embodiments of the application, and therefore should not be considered to limit the scope of the application in any way. Similarly, like reference numerals have been used in the drawings to depict like parts of the application. In the drawings:

[0021] Figure 1 A flowchart of an embodiment of the hand action interactive rendering method provided by the embodiment of the present application is shown;

[0022] Figure 2 A flowchart of an embodiment of step S120 in the hand action interactive rendering method provided by the embodiment of the present application is shown;

[0023] Figure 3A flowchart showing a specific implementation of step S130 in the hand action interaction rendering method provided by the embodiment of the application is shown in FIG. 13;

[0024] Figure 4 A flowchart showing hand action interaction rendering control provided by the embodiment of the application is shown in FIG. 14;

[0025] Figure 5 A structural diagram of a hand action interaction rendering device provided by the embodiment of the application is shown in FIG. 15;

[0026] Figure 6 A structural diagram of an electronic device provided by the embodiment of the application is shown in FIG. 16. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be completely conveyed to those skilled in the art.

[0028] With the rapid development of the fields of virtual reality (VR), augmented reality (AR), and human-computer interaction, hand action capture technology has become an important foundation for realizing a natural interactive experience. There are various hand action capture technology solutions on the current market, which can be mainly divided into categories based on inertial measurement units (IMUs), computer vision and deep learning, and new sensors such as event cameras. Each type of technology has its own advantages in actual applications, but when the hand interacts physically with virtual objects, the current mainstream technologies generally have the following shortcomings: limited action smoothness, especially prone to lag during high-speed and complex operations; weak action sequence continuity, often with action jumps or mutations, affecting the sense of immersion; in virtual scenes, when the hand contacts virtual objects, the phenomenon of "model penetration" (i.e., the hand model penetrates the virtual object) often occurs, seriously damaging the physical authenticity and visual consistency; in complex interactive scenarios, it is often difficult for the system to simultaneously guarantee high precision, low latency, and physically consistent dynamic interaction effects.

[0029] Based on this, the present invention provides a hand motion interaction rendering method, employing a "high-low" dual-model collaborative architecture. A high-precision, detailed model handles visual rendering to enhance the realism of the image, while a low-polygon simplified model is hidden within the physics engine for efficient collision detection and deformation simulation. Based on a state-adaptive hybrid driving mechanism, when physical interaction is detected, real-time collision and deformation data from the low-poly model are used to calculate the skeletal posture that satisfies human physiological constraints through inverse dynamics, effectively preventing the hand from penetrating virtual objects. In free movement, the positions of IK (Inverse Kinematics) control points are collected and combined with finger joint motion constraints to optimize the naturalness and smoothness of finger movements. Therefore, the present invention can improve the realism and immersion of hand-to-virtual object interaction while ensuring smooth movement and consistency of physical rules, thus significantly improving the realism and reliability of the hand motion interaction experience.

[0030] like Figure 1 The diagram shows a flowchart of one embodiment of the hand motion interaction rendering method provided by this invention. The method may include:

[0031] S100. Obtain the target hand model, wherein the target hand model is obtained by binding the first hand model and the second hand model to the same hand bone hierarchy structure. The geometric size of the second hand model is larger than that of the first hand model. The first hand model is a hand detail model used for hand rendering, and the second hand model is a simplified hand model used for physical interaction collision detection.

[0032] The target hand model refers to a 3D (Three-Dimensional) hand model used for hand movement simulation and interaction in a virtual environment. It consists of a first hand model and a second hand model, both bound to the same skeletal hierarchy. The target hand model is rendered using the first hand model, and the visibility of the second hand model is disabled.

[0033] The first hand model, also known as the high-polygon hand model, refers to a 3D hand model with a relatively large number of vertices and rich details. It is mainly used for visual rendering to show fine features such as skin and wrinkles, and to enhance the realism of the image.

[0034] The second hand model, also known as the low-face hand model, refers to a 3D hand model with a relatively small number of vertices and a simplified geometric structure. Its geometric size is slightly larger than that of the first hand model. It is used to hide behind the visual layer and is dedicated to the physics engine for collision detection and deformation simulation.

[0035] The hand skeletal hierarchy refers to the skeletal system used to drive the hand model, including the wrist joint, metacarpal joints, and finger joints, which defines the motion hierarchy and rotational constraints of the hand.

[0036] Specifically, the embodiment of the present application can pre-construct a standard hand skeleton hierarchy containing wrist joints, metacarpal joints and phalangeal joints, ensure that each joint position is in the default state and aligned with the hand model. Then, a first hand model with rich details for final visual rendering is provided, and a second hand model with simplified geometry and fewer vertices is provided, the overall size of the second hand model is slightly larger than that of the first hand model to reduce the risk of wearing the model. The first hand model and the second hand model are respectively bound to the same pre-constructed hand skeleton hierarchy, both of which share the same bone nodes but have independent skin weight data, only the first hand model is rendered at runtime, and the second hand model is hidden and used for collision detection and deformation simulation of the physics engine, thereby completing the construction of the target hand model.

[0037] The default state of the joint position provided by the embodiment of the present application can refer to the initial posture of each joint in the hand skeleton hierarchy without being driven by motion capture data, without being affected by any external force or animation. For example: the phalangeal joints of each finger are in a natural straight state, the rotation angle of each joint is zero, and the overall hand is in a relaxed and flat reference posture. Keeping the default state of each joint position can ensure that the hand skeleton hierarchy is accurately aligned with the first hand model and the second hand model after binding in spatial position, such as the center point of the model driven part, to provide a unified initial reference coordinate system for deformation calculation and inverse solution in subsequent physical interaction.

[0038] S110, detecting whether the second hand model collides with the object in the virtual environment, judging whether the target hand model is currently in a free space mode or a physical interaction mode according to the collision detection result.

[0039] The virtual environment refers to a three-dimensional scene generated by a computer, which can include interactive objects, spaces and virtual objects, etc., and the hand model realizes motion and interaction in the environment.

[0040] The free space mode refers to the state when the hand model does not collide with the virtual object, and the natural motion of the hand is driven by motion capture data.

[0041] The physical interaction mode refers to the state when the second hand model collides with the virtual object, and the bone posture is inversely corrected according to the physical data generated by the collision of the second hand model.

[0042] Specifically, the embodiment of the present application can use a rendering engine or a physics engine to perform real-time collision detection on the second hand model and other objects in the virtual environment, calculate the vertex distance between the second hand model and all relevant virtual objects in the virtual scene, and determine that a collision occurs when some vertex positions overlap or the distance is below a set threshold. According to the collision detection result, if the second hand model does not overlap with any virtual object in the vertex, the target hand model is in the free space mode; otherwise, if any vertex is detected to be in contact, the target hand model switches to the physical interaction mode, providing a basis for subsequent motion driving and physical simulation switching.

[0043] S120, in the case where the target hand model is in the free space mode, using the hand motion data captured by motion capture, calculating the first hand skeleton pose data, and synchronizing the first hand skeleton pose data to the first hand model for hand rendering.

[0044] Wherein, the hand motion data refers to the hand motion information obtained by the motion capture device (such as optical camera and data gloves), including joint position and attitude data.

[0045] Wherein, the first hand skeleton pose data refers to the hand skeleton joint attitude calculated by the motion capture data in the free space mode, which is used to render the first hand model.

[0046] Wherein, the hand rendering refers to applying the calculated skeleton pose data to the first hand model to realize the visual performance and animation effect of the hand.

[0047] Specifically, the embodiment of the present application can use the hand motion data captured by motion capture to perform inverse dynamics calculation in the case where the target hand model is in the free space mode, calculate the first hand skeleton pose data, and synchronize the first hand skeleton pose data to the first hand model for hand rendering.

[0048] Wherein, the inverse dynamics calculation refers to an algorithm that reverses the angles and attitudes of each skeleton joint based on a given target point (such as the position of the fingertips), so that the hand movement is natural and meets the physiological constraints.

[0049] Further, the embodiment of the present application can obtain the position data of the target joint in the world coordinate system in the case where the target hand model is in the free space mode, convert it to the local coordinate system with the palm as the center, and then perform physiological motion range constraint to correct its position. Based on the corrected joint position, the finger motion plane is dynamically determined, and the middle finger joint between the fingertip joint and the finger root joint is projected onto the plane to determine its target position. According to the corrected joint position and the target position of the middle finger joint, the first hand skeleton pose data is solved by inverse kinematics.

[0050] S130, in the case that the target hand model is in the physical interaction mode, determining the second hand skeleton pose data according to the vertex displacement of the second hand model, and synchronizing the second hand skeleton pose data to the first hand model for hand rendering.

[0051] Wherein, the vertex displacement refers to the surface vertex position change of the second hand model due to collision in the physical interaction, and the skeletal motion is deduced by analyzing these changes.

[0052] Wherein, the second hand skeleton pose data refers to the skeletal joint pose calculated according to the physical deformation of the vertex of the second hand model in the physical interaction mode, which is used for rendering the first hand model to realize physically consistent hand animation.

[0053] Specifically, in the case that the target hand model is in the physical interaction mode, the second hand skeleton pose data is obtained by inversely solving the joint-by-joint skeletal rotation data according to the vertex displacement of the second hand model, and the second hand skeleton pose data is synchronized to the first hand model for hand rendering.

[0054] Wherein, the skeletal rotation data refers to the rotation axis and rotation angle corresponding to each joint, which describes the current pose of the skeletal system.

[0055] Further, in the case that the target hand model is in the physical interaction mode, the dominant joint which is most affected by the deformation of the target vertex of the second hand model can be determined based on the displacement vector and the skinning weight of the target vertex, and then the inverse solution is performed on each dominant joint in turn according to the order from the fingertip to the finger root to obtain the joint-by-joint skeletal rotation data, and the second hand skeleton pose data is obtained by collecting these skeletal pose data, and is synchronized to the first hand model to complete the hand rendering.

[0056] The application provides a hand action interactive rendering method, which comprises the following steps: obtaining a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchy, the geometric size of the second hand model is larger than that of the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interactive collision detection; detecting whether the second hand model collides with an object in a virtual environment, and judging whether the target hand model is currently in a free space mode or a physical interactive mode according to a collision detection result; in the case that the target hand model is in the free space mode, calculating first hand skeleton posture data by using hand action data captured by action capture, and synchronizing the first hand skeleton posture data to the first hand model for hand rendering; and in the case that the target hand model is in the physical interactive mode, determining second hand skeleton posture data according to vertex displacement of the second hand model, and synchronizing the second hand skeleton posture data to the first hand model for hand rendering. By binding the hand detail model with high-precision visual performance and the hand simplified model with a larger geometric size on the same hand skeleton hierarchy, the hand detail model is used for visual rendering, the hand simplified model is used for collision detection and distinguishing between the free space mode and the physical interactive mode, and the skeleton posture is inversely corrected according to the vertex displacement of the hand simplified model in the physical interactive mode, so that the phenomenon of model penetration can be effectively prevented, and the authenticity and reliability of hand action are improved.

[0057] Optionally, based on Figure 1 As shown in the method, Figure 2 As shown in the method, the application embodiment provides a flowchart of a specific implementation of step S120 in the hand action interactive rendering method, and step S120 specifically can comprise:

[0058] S200, obtaining first joint position data of a target joint in hand action data captured by action capture.

[0059] The first joint position data refers to original three-dimensional coordinate data of a target joint of a hand in a world coordinate system, which is directly collected by an action capture device (such as an optical camera or a data glove). The target joint can include a wrist joint, a fingertip joint, a middle finger joint and a finger root joint.

[0060] Specifically, the application embodiment can receive original space data from optical motion capture system marker points or data gloves built-in sensors in real time. By solving these original data, three-dimensional coordinates of the target joint including the wrist joint, the fingertip joint, the middle finger joint and the finger root joint in the world coordinate system, i.e. the first joint position data, are directly obtained.

[0061] S210, convert the first joint position data from the world coordinate system to a local coordinate system with the palm as the center to obtain second joint position data.

[0062] The second joint position data refers to three-dimensional coordinate data obtained by converting the first joint position data from the world coordinate system to a local coordinate system with the palm coordinate system as the origin, and is used to unify the reference system through coordinate conversion, eliminate the influence of the overall movement and rotation of the palm on the relative movement of the fingers, and simplify the subsequent inverse kinematics calculation.

[0063] Specifically, the embodiment of the present application can construct a local coordinate system with the wrist joint position as the origin according to the spatial orientation of the palm. Then, the world coordinates of all joints (including the fingertips and the middle fingers) are converted to this local coordinate system with the palm as the reference through a coordinate transformation matrix (including rotation and translation). After conversion, the position of the finger joint is no longer dependent on the absolute placement of the hand in the virtual world, but only represents the relative position of the palm, greatly simplifying the subsequent operation for the movement of the fingers themselves.

[0064] S220, physiological motion range verification is performed on the second joint position data to constrain the position of the target joint within a spherical motion domain with the metacarpophalangeal joint as the center and the maximum length of the phalanx as the radius, and third joint position data is obtained after correction.

[0065] The third joint position data refers to the corrected coordinate data obtained after the physiological motion range verification of the second joint position data.

[0066] The embodiment of the present application can construct a spherical motion domain with the metacarpophalangeal joint as the center and the maximum length of the phalanx when fully stretched as the radius, and if the joint exceeds the domain, it will be projected to the boundary of the sphere, ensuring that the finger movement conforms to the physiological limit of the human body.

[0067] Specifically, the embodiment of the present application can limit the motion range of the fingertip target point of each finger to prevent the finger from producing non-physiological overstretching or bending. A spherical motion domain is constructed with the metacarpophalangeal joint as the center and the maximum length of the phalanx when fully stretched as the radius. Then, it is detected whether the second joint position data of the fingertip joint is located within the sphere. If the point is located within the sphere or on the sphere surface, it is directly taken as the third joint position data; if it exceeds the sphere range, the point is pulled back to the sphere surface along the direction of the connection line between the point and the center (metacarpophalangeal joint), and the corrected position is the third joint position data.

[0068] S230, based on the third joint position data, a finger motion plane is dynamically calculated in combination with a preset back-of-hand direction polar vector.

[0069] The finger motion plane is a plane dynamically determined based on a line connecting the proximal interphalangeal joint and the distal interphalangeal joint and a preset polar vector of the back direction of the hand. A plane normal vector of the finger motion plane can be obtained by normalizing a cross product of a vector from the distal interphalangeal joint to the proximal interphalangeal joint and the polar vector, so as to ensure that the plane contains both the finger direction and the back direction of the hand and is used to constrain the natural motion trajectory of the finger when the finger is bent.

[0070] Specifically, the embodiment of the present application dynamically calculates a dedicated motion plane for each finger to ensure that the finger can naturally fall within a reasonable plane when bent (avoiding twisting). The distal-interphalangeal vector is obtained by using the third joint position data of the distal interphalangeal joint and the proximal interphalangeal joint which have been verified by the physiological range. The vector is cross-multiplied with a preset polar vector in the palm local coordinate system, the polar vector points to the back direction of the hand, and the result is normalized to finally obtain a unit normal vector perpendicular to the target plane. The plane defined by the normal vector contains both the finger direction and the back direction of the hand, thereby determining the natural motion trajectory plane of the finger when the finger is bent.

[0071] S240, projecting the position of the middle interphalangeal joint between the distal interphalangeal joint and the proximal interphalangeal joint to the finger motion plane to obtain fourth joint position data of the middle interphalangeal joint on the finger motion plane.

[0072] The middle interphalangeal joint refers to an intermediate joint between the proximal interphalangeal joint and the distal interphalangeal joint, and is a key transition joint in inverse kinematics solving of the finger. The position of the middle interphalangeal joint after being projected to the finger motion plane can determine the intermediate posture of the finger when the finger is bent, avoiding joint twisting or reverse physiological motion.

[0073] The fourth joint position data refers to the corrected coordinate data of the vertical projection of the original position of the middle interphalangeal joint to the finger motion plane. The middle interphalangeal joint can be constrained in the dynamically generated finger motion plane by the vector projection formula, so as to ensure that the joints are coplanar when the finger is bent as a whole, and the natural deformation is presented.

[0074] Specifically, after the finger motion plane is determined, the middle interphalangeal joint is constrained to the plane to determine the correct bending position. The position vector of the middle interphalangeal joint relative to the proximal interphalangeal joint is projected to the finger motion plane calculated in the previous step. The projection process finds a point through mathematical calculation, which is the vertical projection point of the middle interphalangeal joint on the target plane. The vertical projection point is the fourth joint position data, which is used to ensure that the middle interphalangeal joint can naturally fall within the correct motion plane when the finger is bent towards the target distal interphalangeal joint.

[0075] S250, calculating the first hand skeleton posture data through inverse kinematics according to the third joint position data and the fourth joint position data.

[0076] Specifically, in this embodiment of the invention, the fixed position data of the root and tip joints in the third joint position data, and the projected position data of each middle joint in the fourth joint position data, can be used to calculate the angle of rotation (i.e., skeletal pose data) required by the root and middle joints in order for the end of the skeletal chain (fingertip) to reach the specified position through an inverse kinematics solver. The first hand skeletal pose data containing a set of joint rotation values ​​is output for the first hand model to perform hand rendering, so that it presents a pose that is both consistent with motion capture input and natural and without distortion.

[0077] To facilitate understanding of the hand model driving principle in free space mode provided in this embodiment of the invention, an example is given here: When the second hand model does not collide with any virtual object, the target hand model enters free space mode. In this mode, hand movements are entirely driven by motion capture data to ensure high-fidelity reproduction of the user's real hand movements. By receiving raw input data from an optical motion capture camera or a data glove sensor, the position of the palm (wrist joint) and the positions of key joints of each finger (finger root, middle finger, fingertip) in the world coordinate system are obtained. Subsequently, all joint point data are uniformly transformed into a local coordinate system with the palm coordinates as the origin. This step eliminates the influence of overall palm movement and rotation, simplifying the problem to the processing of relative finger movements, thus facilitating subsequent calculations.

[0078] To overcome the unnatural finger twisting that may result from direct actuation, this invention introduces a motion plane-based optimization algorithm. This algorithm dynamically calculates a unique motion plane for each finger to ensure its bending trajectory conforms to physiological principles. This plane is formed by the finger root joint (…). ) pointing to the fingertip joint ( The vector of ) and a pre-defined polar vector representing the direction of the back of the hand ( Together, they determine whether to ensure that the defined plane simultaneously includes the direction of the finger. — (Connect the lines) and the direction of the back of the hand ( Its plane normal vector. The calculation formula is:

[0079] ;

[0080] After determining the plane of motion, the middle finger joint ( The constraint is applied to this plane. Through projection calculations, we obtain... Correction position on the target plane This ensures that the three joints ( ) when the finger is bent , , The joints always lie in the same natural plane, thus avoiding joint twisting. The calculation formula is:

[0081] ;

[0082] Simultaneously, strict physiological constraints are imposed on the range of motion of the fingertip joints (IK target points) to prevent displacement exceeding physiological limits. This constraint is achieved through a mechanism centered on the finger root joint (…). With the center of the ball as the maximum straight length of the finger bone ( This is achieved through a spherical motion domain with a radius of . For the target point The constraint rules are defined by the following mathematical formula:

[0083] ;

[0084] Where C represents the center of the spherical motion domain, i.e., in the formula... The position, C can also be represented as This formula indicates that if the target point is within the sphere, it is applied directly; if it is outside the sphere, it is pulled back onto the sphere along the direction from the center of the sphere to that point, and the corrected point is denoted as... .

[0085] This invention transforms the original motion capture data into ergonomically accurate skeletal posture data through coordinate system transformation, physiological range of motion verification, dynamic finger motion plane calculation, and joint projection. This provides high-quality input for inverse kinematics solving, thereby ensuring that the rendering result of the first hand model in free space mode can simultaneously achieve high-fidelity motion reproduction and natural and smooth visual performance.

[0086] Optional, based on Figure 1 The method shown is as follows: Figure 3 The diagram shows a specific implementation of step S130 in the hand motion interaction rendering method provided in this embodiment of the invention. Step S130 may specifically include:

[0087] S300: Obtain the target displacement vectors of each target vertex that collided in the second hand model.

[0088] In this context, target vertices refer to the specific vertices in the second hand model that collide with objects in the virtual environment. The physics engine detects that the positions of these vertices change due to the collisions, and these changed positions are the key data points for model deformation, serving as the input basis for subsequent skeletal pose adjustments.

[0089] The target displacement vector refers to the displacement difference vector of each collided target vertex from its position in the previous frame to its current position.

[0090] Specifically, the embodiment of the present application can detect the collision between the second hand model and the object in the virtual environment in real time through the physical engine, and capture all the vertex sets that occur contact. For each collision vertex, the position after the current physical simulation (target position) is obtained, and the position determined by the last frame of the skeleton pose (last frame position) of the last frame is combined, and the target displacement vector of the vertex is calculated through the vector difference.

[0091] S310, using the skinning weight data of the target displacement vector of the target vertex, mapping out the leading joint that has the greatest influence on the deformation of the target vertex.

[0092] The leading joint refers to the skeleton joint that has the greatest influence on a certain vertex in skinning weight distribution. Each vertex is usually affected by the weight of multiple joints, but the leading joint is the main driving force for the displacement of the vertex. By identifying the leading joint, the vertex deformation problem can be decomposed into single joint rotation adjustment.

[0093] Specifically, the embodiment of the present application can use the skinning weight information of the second hand model to map each target vertex that occurs collision and calculates the displacement vector to the skeleton joint with the largest weight, i.e. the leading joint. For example, the embodiment of the present application can find the joint weight data corresponding to the target vertex, determine the joint with the largest weight value as the leading joint, and include the vertex in the controlled vertex set of the joint. Through this mapping, the complex vertex deformation problem is decomposed into multiple single joint driving problems, which facilitates subsequent independent pose adjustment of each leading joint.

[0094] S320, sequentially performing reverse skeleton pose processing on each leading joint in order from the fingertip joint to the finger root joint, to obtain second hand skeleton pose data.

[0095] The reverse skeleton pose processing is a method of inversely calculating the rotation and pose change of the skeleton joint by analyzing the deformation of the model vertex (such as the vertex displacement caused by collision).

[0096] The embodiment of the present application can calculate the joint rotation axis and rotation angle according to the target displacement of the collision vertex and the geometric relationship between the vertex and the leading joint, so that the skeleton pose adjustment can reproduce the vertex deformation effect required by the physical engine, and ensure that the hand model deformation is natural and has no penetration phenomenon.

[0097] Specifically, the embodiment of the present application can process each dominant joint in reverse order from the fingertips (end joints) to the wrist (root joint) according to the hierarchical structure of the hand skeleton. For each dominant joint, the optimal rotation axis and rotation angle of the joint are solved by minimizing the difference between the predicted motion direction and the actual vertex displacement direction according to the target displacement vector and the force arm vector of the vertex set controlled by the dominant joint. Subsequently, the calculated rotation matrix is applied to update the current skeletal pose of the joint, and the vertex positions controlled by the joint are immediately synchronized. This process recursively covers all affected joints, and finally obtains the second hand skeleton pose data that meets the physical collision constraint, which is used to drive the natural rendering of the first hand model.

[0098] When the second hand model is detected to collide with an object in the virtual environment, the embodiment of the present application can effectively decompose the complex vertex deformation problem into single-joint rotation adjustment problem by obtaining the target displacement vector of the collision vertex and accurately mapping it to the corresponding dominant joint using the skinning weight. Then, the skeletal rotation data is calculated in reverse order according to the skeletal hierarchical order from the fingertips to the finger roots, ensuring that the rotation of each joint accurately reflects the deformation caused by the collision, and obtaining the second hand skeleton pose data in the physical interaction mode. Therefore, not only is the high matching between the hand skeleton pose and the physical collision deformation achieved, the naturalness and realism of the hand motion are improved, but also the model penetration phenomenon is effectively avoided, and the interaction experience and rendering effect of the virtual hand are improved.

[0099] Optionally, in the above Figure 3 Based on one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, step S320 can specifically include:

[0100] For the currently processed dominant joint: based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding force arm vectors, the optimal rotation axis of the dominant joint is obtained; using the average force arm vector, the average displacement vector and the optimal rotation axis of the dominant joint, the joint rotation angle of the dominant joint is obtained; the target pose data of the dominant joint is obtained by adjusting the pose of the dominant joint in the second hand model according to the joint rotation angle; and the target pose data of each dominant joint is summarized to obtain the second hand skeleton pose data.

[0101] Optionally, after obtaining the target pose data of the dominant joint by adjusting the pose of the dominant joint in the second hand model according to the joint rotation angle, the embodiment of the present application can update the positions of all vertices mapped to the dominant joint in the second hand model based on the target pose data, and determine the next processed dominant joint in the order of the fingertip joint to the finger root joint based on the second hand model after updating the vertex positions, and return to execute the step of obtaining the optimal rotation axis of the dominant joint based on the displacement vectors of all vertices mapped to the dominant joint and their corresponding force arm vectors.

[0102] Specifically, for the current dominant joint, the embodiment of the present application first collects all vertices mapped to the joint, which have target positions after collision. By calculating the displacement vectors of the vertices and their force arm vectors relative to the center of the joint, an optimization algorithm is used to find a unit vector as the rotation axis, so that the predicted motion direction generated by the rotation axis minimizes the difference with the actual motion direction of the vertices, thereby determining the optimal direction of joint rotation, i.e. the optimal rotation axis of the dominant joint. Next, the average force arm and average displacement of all mapped vertices are calculated. By projecting the average displacement vector onto a plane perpendicular to the rotation axis, the effective tangential displacement is obtained. Based on the size relationship between the force arm length and the tangential displacement, combined with an adjustment parameter, the joint rotation angle of the dominant joint is derived, ensuring that the joint rotation can maximize the reproduction of the actual displacement change of the vertices. According to the determined optimal rotation axis and joint rotation angle, the current pose of the joint is updated, and the joint is rotated around the axis by the corresponding angle to obtain the target pose data of the joint, which represents the reasonable new position of the joint under the constraint of physical collision. Using the new joint pose, the positions of all vertices within the control range of the joint are recalculated, and the vertex coordinates of the second hand model are updated to ensure consistency between the vertex positions and the joint pose, providing the latest vertex data for subsequent joint calculations. In order of the fingertips to the finger roots, each dominant joint is processed in turn. After processing each joint, the updated vertex positions of the joint are used to continue calculating the rotation axis and rotation angle of the next joint, and the loop is executed until the pose adjustment of all affected joints is completed. The updated pose data of all dominant joints is combined to form complete skeletal pose information, which is output as the second hand skeletal pose data. The second hand skeletal pose data can be directly applied to the first hand model to achieve a realistic hand dynamic rendering effect that is highly consistent with physical collision and has no penetration.

[0103] Optionally, the embodiment of the present application can use the average force arm vector and the optimal rotation axis of the dominant joint to obtain the predicted motion direction unit vector of the dominant joint; use the average displacement vector and the optimal rotation axis of the dominant joint to obtain the effective tangential displacement of the dominant joint; and use the predicted motion direction unit vector, the effective tangential displacement, and the average force arm vector to obtain the joint rotation angle of the dominant joint.

[0104] Specifically, after obtaining the optimal rotation axis of the dominant joint, the average force arm vector of all vertices mapped to the joint is first calculated, i.e., the average position vector of the vertices relative to the joint center. Then, a direction vector is obtained by the cross product operation of the rotation axis and the average force arm vector, and the direction represents the predicted motion direction of the vertices when the joint rotates. Next, the direction vector is normalized to obtain a normalized predicted motion direction unit vector. The average displacement vector reflecting the actual movement trend of the vertices is projected into a plane perpendicular to the optimal rotation axis to remove the component parallel to the rotation axis direction, and the remaining part is the tangential displacement that can be effectively driven by the joint rotation. The effective tangential displacement measures the actual vertex deformation component generated by the rotation of the joint around the rotation axis. Based on the mechanical lever principle, the size of the effective tangential displacement is compared with the length of the average force arm, and the relationship is that the joint rotation angle is proportional to the tangential displacement and inversely proportional to the length of the force arm. Combined with the direction relationship between the predicted motion direction unit vector and the tangential displacement (the rotation direction is determined by the sign), a reasonable rotation angle value, i.e., the joint rotation angle of the dominant joint, is calculated. The joint rotation angle can make the motion of the controlled vertex group after the joint rotation as close as possible to the displacement required by the physical engine, ensuring the naturalness and accuracy of the deformation.

[0105] The embodiment of the present application uses the average force arm vector and the optimal rotation axis of the dominant joint to first obtain the predicted motion direction unit vector guided by the joint rotation, which accurately reflects the driving direction of the joint rotation on the vertex motion. The average displacement vector of the dominant joint is projected onto a plane perpendicular to the optimal rotation axis to obtain an effective tangential displacement, which eliminates the invalid component parallel to the rotation axis, ensuring that only the actual vertex movement generated by the joint rotation is considered. Combined with the size and direction relationship of the predicted motion direction unit vector, the effective tangential displacement and the average force arm vector, a reasonable joint rotation angle is calculated, which can make the position change of the vertex group after the joint rotation match the deformation generated by the physical collision to the greatest extent, thereby effectively improving the accuracy and naturalness of the joint pose recovery, avoiding invalid rotation and deformation distortion, and significantly improving the realism and stability of the hand model in the physical interaction mode.

[0106] To facilitate understanding of the hand model driving principle in free space mode provided by the embodiments of the present application, the following is used as an example: when the physical engine detects that the second hand model collides with an object in the virtual environment, the physical interaction mode is automatically switched to. At this time, the second hand model will be in contact with the object in the scene (including the hand itself), causing the position of the vertex in the contact area to be offset. Based on the principle of motion skinning weight, the physical displacement of these local vertices is used to inversely solve the pose adjustment of the corresponding skeleton. The deformation is mainly caused by the rotation of the local joint. After calculating the rotation data of the joint, the data is synchronized to the first hand model, realizing the dynamic and non-clothing hand rendering effect.

[0107] To simplify the calculation, based on the assumption that "the vertices of the hand are mainly affected by a single joint", the overall problem is decomposed into independent processing of a single dominant joint. First, the target displacement vector of the collision vertex in the second hand model is calculated. The target position set of the vertex after collision is obtained through the physical engine , the old position of the corresponding vertex in the last frame (determined by the pose of the skeleton in the last frame), the displacement of each vertex is calculated:

[0108] ;

[0109] Where i represents the current vertex, is the displacement of the current vertex, is the target position of the current vertex, is the last frame position of the current vertex.

[0110] Using the skinning weight, each collision vertex is mapped to the dominant joint J that has the greatest impact on it, and the set of vertices dominated by the joint is denoted as C i .

[0111] Next, for each dominant joint J, the rotation axis is calculated as the basis for the direction of joint rotation. For each vertex v i in the set C i , the force arm vector from the joint center J to the vertex is calculated . According to the physical law, the predicted motion direction caused by joint rotation should be consistent with the actual displacement direction of the vertex, i.e. the predicted motion direction is , and the actual motion direction is . The optimal rotation axis is solved by minimizing the difference between the two:

[0112] ;

[0113] Where represents the value that minimizes the objective function, The vector a is a unit vector, only used to calculate the direction of the vector.

[0114] Obtaining the optimal rotation axis After that, the rotation angle Δθ of the joint is calculated, so that the vertex group displacement controlled by the joint rotation is as close as possible to the simulated average displacement. First, the average force arm vector r avg and the average displacement vector are calculated, and the is projected onto the plane perpendicular to the rotation axis to obtain the effective tangential displacement :

[0115] ;

[0116] The unit vector of the predicted motion direction is:

[0117] ;

[0118] Based on the principle of the lever, the rotation angle of the joint is proportional to the tangential displacement and inversely proportional to the average force arm length, and the calculation formula is:

[0119] ;

[0120] wherein, is an adjustable scaling factor, usually close to 1, used to compensate for errors caused by model simplification; is a sign function, used to ensure the correct direction of rotation. If the dot product is positive, sign takes +1, otherwise -1.

[0121] Subsequently, the joint J is rotated around the parent space rotation axis by an angle of , and the rotation matrix is updated as:

[0122] ;

[0123] wherein, is the updated rotation matrix, is the updated rotation matrix, is the rotation matrix corresponding to the rotation of by an angle of radians.

[0124] According to the new bone pose, the positions of all vertices in the second hand model affected by the joint are updated in real time, providing the latest data for the calculation of subsequent joints. According to the hand bone hierarchy, the above steps are repeated from the fingertips (end effector) to the wrist (root joint), ensuring that the positions of child joints are accurately updated before the calculation of parent joints, avoiding the accumulation of pose errors.

[0125] After the processing of all affected joints is completed, second hand skeleton pose data matching the physical interaction is obtained , and is synchronized to the first hand model, realizing real, non-penetration hand dynamic rendering.

[0126] The embodiment of the application sequentially performs reverse skeleton pose processing on each dominant joint in the order of fingertip joint to finger root joint, accurately solves the optimal rotation axis and rotation angle of each joint through the displacement vector and force arm vector mapped to the vertex of the dominant joint, can effectively reflect the influence of local vertex deformation on the skeleton pose; after the rotation adjustment of each joint, the position of the corresponding vertex of the second hand model is updated in time, ensuring that the subsequent joint calculation is based on the latest geometric state, avoiding cumulative error and penetration problem; finally, through the aggregation of the target pose data of all dominant joints, second hand skeleton pose data highly matching the physical collision is obtained, making the recovery of the hand skeleton pose more accurate and natural, significantly improving the interaction reality and visual effect of the virtual hand and the environment object, and ensuring the stability and non-penetration performance of the rendering of the first hand model.

[0127] Optionally, in the above Figure 1 On the basis of one or more embodiments corresponding thereto, another optional embodiment provided by the embodiment of the application can further include:

[0128] When it is detected that the target hand model performs mode switching, the hand skeleton pose data calculated in the free space mode and the physical interaction mode is interpolated and mixed to enable the first hand model to be smoothly rendered in the mode transition period.

[0129] Specifically, to ensure the visual continuity between the free space mode and the physical interaction mode, the embodiment of the application can interpolate the skeleton rotation data during the transition of the two states to avoid abrupt jumps. Specifically, spherical linear interpolation (Slerp) is adopted to fuse the two sets of skeleton poses:

[0130] ;

[0131] Among them, is the interpolated and mixed hand skeleton pose data; is the first hand skeleton pose data calculated in the free space mode; is the second hand skeleton pose data calculated in the physical interaction mode; is a mixing weight function that increases from 0 to 1 in the transition period, and the transition time is usually set within 0.1 to 0.3 seconds, realizing a smooth and natural visual experience.

[0132] In this embodiment of the invention, when the target hand model is detected to be switching between free space mode and physical interaction mode, the hand skeleton posture data calculated separately in the two modes are interpolated and mixed to achieve a smooth transition of skeleton posture, avoiding visual jumps or unnatural phenomena caused by sudden changes in posture, thereby ensuring the rendering continuity and natural performance of the first hand model during the mode switching process.

[0133] To facilitate understanding of the overall control process of hand motion interaction rendering provided in the embodiments of the present invention, this section combines... Figure 4 Explanation: Figure 4 This is a flowchart illustrating the hand motion interaction rendering control provided in this embodiment of the invention. During system initialization: a complete hand skeleton hierarchy is created, and a high-precision hand model (high-poly) and a simplified hand model (low-poly) are loaded. The high-poly and low-poly models are bound to the same skeleton system for unified driving. The low-poly model is set as a physical collider, assigned physical properties (mass, friction, etc.), and loaded into the physics engine, but it does not participate in the final rendering. The initial system state can be set to "free space mode." During the real-time running loop (frame-by-frame processing) stage: raw data from the motion capture device is received and converted into initial pose data for the hand skeleton. The physics engine monitors the collision between the low-poly model and virtual environment objects in real time (based on collision intensity or depth), automatically determining whether the current state is "free space mode" or "physical interaction mode." During the mode splitting processing stage: for free space mode, the range of motion of the fingertips and key control points is limited according to ergonomics to avoid unreasonable movements. Based on the captured data, the natural hand skeleton pose is calculated, and the finger bending plane is dynamically adjusted to prevent distortion and deformation. The calculation results are simultaneously assigned to the high-poly and low-poly models for rendering and physics calculations. For the physics-based interaction mode, the physics engine calculates the target position changes of low-poly vertices based on the laws of mechanics such as collision and friction. Inverse dynamics solution: Vertex mapping: Mapping deformed vertices to their corresponding dominant joints. Joint-by-joint iteration: Processing each joint sequentially from fingertip to finger root: Dynamically determining the rotation axis, analyzing the joint rotation direction based on the current vertex movement. Estimating the rotation angle, combining vertex displacement and lever arm distance to determine the joint rotation amount. Updating the bone pose, adjusting the low-poly vertex position in real time to provide the latest data for the next joint calculation. After all joints are processed, the final bone pose is simultaneously assigned to both the high-poly and low-poly models. During the mode transition processing phase, when a mode switch is detected (such as switching back from physics-based interaction to free space), a brief smooth transition mechanism is activated. By interpolating and blending the bone poses of the two modes, a smooth pose transition is achieved in a very short time (a fraction of a second), avoiding abrupt changes in movement and ensuring the continuity and natural smoothness of the rendered visuals.

[0134] Although the operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated, or that they are all performed, depending on the circumstances. Under certain circumstances, multitasking and parallel processing can be advantageous.

[0135] It should be understood that each of the steps recited in the method embodiments of the present application can be performed in a different order, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present application is not limited in this regard.

[0136] Corresponding to the above method embodiments, the embodiments of the present application also provide a hand action interactive rendering device, which can include a target hand model obtaining unit 10, a driving discrimination unit 20, a free space driving unit 30, and a physical interaction driving unit 40, as shown in the structure. Figure 5

[0137] The target hand model obtaining unit 10 is configured to obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on a same hand skeleton hierarchy, the second hand model has a larger geometric size than the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a hand simplified model for physical interaction collision detection.

[0138] The driving discrimination unit 20 is configured to detect whether the second hand model collides with an object in a virtual environment, and determine whether the target hand model is currently in a free space mode or a physical interaction mode according to a collision detection result.

[0139] The free space driving unit 30 is configured to, in a case where the target hand model is in the free space mode, calculate first hand skeleton pose data by using hand action data captured by motion capture, and synchronize the first hand skeleton pose data to the first hand model for hand rendering.

[0140] The physical interaction driving unit 40 is configured to, in a case where the target hand model is in the physical interaction mode, determine second hand skeleton pose data according to vertex displacement of the second hand model, and synchronize the second hand skeleton pose data to the first hand model for hand rendering.

[0141] Optionally, the free space driving unit 30 can include a first joint position data obtaining subunit, a second joint position data obtaining subunit, a third joint position data obtaining subunit, a finger motion plane calculation subunit, a fourth joint position data obtaining subunit, and an inverse kinematics calculation subunit.

[0142] The first joint position data obtaining subunit is configured to obtain first joint position data of a target joint in the hand action data captured by motion capture.​

[0143] The second joint position data obtaining subunit is configured to convert the first joint position data from a world coordinate system to a local coordinate system centered on the palm to obtain second joint position data.

[0144] The third joint position data obtaining subunit is configured to perform physiological motion range verification on the second joint position data to constrain the position of the target joint within a spherical motion domain centered on the metacarpophalangeal joint and having a maximum length of the phalanx as a radius to obtain third joint position data after correction.

[0145] The finger motion plane calculating subunit is configured to perform physiological motion range verification on the second joint position data to constrain the position of the target joint within a spherical motion domain centered on the metacarpophalangeal joint and having a maximum length of the phalanx as a radius to obtain third joint position data after correction.

[0146] The fourth joint position data obtaining subunit is configured to project the position of the middle phalanx joint between the fingertip joint and the metacarpophalangeal joint to the finger motion plane to obtain fourth joint position data of the middle phalanx joint on the finger motion plane.

[0147] The fourth joint position data obtaining subunit is configured to project the position of the middle phalanx joint between the fingertip joint and the metacarpophalangeal joint to the finger motion plane to obtain fourth joint position data of the middle phalanx joint on the finger motion plane.

[0148] Optionally, the physical interaction driving unit 40 can include a target displacement vector obtaining subunit, a leading joint mapping subunit, and a reverse skeletal pose processing subunit.

[0149] The target displacement vector obtaining subunit is configured to obtain target displacement vectors of each target vertex in the second hand model that has collision.

[0150] The leading joint mapping subunit is configured to map the leading joint that has the greatest influence on the deformation of the target vertex by using the skinning weight data of the target displacement vector of the target vertex.

[0151] The reverse skeletal pose processing subunit is configured to sequentially perform reverse skeletal pose processing on each leading joint in the order of the fingertip joint to the metacarpophalangeal joint to obtain second hand skeletal pose data.

[0152] Optionally, the inverse skeleton pose processing subunit can be specifically configured to: obtain an optimal rotation axis of the dominant joint based on the displacement vectors of all the vertices mapped to the dominant joint and the corresponding force arm vectors; obtain a joint rotation angle of the dominant joint by using the average force arm vector, the average displacement vector and the optimal rotation axis of the dominant joint; perform pose adjustment on the dominant joint in the second hand model according to the joint rotation angle to obtain target pose data of the dominant joint; and aggregate the target pose data of each dominant joint to obtain second hand skeleton pose data.

[0153] Optionally, the inverse skeleton pose processing subunit can be specifically configured to: obtain a predicted motion direction unit vector of the dominant joint by using the average force arm vector and the optimal rotation axis of the dominant joint; obtain an effective tangential displacement of the dominant joint by using the average displacement vector and the optimal rotation axis of the dominant joint; and obtain a joint rotation angle of the dominant joint by using the predicted motion direction unit vector, the effective tangential displacement and the average force arm vector.

[0154] Optionally, the inverse skeleton pose processing subunit can be further configured to: after obtaining the target pose data of the dominant joint by performing pose adjustment on the dominant joint in the second hand model according to the joint rotation angle, update the positions of all the vertices mapped to the dominant joint in the second hand model based on the target pose data, so as to determine the next dominant joint to be processed in the order from the fingertip joint to the root joint based on the second hand model after the vertex positions are updated, and obtain target pose data of the next dominant joint to be processed.

[0155] Optionally, the hand action interaction rendering device can further include a smoothing rendering unit.

[0156] The smoothing rendering unit is configured to, when it is detected that the target hand model performs mode switching, interpolate and mix the hand skeleton pose data calculated in the free space mode and the physical interaction mode respectively, so as to perform smoothing rendering on the first hand model in the mode transition period.

[0157] The application provides a hand action interactive rendering device, which is used for obtaining a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on a same hand skeleton hierarchy, the second hand model has a larger geometric size than the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interactive collision detection; whether the second hand model collides with an object in a virtual environment is detected, and whether the target hand model is currently in a free space mode or a physical interactive mode is determined according to a collision detection result; in the case that the target hand model is in the free space mode, first hand skeleton posture data is calculated by using hand action data captured by action capture, and the first hand skeleton posture data is synchronized to the first hand model for hand rendering; in the case that the target hand model is in the physical interactive mode, second hand skeleton posture data is determined according to vertex displacement of the second hand model, and the second hand skeleton posture data is synchronized to the first hand model for hand rendering. By binding the hand detail model with high-precision visual performance and the hand simplified model with a larger geometric size on the same hand skeleton hierarchy, the hand detail model is used for visual rendering, the hand simplified model is used for collision detection and distinguishing between the free space mode and the physical interactive mode, the skeleton posture is inversely corrected according to the vertex displacement of the hand simplified model in the physical interactive mode, the penetration phenomenon can be effectively prevented, and the authenticity and reliability of hand action are improved.

[0158] As to the device in the above-mentioned embodiments, the specific manners in which various units perform operations have been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0159] The hand action interactive rendering device includes a processor and a memory, and the above-mentioned target hand model obtaining unit 10, the driving distinguishing unit 20, the free space driving unit 30 and the physical interactive driving unit 40 are all stored in the memory as program units, and the corresponding functions are realized by executing the above-mentioned program units stored in the memory by the processor.

[0160] The processor includes a core, and the corresponding program units are called from the memory by the core. One or more than one core can be set, and the core parameters are adjusted to detect the collision between the hand simplified model and the object in the virtual environment, and to determine the mode in which the target hand model is located. In the free space mode, the skeleton posture is calculated by using the action capture data, is synchronized to the hand detail model for rendering, in the physical interactive mode, more accurate posture is obtained according to the vertex displacement of the hand simplified model, and is synchronized to the hand detail model for rendering, so that the hand model can effectively prevent the penetration of the virtual object, and the authenticity and reliability of the hand action are improved.

[0161] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the hand gesture interactive rendering method.

[0162] This invention provides a processor for running a program, wherein the program executes the hand motion interaction rendering method during runtime.

[0163] like Figure 6 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned hand gesture interaction rendering method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0164] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program that initializes a hand gesture interactive rendering method step.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0167] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0168] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0169] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0170] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scene, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0171] In the description of the present application, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the position or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation of the present application.

[0172] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other present or future technologies can provide. It must be noted that as used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. The terms "includes" and "comprising," as well as derivatives thereof, mean that various embodiments include, but are not limited to, the listed material or step or steps. The terms "sub- steps" of any method, "steps" of any method, "elements" of any system, and "components" of any system are interchangeable and do not require a particular order of execution. The terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term "about" when used before a numerical designation, has its ordinary meaning in the art, for example, it can refer to an approximation within 1, 2, 3, 4, 5%, 10% or another suitable non-zero percentage of the indicated value.

[0173] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0174] The embodiments of the present application are only for illustrating the present application but not for limiting the present application. The present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A hand motion interaction rendering method, characterized in that, The method comprises the following steps: obtaining a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on the same hand skeleton hierarchy, the second hand model has a larger geometric size than the first hand model, the first hand model is a hand detail model for hand rendering, and the second hand model is a hand simplified model for physical interaction collision detection; detecting whether the second hand model collides with an object in a virtual environment, and judging whether the target hand model is currently in a free space mode or a physical interaction mode according to a collision detection result; in a case where the target hand model is in the free space mode, calculating first hand skeleton pose data by using hand motion data captured by motion capture, and synchronizing the first hand skeleton pose data to the first hand model for hand rendering; in a case where the target hand model is in the physical interaction mode, determining second hand skeleton pose data according to vertex displacement of the second hand model, and synchronizing the second hand skeleton pose data to the first hand model for hand rendering.

2. The method of claim 1, wherein, The method for calculating the first hand skeleton pose data by using the hand motion data captured by motion capture comprises the following steps: obtaining first joint position data of a target joint in hand motion data captured by motion capture; converting the first joint position data from a world coordinate system to a local coordinate system with a palm as a center to obtain second joint position data; performing physiological motion range verification on the second joint position data to constrain the position of the target joint within a spherical motion domain with a metacarpal joint as a center and with a maximum length of a phalanx as a radius, and obtaining third joint position data after correction; dynamically calculating a finger motion plane based on the third joint position data and in combination with a preset back-of-hand direction polar vector; projecting a position of a middle finger joint between a fingertip joint and the metacarpal joint to the finger motion plane to obtain fourth joint position data of the middle finger joint on the finger motion plane; calculating the first hand skeleton pose data through inverse kinematics according to the third joint position data and the fourth joint position data.

3. The method of claim 1, wherein, The method for determining the second hand skeleton pose data according to the vertex displacement of the second hand model comprises the following steps: obtaining a target displacement vector of each target vertex of the second hand model that collides; mapping out a dominant joint that has the greatest influence on deformation of the target vertex by using skinning weight data of the target displacement vector of the target vertex; sequentially performing inverse skeleton pose processing on each dominant joint in an order from a fingertip joint to a metacarpal joint to obtain second hand skeleton pose data.

4. The method of claim 3, wherein, The method for sequentially performing inverse skeleton pose processing on each dominant joint in an order from a fingertip joint to a metacarpal joint to obtain second hand skeleton pose data comprises the following steps: for the currently processed dominant joint, obtaining an optimal rotation axis of the dominant joint based on displacement vectors of all vertices mapped to the dominant joint and corresponding force arm vectors of the displacement vectors. obtaining a joint rotation angle of the dominant joint by using the average force arm vector, the average displacement vector and the optimal rotation axis of the dominant joint; adjusting a pose of the dominant joint in the second hand model according to the joint rotation angle to obtain target pose data of the dominant joint; collecting the target pose data of each dominant joint to obtain second hand skeleton pose data.

5. The method of claim 4, wherein, The method further comprises: obtaining a predicted motion direction unit vector of the dominant joint by using the average force arm vector and the optimal rotation axis of the dominant joint; obtaining an effective tangential displacement of the dominant joint by using the average displacement vector and the optimal rotation axis of the dominant joint; obtaining a joint rotation angle of the dominant joint by using the predicted motion direction unit vector, the effective tangential displacement and the average force arm vector.

6. The method of claim 4, wherein, After the step of adjusting a pose of the dominant joint in the second hand model according to the joint rotation angle to obtain target pose data of the dominant joint, the method further comprises: updating positions of all vertices mapped to the dominant joint in the second hand model based on the target pose data, so as to determine the dominant joint to be processed next in the order of fingertip joint to finger root joint based on the second hand model after updating the vertex positions, and return to execute the step of obtaining the optimal rotation axis of the dominant joint based on the displacement vectors of all vertices mapped to the dominant joint and the corresponding force arm vectors.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: when it is detected that the target hand model switches modes, interpolating and mixing hand skeleton pose data calculated in the free space mode and the physical interaction mode respectively, so as to smoothly render the first hand model in a mode transition period.

8. An apparatus for rendering hand action interaction, the apparatus comprising: a hand action interaction rendering unit configured to: receive a hand action interaction input; and render a hand action interaction effect based on the hand action interaction input. The method further comprises: a target hand model obtaining unit, a driving discrimination unit, a free space driving unit and a physical interaction driving unit, the target hand model obtaining unit is configured to obtain a target hand model, wherein the target hand model is obtained by binding a first hand model and a second hand model on a same hand skeleton hierarchy, the second hand model has a larger geometric size than the first hand model, the first hand model is a hand detail model used for hand rendering, and the second hand model is a hand simplified model used for physical interaction collision detection; the driving discrimination unit is configured to detect whether the second hand model collides with an object in a virtual environment, and determine whether the target hand model is currently in a free space mode or a physical interaction mode according to a result of the collision detection; the free space driving unit is configured to, when the target hand model is in the free space mode, calculate first hand skeleton pose data by using hand motion data captured by motion capture, and synchronize the first hand skeleton pose data to the first hand model for hand rendering; and the physical interaction driving unit is configured to, when the target hand model is in the physical interaction mode, calculate second hand skeleton pose data by using hand motion data captured by motion capture, and synchronize the second hand skeleton pose data to the second hand model for physical interaction collision detection. The physical interaction driving unit is configured to, when the target hand model is in the physical interaction mode, determine second hand skeleton pose data according to vertex displacement of the second hand model, and synchronize the second hand skeleton pose data to the first hand model for hand rendering.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program, when executed by a processor, implements the hand motion interaction rendering method according to any one of claims 1 to 7.

10. An electronic device, comprising: The electronic device comprises at least one processor, at least one memory connected to the processor, and a bus; wherein the processor, the memory and the bus complete mutual communication; the processor is configured to call program instructions in the memory to execute the hand motion interaction rendering method according to any one of claims 1 to 7.

Images