Virtual reality interaction method, electronic device, storage medium and program product

Abstract

The invention provides a virtual reality interaction method, electronic equipment, a storage medium and a program product. The virtual reality interaction method comprises the following steps: when interacting with a virtual world, fusing a skeleton posture captured by a motion with a skeleton posture of a virtual world animation system; a physical effect is added to the virtual world in an interaction process, so that a real interaction behavior is simulated in the virtual world. According to the method, the motion space of an operator is expanded to an infinite virtual world by fusing the skeleton posture captured by the motion with the skeleton posture of the virtual world animation system, and the reality of interaction is enhanced by adding a physical effect to the virtual world, so that the purposes of improving the reality of user interaction and expanding the motion mapping space are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a virtual reality interaction method, electronic equipment, storage medium and program product. Background Art

[0002] Motion capture products are diverse in the market, and there are two main application scenarios: one is to collect sensor information and generate skeleton animation data offline for use in 3D games or movies, and the other is to collect sensor information in real time and project it into the skeleton for live broadcasts, etc. However, there are few that directly enhance the operator's immersive experience by combining XR (Extended Reality) technology, so motion capture operators cannot produce physically realistic interactive effects on the virtual world based on their own actions. In the process of implementing this embodiment, the inventor found that the prior art has the following problems: 1. Virtual characters cannot produce physically realistic interactive effects in the virtual world, which limits the realism of the user experience.

[0003] 2. The range of activities in the virtual world is almost unlimited, but the operator's movement space in reality is limited. There is a contradiction between the two, which leads to limited action mapping.

[0004] 3. The current focus of motion capture is on the accuracy of restoring motion postures, without considering accurately mapping real-world non-physical interactive actions to the virtual world and ensuring the realism of interactive behaviors. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a virtual reality interaction method, electronic device, storage medium and program product, which at least partially solve the problems of limited sense of reality and mapping existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a virtual reality interaction method, comprising: When interacting with the virtual world, the motion-captured skeletal poses are merged with the skeletal poses of the virtual world animation system; Add physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world.

[0007] Optionally, the step of fusing the motion-captured skeletal posture with the skeletal posture of the virtual world animation system includes: The center point of the skeleton is obtained according to the motion capture skeleton posture, the change value of the center point in each frame is mapped to the root skeleton point of the animation system, and the change value is subtracted from the relative motion of the root node of the motion capture.

[0008] Optionally, mapping the change value of the center point in each frame to the root bone point of the animation system and subtracting the change value from the relative motion of the motion-captured bone pose includes: Map the change value of the center point in each frame to the X-axis and Y-axis of the root bone point of the animation system, and subtract the change value on the X-axis and Y-axis from the relative motion of the pelvis captured by motion capture.

[0009] Optionally, obtaining the center point of the skeleton according to the motion captured skeleton posture includes: The midpoints of the paired bone points in the motion captured skeleton pose are projected onto the ground and then the center points of the bones are obtained by counting.

[0010] Optionally, the step of fusing the motion-captured skeletal posture with the skeletal posture of the virtual world animation system includes: The animation weight of the bone posture change is obtained according to the transition sequence of the interaction mode and the preset curve. The interpolation method is used based on the animation weight to fuse the motion captured bone posture with the bone posture of the virtual world animation system to obtain the fused bone posture.

[0011] Optionally, the step of fusing the motion-captured skeletal posture with the skeletal posture of the virtual world animation system includes: Get the bone chain that needs to be fused according to the interaction mode.

[0012] Optionally, the skeleton chain includes a left arm skeleton chain, a right arm skeleton chain, a spine skeleton chain, a head skeleton chain, a left leg skeleton chain and a right leg skeleton chain.

[0013] Optionally, adding physical effects to the virtual world during the interaction process includes: Add collision bodies to the motion-captured bones and implement physical interaction through collision detection.

[0014] Optionally, the collision detection includes: Use colliders to define the collision shape of objects; Filter out the collision bodies that need to be detected based on the collision shape, and use the detection algorithm to determine whether the collision occurs in the screening results; If a collision occurs, a collision response is calculated based on the physics information of the colliding surfaces.

[0015] Optional, collision detection, including: Add physical constraints at the collision point according to the collision detection results. The physical constraints include: Use the physics engine for geometry offset; Use sweeps for geometry offset; Add stiffness and damping settings based on the geometry offset using the physics engine; and / or Directly bind the relative transformation to a fixed value without offsetting the geometry.

[0016] In a second aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any virtual reality interaction method described in the first aspect.

[0017] In a third aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any virtual reality interaction method described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements any virtual reality interaction method described in the first aspect.

[0019] The virtual reality interaction method, electronic device, storage medium and program product provided by the present invention, wherein the virtual reality interaction method expands the operator's movement space into an infinite virtual world by fusing the skeletal posture of motion capture with the skeletal posture of a virtual world animation system, and enhances the authenticity of the interaction by adding physical effects to the virtual world, thereby achieving the purpose of improving the realism of user interaction and expanding the action mapping space. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0021] Figure 1 A framework diagram of a virtual reality interaction method provided by an embodiment of the present disclosure; Figure 2 A flowchart of a virtual reality interaction method provided by an embodiment of the present disclosure; Figure 3 A block diagram of the principle of a virtual reality interaction device provided in an embodiment of the present disclosure; Figure 4 A functional block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0023] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0024] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0025] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0026] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0027] Motion capture refers to the process of transcribing the movements of real actors into those of digital models through sensors and software in 3D games or animations.

[0028] For ease of understanding, Figure 1 and Figure 2 As shown, this embodiment discloses a virtual reality interaction method, including: Step S101: When interacting with the virtual world, the motion captured skeleton posture is merged with the skeleton posture of the virtual world animation system; Optionally, the fusion of the motion captured skeletal posture with the skeletal posture of the virtual world animation system includes: obtaining the center point of the skeleton according to the motion captured skeletal posture, mapping the change value of the center point in each frame to the root skeletal point of the animation system, and subtracting the change value from the relative motion of the root node of the motion capture.

[0029] Optionally, mapping the change value of the center point in each frame to the root bone point of the animation system and subtracting the change value from the relative motion of the motion-captured bone pose includes: Map the change value of the center point in each frame to the X-axis and Y-axis of the root bone point of the animation system, and subtract the change value on the X-axis and Y-axis from the relative motion of the pelvis captured by motion capture.

[0030] Optionally, obtaining the center point of the skeleton according to the motion captured skeleton posture includes: The midpoints of the paired bone points in the motion captured skeleton pose are projected onto the ground and then the center points of the bones are obtained by counting.

[0031] Action and posture fusion: Regardless of the motion capture technology used, the range of motion of the motion capture operator in the real world is a limited space, which cannot directly correspond to the nearly infinite virtual world. At the same time, similar to the character animation performance in the game, the skeletal animation performance of the virtual character movement or some complex interactions can be obtained through pre-set animation sequences and animation state machines and other technologies. This embodiment designs a method for fusion of action and posture based on the characteristics of the motion capture character, mixing the skeletal posture of the animation system and the skeletal posture of the motion capture, so that the virtual person can move and interact freely in the virtual world.

[0032] The output of motion capture technology is generally described as the relative transformation (including position transformation and rotation transformation) between each bone joint point on a complete set of bone chains. In the industry bone specification standard, the root node of the human skeleton is the pelvis, so the overall motion information of the motion capture virtual human is described by the transformation matrix of the pelvis. In general animation systems, the overall motion information of the virtual human is described by the movement of the root bone point on the X-axis and Y-axis of the world coordinate system. The pelvis joint point of the virtual human is generally used to describe the motion information changes of the human hip in the animation system. Therefore, before the action fusion, the motion information of the two systems needs to be matched: first, the center point of the character is calculated based on the distribution of the character's bone points. The center point of the character is obtained by projecting the midpoint of the paired bone points onto the ground and then counting. The change value of the center point of the character in each frame is mapped to the X-axis and Y-axis of the root bone point of the character, and the change value on the X-axis and Y-axis is subtracted from the relative movement of the pelvis.

[0033] The relative transformation of each skeletal joint can be expressed as a transformation matrix , including position and translation: , Calculate the position change of the center point of each frame: , in, .

[0034] Map the changes of the center point to the X and Y axes of the root bone point: , The new position of the root bone point is: , Subtract the change in the center point on the X and Y axes from the relative motion of the pelvis to eliminate the effect of the global displacement on the pelvis: , in Is The translation matrix is: , Its inverse matrix is ​​the inverse of the translation: .

[0035] Optionally, the fusion of the motion captured skeletal posture with the skeletal posture of the virtual world animation system includes: obtaining the animation weight of the skeletal posture change according to the transition order of the interaction mode and a preset curve, and fusing the motion captured skeletal posture with the skeletal posture of the virtual world animation system using an interpolation method based on the animation weight to obtain a fused skeletal posture.

[0036] Optionally, the fusing of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: acquiring a skeleton chain to be fused according to an interaction mode.

[0037] Optionally, the skeleton chain includes a left arm skeleton chain, a right arm skeleton chain, a spine skeleton chain, a head skeleton chain, a left leg skeleton chain and a right leg skeleton chain.

[0038] After the motion information of the motion capture and general animation systems are matched, the motion fusion begins, and the motion fusion operation is split into multiple local skeleton chains. The skeleton chains designed in this embodiment during verification are split as follows: left arm skeleton chain, right arm skeleton chain, spine skeleton chain, head skeleton chain, left leg skeleton chain, and right leg skeleton chain. The skeleton structure is a tree structure, from the root node down, from the root node down, specifically the pelvis is the root node and the parent node, and then the leg bones are child nodes, from the root node down, that is, from the parent node to a branch node, layer by layer. According to the current interaction mode, which skeleton chains need to be fused are obtained. During the fusion process, the moving part is not fused with the animation of the virtual world, the moving part is controlled according to the motion capture, and the non-moving part is fused with the animation of the virtual world. For example, when writing, the right arm skeleton chain moves in the virtual world according to the motion capture signal, and other skeleton chains are fused with the animation of the virtual world. The interaction mode supports user customization. That is, when the person in motion capture squats, the interaction mode is that the lower body moves first and then drives the upper body to move. Then, according to the transition order of the interaction mode, the weight change trend of different bone postures is specified. The weight change trend is determined according to the preset curve provided. After the weights of each animation are determined, the two animations are mixed using classic interpolation methods such as linear interpolation, spherical linear interpolation, and Bezier curve interpolation to obtain the final bone posture result.

[0039] Bezier curve, also known as Bezier curve or Bezier curve, is a mathematical curve used in two-dimensional graphics applications. General vector graphics software uses it to accurately draw curves. Bezier curves are composed of line segments and nodes. Nodes are draggable fulcrums, and line segments are like retractable rubber bands.

[0040] The weight function is:

[0041] Position Interpolation (LERP):

[0042] Rotational interpolation (SLERP):

[0043] Bezier interpolation: , in, , , Bezier curve control points.

[0044] Skeletal animation is a technique in computer graphics used to animate the movement of characters or objects. Skeletal animation manipulates the character model using an internal bone structure instead of directly manipulating each vertex of the model. The following are some key implementation details of skeletal animation: Skeleton: Usually composed of a series of bones or joints connected in a hierarchical structure to form an internal framework, similar to the skeleton of real-world organisms.

[0045] Joints: Each joint (or bone node) has its own transformation (including position, rotation and scale) which can be relative to the parent joint.

[0046] Skin Binding: Vertex Weights: Each vertex of the model is assigned one or more joints, and each joint has a weight value that indicates how much influence the joint has on the vertex position.

[0047] Bind Pose: During the rigging process, the initial shape of the model (the bind pose) is recorded, usually in a T-pose.

[0048] Animation Keyframes: Create keyframes that define the transformation of a joint at a specific point in time. Interpolation: When the animation plays, interpolation (usually linear or non-linear) is performed between keyframes to produce smooth motion. Real-time Transformations: Bone Transformations: As the animation plays, the transformation of each joint is calculated in real time based on the key animation. Vertex Transformations: The final position of each vertex is then calculated based on the joint's transformation and the vertex weights. Normal Transformations: In order to render lighting correctly, the vertex normals also need to be updated based on the bone transformations.

[0049] Vertex shader: Modern graphics use vertex shaders on the GPU to accelerate the calculation of skeletal animation. The vertex shader can process each vertex in parallel, greatly improving efficiency.

[0050] Animation blending: In practical applications, you may need to play multiple animations at the same time (for example, shooting the upper body while walking). In this case, the system will perform a weighted blend of the joint transformations of multiple animations to generate the final animation.

[0051] Inverse Kinematics: Used to manipulate models more intuitively in some cases. Allows you to specify the position of an end effector (such as a hand or foot), and then automatically calculates the necessary joint angles.

[0052] The implementation of skeletal animation involves complex mathematics and programming techniques.

[0053] In skeletal animation, the skeletal structure (also called rigid body animation or character skeleton) refers to a set of bones or joints organized in a hierarchical manner that form the internal framework of a character or object. This structure is key in the animation process because it defines how the character moves.

[0054] Here are some key features of bone structure: Hierarchy, Root Bone: Usually located at the top of the bone hierarchy, controls the global position and orientation of the entire bone structure.

[0055] Parent bones and child bones: Each bone (except the root bone) has a parent bone and may have multiple child bones. The transformation of a child bone is relative to the transformation of its parent bone.

[0056] Bone properties: Joint: Each node in the bone structure can be understood as a joint, which allows rotation and sometimes translation. Bone: The part that connects two joints, which can be regarded as a bone, which defines the distance and direction between the joints.

[0057] Animation Control: Transformations: Each bone has its own transformation, including translation, rotation, and scale. These transformations can be controlled through keyframe animations.

[0058] Inverse Kinematics: In some cases, it may be used to more intuitively manipulate skeletons, especially when precise control of the position of end effectors (such as hands or feet) is required.

[0059] Weighted skinning associated with the model: Each model vertex is assigned one or more bone weights, which determine how much the bone transformation affects the vertex position. Binding pose, the initial pose when the model and the bone structure are bound together, is usually a neutral pose, such as a T-pose. Animation data: Set keyframes to define the transformation of bones at specific points in time.

[0060] Animation Curves: These curves describe the details of a bone's transformation over time and are used to interpolate between keyframes.

[0061] The design of the bone structure is critical to the quality of the animation. A good bone structure should be able to simulate real-world biological movements while remaining simple enough to be efficiently processed by computers for control and animation. In practice, the complexity of the bone structure will vary depending on the needs of the character and the level of detail of the animation.

[0062] Skin binding, in computer graphics, and specifically in skeletal animation, refers to the process of associating the surface (skin) of a 3D model with its skeletal structure. This process ensures that when the skeleton moves, the surface of the model deforms accordingly. The key to skin binding is to ensure that the model's movement looks natural and realistic. Here are some of the main components and steps of skin binding: Bind Pose: Before you begin skin rigging, the model needs to be in a bind pose, usually a neutral pose like a T- or A-pose that maximizes the range of motion of the joints.

[0063] Weight distribution: Vertex weight: Each model vertex is assigned the weight of one or more bones. This weight determines the degree of influence of the bone action on the vertex position.

[0064] Weight normalization: Usually all weights of a vertex should add up to 1 to keep the model volume consistent.

[0065] Linear blend skinning, also known as vertex blending, is the most common skinning technique, where the final position of a vertex is calculated as a weighted average of the transforms of all the bones that affect it.

[0066] Dual quaternion skinning can better handle deformation caused by rotation and reduce the candy wrapper effect.

[0067] Skinning process: Initial binding: In the binding pose, each vertex of the model is associated with one or more bones and assigned weights.

[0068] Real-time deformation: As the animation plays, the new position of each vertex is calculated in real time based on the bone's current transformation and the vertex weights.

[0069] Test and tweak: Test the rig, running it through various poses in the animation to check for unnatural deformations. Adjust weights as needed to ensure the animation is visually smooth and realistic.

[0070] Binding pose: The binding pose is the reference pose of the 3D model when it is bound to its skeleton system. This pose is usually a standardized standing pose of the character, such as a T-pose, where the character's arms are stretched horizontally, legs are upright, and facing forward. Such a pose provides a clear starting point for them to observe and edit the weights of the bones and ensure that the transition between different movements is smooth.

[0071] In the bind pose, each bone is in its default rotation and position. This defines the initial assignment of vertex weights during skinning.

[0072] Once a model is bound to a skeleton, all animation transforms are calculated relative to the bound pose.

[0073] Choosing a bind pose is crucial for later animation work, as it affects how well the weights are distributed and how natural the animation is. For example, if a character's arms are hanging down in the bind pose, then raising the arms up during animation may result in unnatural stretching of the skin. Therefore, a T-pose is often chosen because it allows unrestricted movement of the arms and legs, and makes it easier to distribute weights to get more natural movement in animation.

[0074] In the actual animation process, once the binding pose is determined and the model is bound to the skeleton, try to avoid changing the binding pose, because this will affect the weight distribution that has been set, which may require a lot of re-adjustment work.

[0075] Weight assignment: Weight assignment, also known as weight skinning or vertex weighting, is a key step in the skinning process of a 3D model. It involves associating each vertex of the model with one or more bones and assigning a weight value that determines the extent to which the vertex follows the movement of the bone. This process is to ensure that the skin of the model can deform in a natural way when the skeleton animation is played.

[0076] Key concepts of weight assignment: Vertex weight: Each vertex can be assigned to one or more bones, and each bone has a weight value for the vertex. The weight value is usually between 0 and 1, where 0 means the vertex is not affected by the bone and 1 means the vertex is fully affected by the bone.

[0077] Weight normalization: The sum of all weight values ​​of a vertex should usually equal 1. That is, if a vertex is affected by multiple bones, the sum of the influence of these bones on the vertex should be the full influence.

[0078] Weight smoothing: To avoid sharp or unnatural deformations of the model during animation, weight assignments should transition smoothly between bones. This is usually achieved with weight smoothing tools or by manually adjusting weights.

[0079] Weight painting: Many 3D software provide weight painting tools that allow weights to be painted directly on the model in a way similar to painting. These tools usually provide different brushes and strengths for fine control of weight distribution.

[0080] Steps of weight distribution: 1. Initial weight assignment: Weights are assigned to vertices automatically or manually in the bind pose. Automatic weight assignment can be used as a starting point, but usually requires subsequent manual adjustments.

[0081] 2. Test animation: Create simple test animations or poses to see how the weight assignment affects the deformation of the model.

[0082] 3. Adjust weights: Based on the results of the test animation, adjust the vertex weights to ensure that the deformation of the model looks natural and correct. This may involve increasing or decreasing the influence of a specific bone on the vertex.

[0083] 4. Refine and smooth: Use the weight smoothing tool or manual editing to refine the weights, ensuring there is a smooth transition between bones.

[0084] 5. Final Test: Test the weight assignments in more complex animations to ensure that the model deforms satisfactorily in all expected movements and poses.

[0085] Weight assignment is an iterative process that may require multiple adjustments and tests to achieve the best result. It requires patience and careful observation to ensure that every movement of the character is as realistic as possible.

[0086] Weight assignment is a crucial step in 3D modeling and animation, which ensures the natural movement of the model during animation. Here are some key concepts of weight assignment: Vertex weights: Each vertex is assigned a weight for one or more bones (or joints) connected to it. Weight values ​​are usually between 0 and 1, indicating the degree to which the vertex is influenced by the associated bones. A weight of 1 means that the vertex moves completely with the bone, while a weight of 0 means that the vertex is not affected by the bone.

[0087] Weight normalization: All weights of a vertex should add up to 1, which ensures that the vertex does not move beyond a reasonable range. Normalization is to prevent the vertex from scaling or distorting unnaturally when the bone moves.

[0088] Weight smoothing: Weight smoothing refers to the smooth transition of weights between adjacent vertices to avoid sharp or unnatural deformations of the model during animation. Smoothing can help create more realistic muscle and skin movement effects.

[0089] Weight painting: Weight painting is an interactive process that allows you to paint weights directly on your model, usually using different colors to represent different weight values. This process allows for precise control over the weight assignment of each vertex.

[0090] Weight Mirroring: For symmetrical models, weights on one side can be copied and mirrored to the other to save time and maintain consistency.

[0091] Weight testing: During the weight distribution process, you need to constantly test the animation to see the effect of the weight distribution. Move the bones to observe the model's reaction to ensure that the weight distribution achieves the expected natural effect.

[0092] Weight modification: Based on the test results, refine and adjust the weights. This is an iterative process and may require multiple adjustments to achieve the best results.

[0093] Weight distribution is a process that combines technology and art, requiring a deep understanding of movement and anatomy, as well as patience and fine manipulation. Correct weight distribution can greatly improve the quality of animation, making the character's movements look more fluid and realistic.

[0094] Step S102: adding physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world.

[0095] Optionally, adding physical effects to the virtual world during the interaction process includes: Add collision bodies to the motion-captured bones and implement physical interaction through collision detection.

[0096] Optionally, the collision detection includes: defining a collision shape of an object using a collision body; Filter out the collision bodies that need to be detected based on the collision shape, and use the detection algorithm to determine whether the collision occurs in the screening results; If a collision occurs, a collision response is calculated based on the physics information of the colliding surfaces.

[0097] Optionally, the collision detection includes: adding physical constraints at the collision location according to the collision detection result, wherein the physical constraints include: Use the physics engine for geometry offset; Use sweeps for geometry offset; Add stiffness and damping settings based on the geometry offset using the physics engine; And / or bind the relative transform directly to a fixed value without offsetting the geometry.

[0098] Unreal Engine has built-in physics engines such as Chaos and PhysX to simulate physical effects in virtual environments, and provides rich physical properties, including rigid bodies, joints, collision bodies, etc., which can simulate real object movement, collision, friction and other interactive behaviors in virtual environments.

[0099] Based on this, first add collision bodies of various shapes to the skeleton of the motion capture virtual character, and further realize collision and touch type physical interaction through collision detection. For example, when the character's finger touches the touchpad, the collision detection can obtain the contact information on the touchpad, and simulate the touchpad physical interaction behavior according to the normal vector of the contact point and the contact surface and the contact speed. The specific implementation method of collision detection is: a. Each object has one or more collision bodies to define its collision shape. The collision body can be a simple geometric shape such as a sphere, a box, a capsule, or a complex geometric model such as a convex polyhedron. b. When performing collision detection, first use the BVH algorithm to discretely filter the collision bodies that may require fine detection, and then use accurate detection algorithms such as SAT (Separating Axis Theorem) or GJK (Gilbert-Johnson-Keerthi) algorithm to determine whether a collision occurs. c. If a collision occurs, the collision response will be calculated based on the physical material of the collision surface (which can be customized) and the mass, damping and other information of the object. This includes applying force, modifying the object's speed and position, etc. to simulate a real collision effect.

[0100] The BVH algorithm (Bounding Volume Hierarchy) is an algorithm used to accelerate ray tracing and collision detection, and is mainly used to manage objects in 3D scenes. It reduces unnecessary calculations by building a hierarchical bounding box structure, thereby improving the efficiency of rendering and collision detection.

[0101] The basic principle of the BVH algorithm is to divide the objects in the scene into a series of bounding boxes and build a hierarchical structure. Each node contains a bounding box. If the ray does not intersect with the bounding box of a node, the subtree of the node can be skipped directly, thereby reducing the amount of calculation. This structure usually exists in the form of a binary tree, where each internal node holds a pointer to its child node, and each leaf node holds a reference to one or more primitives.

[0102] Application scenarios of BVH algorithm: Ray tracing: In the ray tracing algorithm, BVH is used to accelerate the intersection detection between light and objects in the scene. Through the pre-built BVH structure, it is possible to quickly determine whether the light needs to be further intersected with an object.

[0103] Collision detection: In fields such as physics simulation, game development, and robot navigation, BVH is used to accelerate collision detection between objects and improve computational efficiency.

[0104] Implementation steps of the BVH algorithm: Calculate the boundary information of each primitive: First calculate the minimum bounding box of each object (usually using AABB) and store this information in an array.

[0105] Construct a BVH tree: Use a partitioning algorithm (such as SAH, HLBVH, Middle, EqualCounts) to divide the primitives into disjoint sets and construct a binary tree. Each internal node holds a pointer to its child node, and each leaf node holds a reference to one or more primitives.

[0106] Optimize storage: Convert the tree to a more compact pointer-free representation for use during rendering.

[0107] The SAT (Separating Axis Theorem) algorithm is an algorithm used to detect whether two convex polygons intersect. The core idea is to find an axis so that the projections of the two polygons on this axis do not intersect, thereby determining whether the two polygons intersect. If such an axis cannot be found, it means that the two polygons intersect.

[0108] The basic steps of the SAT algorithm are as follows: Select a separating axis: Select an axis, usually the normal direction of the polygon edge as the separating axis. For each edge of each polygon, calculate its normal direction as a potential separating axis. Projection judgment: Project all vertices of the two polygons onto the selected separating axis. If the projection intervals of the two polygons do not overlap, it means that they do not intersect.

[0109] Traverse all possible separating axes: For each edge of each polygon, calculate its normal direction as the separating axis and check whether the projections overlap. If the projections of all separating axes do not overlap, the two polygons do not intersect.

[0110] The complexity of the SAT algorithm depends mainly on the number of vertices of the polygon. For each edge of each polygon, its normal direction needs to be calculated and a projection operation needs to be performed.

[0111] The SAT algorithm is suitable for collision detection between convex polygons. Since it only applies to convex polygons, for concave polygons, it is necessary to decompose them into multiple convex polygons before detection. In addition, SAT can also be used for collision detection of three-dimensional objects, by determining whether the projections of any two convex polyhedrons at any angle overlap to determine whether a collision occurs.

[0112] The SAT algorithm is an effective collision detection method for convex polygons and is suitable for scenarios that require accurate collision detection.

[0113] The GJK algorithm is an algorithm for collision detection based on the Gilbert-Johnson-Keerthi mathematical model. The algorithm is mainly used to determine whether two or more objects are in contact or penetration, and is widely used in robotics, animation simulation, virtual reality and other fields.

[0114] The basic steps of the GJK algorithm are: Select the initial direction: The initial direction can be a vector formed by the center points of the two polygons, a vector formed by selecting a vertex for each polygon, or a given arbitrary vector.

[0115] Calculate the first support point: Based on the initial direction, use the Support function to get the first support point and put it into the simplex.

[0116] Iterative calculation: The initial direction is reversed and used as the next iteration direction. The loop starts, and new support points are calculated based on the iteration direction and the Support function. If the dot product of the new support point and the iteration direction is less than 0, it means that a support point that can cross the origin cannot be found in this iteration direction, that is, the two polygons do not intersect, and the algorithm exits; otherwise, it is added to the simplex, and the simplex and iteration direction are updated.

[0117] Determine whether the simplex contains the origin: If the simplex has two points, the two points form a straight line, and the direction of the perpendicular line of the straight line toward the origin is used as the direction of the next iteration; if there are three points, it is necessary to determine whether the triangle formed by the three points contains the origin. If not, keep the two points on the edge closest to the origin, and use the direction of the perpendicular line of the two points toward the origin as the direction of the next iteration.

[0118] Furthermore, when the specific collision body of the character's hand interacts with the object for a long time, various forms of physical constraints (Physics Constraint) are added between the specific bone joints of the character and the object according to the collision detection results. This allows the user to achieve highly realistic physical simulation grasping according to the hand movements output by motion capture. The types of physical constraints designed in this embodiment include: a. Using the physical engine to offset the geometry: the most commonly used, suitable for objects of normal size and mass. Used to simulate real physical interaction effects, such as hammers, building blocks, etc. b. Using scanning to offset the geometry and push the physically simulated object: This solution does not use the weight of the object. It is suitable for objects with large or small weights, such as picking up large objects with both hands. Because the object is too heavy, it cannot be moved by the intelligent agent Baby, so the weight is set to infinity. c. Adding stiffness and damping settings on the basis of using the physical engine to offset the geometry can simulate some movements with large damping and the need to reduce swinging, such as holding chains, springs, rubber bands, etc. d. Directly bind the relative Transform to a fixed value without offsetting the geometry: some imaginary captures without entities in the scene, such as user interaction illuminants (UI) and static projections.

[0119] like Figure 3As shown, this embodiment also discloses a virtual reality interaction device, including: A fusion module for fusing the motion-captured skeletal posture with the skeletal posture of the virtual world animation system when interacting with the virtual world; The physical simulation module is used to add physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world.

[0120] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0121] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the virtual reality interaction method of each embodiment of the present disclosure.

[0122] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0123] like Figure 4 The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 4 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0125] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the virtual reality interaction method of the embodiment of the present disclosure are executed.

[0127] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0128] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the virtual reality interaction method of each embodiment of the present disclosure are executed.

[0129] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0130] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0131] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0132] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0133] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0134] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0135] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0136] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0137] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A virtual reality interaction method, characterized in that: include: When interacting with the virtual world, the motion-captured skeletal poses are merged with the skeletal poses of the virtual world animation system; Add physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world.

2. The virtual reality interaction method according to claim 1, characterized in that: The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: The center point of the skeleton is obtained according to the motion capture skeleton posture, the change value of the center point in each frame is mapped to the root skeleton point of the animation system, and the change value is subtracted from the relative motion of the root node of the motion capture.

3. The virtual reality interaction method according to claim 2, characterized in that: The method of mapping the change value of each frame of the center point to the root bone point of the animation system and subtracting the change value from the relative motion of the motion-captured bone pose includes: Map the change value of the center point in each frame to the X-axis and Y-axis of the root bone point of the animation system, and subtract the change value on the X-axis and Y-axis from the relative motion of the pelvis captured by motion capture.

4. The virtual reality interaction method according to claim 2, characterized in that: The step of obtaining the center point of the skeleton according to the skeleton posture captured by motion includes: The midpoints of the paired bone points in the motion captured skeleton pose are projected onto the ground and then the center points of the bones are obtained by counting.

5. The virtual reality interaction method according to claim 2, characterized in that: The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: The animation weight of the bone posture change is obtained according to the transition sequence of the interaction mode and the preset curve. The interpolation method is used based on the animation weight to fuse the motion captured bone posture with the bone posture of the virtual world animation system to obtain the fused bone posture.

6. The virtual reality interaction method according to claim 2, characterized in that: The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: Get the bone chain that needs to be fused according to the interaction mode.

7. The virtual reality interaction method according to claim 6, characterized in that: The skeleton chain includes a left arm skeleton chain, a right arm skeleton chain, a spine skeleton chain, a head skeleton chain, a left leg skeleton chain and a right leg skeleton chain.

8. The virtual reality interaction method according to claim 1, characterized in that: Adding physical effects to the virtual world during the interaction process includes: Add collision bodies to the motion-captured bones and implement physical interaction through collision detection.

9. The virtual reality interaction method according to claim 8, characterized in that: The collision detection comprises: Use colliders to define the collision shape of objects; Filter out the collision bodies that need to be detected based on the collision shape, and use the detection algorithm to determine whether the collision occurs in the screening results; If a collision occurs, a collision response is calculated based on the physics information of the colliding surfaces.

10. The virtual reality interaction method according to claim 8, characterized in that: Collision detection, including: Add physical constraints at the collision point according to the collision detection results. The physical constraints include: Use the physics engine for geometry offset; Use sweeps for geometry offset; Add stiffness and damping settings based on the geometry offset using the physics engine; and / or Directly bind the relative transformation to a fixed value without offsetting the geometry.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual reality interaction method described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the virtual reality interaction method described in any one of claims 1-10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the virtual reality interaction method described in any one of claims 1-10 is implemented.

Images