Virtual reality interaction method, electronic device, storage medium and program product

Abstract

The present invention provides a virtual reality interaction method, electronic device, storage medium and program product. The virtual reality interaction method includes: when interacting with a virtual world, fusing the motion-captured skeletal posture with the skeletal posture of a virtual world animation system; adding physical effects to the virtual world during the interaction process, thereby simulating real interactive behavior in the virtual world. By fusing the motion-captured skeletal posture with the skeletal posture of the virtual world animation system, the operator's motion space is expanded to an infinite virtual world, and by adding physical effects to the virtual world, the authenticity of the interaction is enhanced, thereby achieving the purpose of improving the realism of user interaction and expanding the action mapping space.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a virtual reality interaction method, electronic equipment, storage medium and program product. Background Art

[0002] Motion capture products are diverse in the market, with two main application scenarios: collecting sensor information and generating skeletal animation data offline for use in 3D games or film and television; and projecting real-time sensor information onto skeletons for live broadcasts. However, few products directly enhance the operator's immersive experience by integrating XR (Extended Reality) technology. Therefore, motion capture operators are unable to generate realistic physical interactions with the virtual world based on their movements. While implementing this embodiment, the inventors discovered the following problems with the existing technology:

[0003] 1. Virtual characters cannot produce physically realistic interactive effects in the virtual world, which limits the realism of the user experience.

[0004] 2. The activity range in the virtual world is almost unlimited, but the operator's movement space in reality is limited. There is a contradiction between the two, which leads to limited motion mapping.

[0005] 3. The current focus of motion capture is on the accuracy of restoring action postures, without considering accurately mapping real-world non-physical interactive actions to the virtual world while ensuring the realism of interactive behaviors. Summary of the Invention

[0006] In view of the problems existing in the prior art, the present invention provides a virtual reality interaction method, electronic device, storage medium and program product, which at least partially solves the problems of limited realism and mapping in the prior art.

[0007] In a first aspect, embodiments of this disclosure provide a virtual reality interaction method, including:

[0008] When interacting with the virtual world, the skeletal poses from motion capture are fused with the skeletal poses from the virtual world's animation system;

[0009] Physical effects are added to the virtual world during the interaction process, thereby simulating real-world interactive behavior in the virtual world.

[0010] Optionally, the fusion of motion-captured skeletal poses with the skeletal poses of the virtual world animation system includes:

[0011] The center point of the bone is obtained from the motion capture bone pose. The change value of the center point in each frame is mapped to the root bone point of the animation system, and the change value is subtracted from the relative motion of the root node of the motion capture.

[0012] Optionally, mapping the change value of the center point in each frame to the root bone point of the animation system and subtracting the change value from the relative motion of the motion-captured skeletal pose includes:

[0013] Map the change value of the center point in each frame to the X-axis and Y-axis of the root bone point of the animation system, and subtract the change value on the X-axis and Y-axis from the relative motion of the pelvis captured by motion capture.

[0014] Optionally, obtaining the center point of the skeleton according to the motion-captured skeleton posture includes:

[0015] The center point of the skeleton is obtained by projecting the midpoints of paired bone points in the motion capture skeleton pose onto the ground.

[0016] Optionally, the fusion of motion-captured skeletal poses with the skeletal poses of the virtual world animation system includes:

[0017] The animation weight of the skeletal posture change is obtained according to the transition sequence of the interaction mode and the preset curve. Based on the animation weight, the skeletal posture of the motion capture is fused with the skeletal posture of the virtual world animation system using the interpolation method to obtain the fused skeletal posture.

[0018] Optionally, the fusion of motion-captured skeletal poses with the skeletal poses of the virtual world animation system includes:

[0019] Obtain the skeletal chains that need to be merged based on the interaction mode.

[0020] Optionally, the skeleton chain includes a left arm skeleton chain, a right arm skeleton chain, a spine skeleton chain, a head skeleton chain, a left leg skeleton chain and a right leg skeleton chain.

[0021] Optionally, adding physical effects to the virtual world during the interaction includes:

[0022] Add collision bodies to the motion capture skeleton and implement physical interaction through collision detection.

[0023] Optionally, the collision detection includes:

[0024] Use colliders to define the collision shape of an object;

[0025] Based on the collision shape, the colliders to be detected are selected, and the detection algorithm is used to determine whether a collision has occurred in the selected results.

[0026] If a collision occurs, the collision response is calculated based on the physical information of the colliding surfaces.

[0027] Optional collision detection includes:

[0028] Add physical constraints at the collision point based on the collision detection results. The physical constraints include:

[0029] Use the physics engine for geometry offset;

[0030] Use sweeps for geometry offset;

[0031] Add stiffness and damping settings based on the geometry offset using the physics engine;

[0032] and / or

[0033] The relative transformation is directly bound to a constant value, without geometric offset.

[0034] Secondly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0035] at least one processor; and,

[0036] a memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform any of the virtual reality interaction methods described in the first aspect.

[0038] Thirdly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to perform any of the virtual reality interaction methods described in the first aspect.

[0039] Fourthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the virtual reality interaction method described in any of the first aspects.

[0040] The present invention provides a virtual reality interaction method, electronic device, storage medium, and program product. The virtual reality interaction method integrates the skeletal posture of motion capture with the skeletal posture of a virtual world animation system, thereby expanding the operator's movement space into an infinite virtual world. By adding physical effects to the virtual world, the realism of the interaction is enhanced, thereby improving the realism of user interaction and expanding the motion mapping space. Attached Figure Description

[0041] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0042] Figure 1 A framework diagram of a virtual reality interaction method provided in an embodiment of the present disclosure;

[0043] Figure 2 A flowchart of a virtual reality interaction method provided in an embodiment of the present disclosure;

[0044] Figure 3 A schematic block diagram of a virtual reality interaction device provided in an embodiment of this disclosure;

[0045] Figure 4 This is a functional block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0047] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0048] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0049] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0050] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0051] Motion capture refers to the process of transcribing the movements of live actors into digital models in 3D games or animations using sensors and software.

[0052] For ease of understanding, Figure 1 and Figure 2 As shown, this embodiment discloses a virtual reality interaction method, including:

[0053] Step S101: When interacting with the virtual world, the skeletal pose captured by motion capture is fused with the skeletal pose of the virtual world animation system;

[0054] Optionally, the fusion of the motion capture skeletal pose with the virtual world animation system includes: obtaining the center point of the bone based on the motion capture skeletal pose, mapping the change value of the center point in each frame to the root bone point of the animation system, and subtracting the change value from the relative motion of the root node of the motion capture.

[0055] Optionally, mapping the change value of the center point for each frame to the root bone point of the animation system, and subtracting the change value from the relative motion of the motion-captured bone pose, includes:

[0056] The change value of the center point in each frame is mapped to the X and Y axes of the root bone point of the animation system, and the change values ​​on the X and Y axes are subtracted from the relative motion of the pelvis captured by motion capture.

[0057] Optionally, obtaining the center point of the skeleton based on the motion capture skeleton pose includes:

[0058] The center point of the skeleton is obtained by projecting the midpoints of paired bone points in the motion capture skeleton pose onto the ground.

[0059] Motion pose fusion: Regardless of the motion capture technology used, the range of movement for a motion capture operator in the real world is a finite space, which cannot be directly correlated with the near-infinite virtual world. Meanwhile, similar to character animation in games, skeletal animation of virtual character movement or complex interactions can be obtained through pre-set animation sequences and animation state machines. This embodiment designs a motion pose fusion method based on the characteristics of motion-captured characters, mixing the skeletal poses of the animation system and the motion capture system, allowing the virtual human to move and interact freely in the virtual world.

[0060] The output of motion capture technology is generally described as the relative transformations (including position and rotation transformations) between the joints of a complete skeletal chain. In industry skeletal standards, the root node of a human skeleton is the pelvis, so the overall motion information of a motion-captured virtual human is described by the transformation matrix of the pelvis. In general animation systems, the overall motion information of a virtual human is described by the movement of the root bone points along the X and Y axes of the world coordinate system. The pelvis joints of a virtual human are generally used in animation systems to describe the changes in motion information of the human hips. Therefore, before motion fusion, the motion information of the two systems needs to be matched: first, the center point of the human is calculated based on the distribution of the human's skeletal points. The center point is obtained by projecting the midpoints of paired bone points onto the ground and then statistically analyzing the changes in the center point for each frame. The changes in the center point are then mapped to the X and Y axes of the root bone points of the human, and the changes in the X and Y axes are subtracted from the relative motion of the pelvis.

[0061] The relative transformation of each skeletal joint can be represented by a transformation matrix. , including position and translation:

[0062] ,

[0063] Calculate the position change of the center point in each frame:

[0064] ,

[0065] in, .

[0066] Map the changes in the center point to the X and Y axes of the root bone point:

[0067] ,

[0068] The new position of the root bone point is:

[0069] ,

[0070] Subtract the changes in the center point along the X and Y axes from the relative motion of the pelvis to eliminate the effect of global displacement on the pelvis:

[0071] ,

[0072] in is The translation matrix is:

[0073] ,

[0074] It is the inverse matrix, which is obtained by translating and inverting:

[0075] .

[0076] Optionally, the process of fusing the motion-captured skeletal pose with the skeletal pose of the virtual world animation system includes: obtaining the animation weights of the skeletal pose changes based on the transition sequence of the interaction mode and a preset curve, and using an interpolation method based on the animation weights to fuse the motion-captured skeletal pose with the skeletal pose of the virtual world animation system to obtain the fused skeletal pose.

[0077] Optionally, the fusion of motion-captured skeletal poses with the skeletal poses of the virtual world animation system includes: obtaining the skeletal chains to be fused based on the interaction mode.

[0078] Optionally, the skeletal chains include a left arm skeletal chain, a right arm skeletal chain, a spinal skeletal chain, a head skeletal chain, a left leg skeletal chain, and a right leg skeletal chain.

[0079] After the motion information from the motion capture and general animation systems is matched, motion fusion begins. The motion fusion calculations are split into multiple local skeletal chains. The skeletal chains designed during verification for this embodiment are divided into the following: left arm skeletal chain, right arm skeletal chain, spine skeletal chain, head skeletal chain, left leg skeletal chain, and right leg skeletal chain. The skeletal structure is a tree-like structure, starting from the root node. Specifically, the pelvis is the root node and parent node, and the leg bones and other child nodes are descended from the root node, layer by layer. The skeletal chains to be fused are determined based on the current interaction mode. During the fusion process, the moving parts are not fused with the virtual world animation. The moving parts are controlled by the motion capture, and the non-moving parts are fused with the virtual world animation. For example, when writing, the right arm skeletal chain moves in the virtual world based on the motion capture signal, while the other skeletal chains are fused with the virtual world animation. The interaction mode supports user customization. For example, when the motion capture subject squats, the interaction mode is such that the lower body moves first, then drives the upper body movement. Next, the weight change trend of different bone poses is specified according to the transition order of the interaction mode. The weight change trend is determined according to the provided preset curve. After the weights of each animation are determined, the two animations are mixed using classic interpolation methods such as linear interpolation, spherical linear interpolation, and Bézier curve interpolation to obtain the final bone pose result.

[0080] Bézier curves, also known as Bézier curves or Bézier curves, are mathematical curves used in two-dimensional graphics applications. General vector graphics software uses them to accurately draw curves. A Bézier curve consists of line segments and nodes; nodes are draggable pivot points, and line segments are like stretchable rubber bands.

[0081] The weight function is:

[0082]

[0083] Position Interpolation (LERP):

[0084]

[0085] Rotational interpolation (SLERP):

[0086]

[0087] Bezier interpolation:

[0088] ,

[0089] in, , , Bezier curve control points.

[0090] Skeletal animation is a technique in computer graphics used to animate the movement of characters or objects. Skeletal animation manipulates the character model using an internal skeleton structure, rather than manipulating each vertex of the model directly. The following are some key implementation details of skeletal animation:

[0091] Skeleton: Typically composed of a series of bones or joints connected in a hierarchical structure to form an internal framework, similar to the skeleton of a real-world creature.

[0092] Joints: Each joint (or bone node) has its own transformation (including position, rotation and scale), which can be relative to the parent joint.

[0093] Skin Binding: Vertex Weights: Each vertex of the model is assigned one or more joints, and each joint has a weight value that indicates how much influence the joint has on the vertex position.

[0094] Bind Pose: During the binding process, the initial shape of the model (the bind pose) is recorded, usually in a T-pose.

[0095] Animation Keyframes: Create keyframes that define the transformation of a joint at a specific point in time. Interpolation: As the animation plays, interpolation (usually linear or nonlinear) is performed between the keyframes to produce smooth motion. Real-time Transformations: Bone Transformations: As the animation plays, the transformation of each joint is calculated in real time based on the key animation. Vertex Transformations: The final position of each vertex is then calculated based on the joint's transformation and the vertex weights. Normal Transformations: In order to render lighting correctly, the vertex normals also need to be updated based on the bone transformations.

[0096] Vertex shader: Modern graphics use vertex shaders on the GPU to accelerate the calculation of skeletal animation. The vertex shader can process each vertex in parallel, greatly improving efficiency.

[0097] Animation blending: In practical applications, it may be necessary to play multiple animations simultaneously (e.g., shooting from the upper body while walking). In this case, the system will perform weighted blending of the joint transformations of multiple animations to generate the final animation.

[0098] Inverse Kinematics: In some cases, it is used to manipulate models more intuitively. It allows you to specify the position of end effectors (such as hands or feet), and then automatically calculates the necessary joint angles.

[0099] The implementation of skeletal animation involves complex mathematics and programming techniques.

[0100] In skeletal animation, the skeletal structure (also known as rigid body animation or character skeleton) refers to a hierarchical set of bones or joints that form the internal framework of a character or object. This structure is crucial in the animation process because it defines how the character moves.

[0101] Here are some key features of bone structure:

[0102] Hierarchy, Root Bone: Usually located at the top of the bone hierarchy, it controls the global position and orientation of the entire bone structure.

[0103] Parent bones and child bones: Each bone (except the root bone) has a parent bone and may have multiple child bones. The transformation of a child bone is relative to the transformation of its parent bone.

[0104] Skeletal Properties: Joints: Each node in the skeletal structure, which can be understood as a joint, allows rotation and sometimes translation. Bones: The parts that connect two joints, which can be considered as bones, defining the distance and orientation between the joints.

[0105] Animation Control: Transformation: Each bone has its own transformation, including translation, rotation and scale. These transformations can be controlled by keyframe animation.

[0106] Inverse Kinematics: In some cases, it may be used to more intuitively manipulate skeletons, especially when precise control of the position of end effectors (such as hands or feet) is required.

[0107] Model-associated weighted skinning: Each model vertex is assigned weights from one or more bones, which determines the degree to which bone transformations affect vertex positions. Pose binding: The initial pose when the model and skeletal structure are bound together, typically a neutral pose, such as a T-pose.

[0108] Animation data: Set keyframes to define the changes of the skeleton at specific points in time.

[0109] Animation curves: These curves describe the details of how time changes with the bone transformation and are used for interpolation between keyframes.

[0110] The design of the skeletal structure is crucial to the quality of animation. A good skeletal structure should be able to simulate real-world biological movement while remaining sufficiently simplified to allow for efficient computer control and animation. In practice, the complexity of the skeletal structure varies depending on the character's needs and the level of detail required for the animation.

[0111] Skin rigging, in computer graphics, especially in skeletal animation, refers to the process of associating the surface (skin) of a 3D model with its skeletal structure. This process ensures that the model's surface deforms accordingly as the skeleton moves. The key to skin rigging is ensuring that the model's movement looks natural and realistic. Here are some of the main components and steps of skin rigging:

[0112] Skin binding pose: Before skin binding begins, the model needs to be in a binding pose, usually a neutral pose, such as a T-shape or A-shape, to maximize the range of motion of the joints.

[0113] Weighting: Vertex weights: Each model vertex is assigned the weights of one or more bones, and these weights determine the degree to which the bone movements affect the vertex position.

[0114] Weight normalization: Typically, the sum of all weights of a vertex should equal 1, thus maintaining the consistency of the model volume.

[0115] Linear blending skinning, also known as vertex blending, is the most common skinning technique, in which the final position of a vertex is calculated by a weighted average of all bone transformations that affect it.

[0116] Double quaternion skins can better handle deformation caused by rotation and reduce the candy wrapper effect.

[0117] Skinning process: Initial binding: In the binding pose, each vertex of the model is associated with one or more bones and weights are assigned.

[0118] Real-time deformation: During animation playback, the new position of each vertex is calculated in real time based on the current transformation of the bones and vertex weights.

[0119] Testing and Adjustment: Test the binding effect by checking for any unnatural distortions through various poses in the animation. Adjust the weights as needed to ensure the animation is visually smooth and realistic.

[0120] Attached Pose: A attached pose is a reference pose when a 3D model is attached to its skeletal system. This pose is typically a standardized standing posture for the character, such as a T-pose, where the character's arms are outstretched horizontally, legs are straight, and the face is forward. Such a pose provides a clear starting point for observing and editing bone weights and ensuring smooth transitions between different movements.

[0121] In the bind pose, each bone is in its default rotation and position. This defines the initial assignment of vertex weights during skinning.

[0122] Once a model is bound to a skeleton, all animation transforms are calculated relative to the bound pose.

[0123] Choosing a bind pose is crucial for subsequent animation work because it affects the effectiveness of weight distribution and the naturalness of the animation. For example, if a character's arms are hanging down in the bind pose, then raising the arms during animation may cause the skin to stretch unnaturally. Therefore, the T-pose is often chosen because it allows unrestricted movement of the arms and legs and makes it easier to distribute weights for more natural movement in animation.

[0124] In the actual animation process, once the rigging pose is determined and the model is rigged to the skeleton, it is best to avoid changing the rigging pose, as this will affect the already set weight distribution and may require a lot of readjustment work.

[0125] Weight assignment: Weight assignment, also known as weight skinning or vertex weighting, is a key step in the skinning process for 3D models. It involves associating each vertex of the model with one or more bones and assigning a weight value. This value determines how much the vertex follows the movement of the bone. This process ensures that the model's skin deforms in a natural way when the skeleton animation plays.

[0126] Key concepts in weight allocation: Vertex weights: Each vertex can be assigned to one or more bones, and each bone has a weight value for that vertex. Weight values ​​are typically between 0 and 1, where 0 means the vertex is not affected by the bones, and 1 means the vertex is fully affected by the bones.

[0127] Weight normalization: The sum of all the weights of a vertex should generally equal 1. That is, if a vertex is affected by multiple bones, the sum of the influences of those bones on the vertex should be the complete influence.

[0128] Weight smoothing: To avoid sharp or unnatural deformations in the model during animation, weight distribution should transition smoothly between bones. This is typically achieved using weight smoothing tools or by manually adjusting the weights.

[0129] Weight painting: Many 3D software provide weight painting tools that allow you to paint weights directly on the model in a way similar to painting. These tools usually provide different brushes and intensities for fine control of weight distribution.

[0130] The steps of weight allocation:

[0131] 1. Initial Weight Assignment: Under the bound pose, weights are automatically or manually assigned to vertices. Automatic weight assignment can serve as a starting point, but usually requires subsequent manual adjustments.

[0132] 2. Test Animations: Create simple test animations or poses to see how weight distribution affects the model's deformation.

[0133] 3. Adjust weights: Based on the results of the test animation, adjust the vertex weights to ensure that the model's deformation looks natural and correct. This may involve increasing or decreasing the influence of specific bones on the vertices.

[0134] 4. Refine and smooth: Use the weight smoothing tool or manual editing to refine the weights and ensure a smooth transition between bones.

[0135] 5. Final testing: Test the weight distribution in more complex animations to ensure that the model's deformation is satisfactory in all expected movements and poses.

[0136] Weight allocation is an iterative process that may require multiple adjustments and tests to achieve optimal results. Patience and meticulous observation are essential to ensure that every movement of the character is as realistic as possible.

[0137] Weighting is a crucial step in 3D modeling and animation, ensuring the model moves naturally during animation. Here are some key concepts related to weighting:

[0138] Vertex Weights: Each vertex is assigned a weight associated with one or more bones (or joints) connected to it. Weight values ​​are typically between 0 and 1, indicating the degree to which the vertex is influenced by the associated bone. A weight of 1 means the vertex moves completely with the bone, while a weight of 0 means the vertex is not affected by the bone at all.

[0139] Weight normalization: All weights of a vertex should add up to 1, which ensures that the vertex does not move beyond a reasonable range. Normalization is to prevent the vertex from scaling or distorting unnaturally when the bone moves.

[0140] Weight smoothing: Weight smoothing refers to the smooth transition of weights between adjacent vertices to avoid sharp or unnatural deformation of the model during animation. Smoothing can help create more realistic muscle and skin movement effects.

[0141] Weight painting: Weight painting is an interactive process that allows you to paint weights directly on your model, usually using different colors to represent different weight values. This process allows you to precisely control the weight distribution of each vertex.

[0142] Weight mirroring: For symmetrical models, the weights on one side can be copied and mirrored to the other side to save time and maintain consistency.

[0143] Weight testing: During the weight distribution process, you need to constantly test the animation to see the effect of the weight distribution. By moving the bones and observing the model's reaction, you can ensure that the weight distribution achieves the expected natural effect.

[0144] Weight adjustment: Based on the test results, the weights are refined and adjusted. This is an iterative process and may require multiple adjustments to achieve the best results.

[0145] Weight distribution is a process that combines both technology and art, requiring a deep understanding of movement and anatomy, as well as patience and finesse. Correct weight distribution can greatly improve the quality of animation, making the character's movements appear more fluid and realistic.

[0146] Step S102: Add physical effects to the virtual world during the interaction process to simulate real interactive behavior in the virtual world.

[0147] Optionally, adding physical effects to the virtual world during the interaction includes:

[0148] Add collision bodies to the motion capture skeleton and implement physical interaction through collision detection.

[0149] Optionally, the collision detection includes: defining the collision shape of the object using a collider;

[0150] Based on the collision shape, the colliders to be detected are selected, and the detection algorithm is used to determine whether a collision has occurred in the selected results.

[0151] If a collision occurs, the collision response is calculated based on the physical information of the colliding surfaces.

[0152] Optionally, collision detection includes: adding physical constraints at the collision site based on the collision detection results, wherein the physical constraints include:

[0153] Use the physics engine for geometry offset;

[0154] Use sweeps for geometry offset;

[0155] Add stiffness and damping settings based on the geometry offset using the physics engine;

[0156] And / or directly bind the relative transformation to a constant value, without performing geometric offset.

[0157] Unreal Engine has built-in physics engines such as Chaos and PhysX to simulate physical effects in virtual environments. It provides rich physical properties, including rigid bodies, joints, collision bodies, etc., which can simulate real object movement, collisions, friction and other interactive behaviors in virtual environments.

[0158] Based on this, various colliders of different shapes are first added to the motion-captured virtual character skeleton. Then, collision detection is used to implement physical interactions such as collisions and touches. For example, when a character's finger touches a touchpad, collision detection can obtain contact information on the touchpad and simulate the physical interaction behavior based on the contact point, the normal vector of the contact surface, and the contact velocity. The specific implementation of collision detection is as follows: a. Each object has one or more colliders to define its collision shape. Colliders can be simple geometric shapes such as spheres, boxes, and capsules, or complex geometric models such as convex polyhedra. b. During collision detection, the BVH algorithm is first used to discretely filter colliders that may require fine-grained detection. Then, precise detection algorithms such as SAT (Separating Axis Theorem) or GJK (Gilbert-Johnson-Keerthi) are used to determine whether a collision has occurred. c. If a collision occurs, the collision response is calculated based on the physical material of the collision surface (which can be customized) and the object's mass, damping, and other information. This includes applying forces, modifying the object's velocity and position, etc., to simulate realistic collision effects.

[0159] The BVH algorithm (Bounding Volume Hierarchy) is an algorithm used to accelerate ray tracing and collision detection, primarily for managing objects in 3D scenes. It reduces unnecessary calculations by constructing a hierarchical bounding box structure, thereby improving rendering and collision detection efficiency.

[0160] The basic principle of the BVH algorithm is to divide objects in a scene into a series of bounding boxes and construct a hierarchical structure. Each node contains a bounding box, and if a ray does not intersect with the bounding box of a node, the subtree of that node can be skipped directly, thus reducing computation. This structure is usually in the form of a binary tree, where each internal node holds pointers to its child nodes, and each leaf node holds references to one or more primitives.

[0161] Application scenarios of the BVH algorithm: Ray tracing: In ray tracing algorithms, BVH is used to accelerate the intersection detection between light rays and objects in the scene. The pre-built BVH structure can quickly determine whether a ray needs to further intersect with a specific object.

[0162] Collision detection: In fields such as physics simulation, game development, and robot navigation, BVH is used to accelerate collision detection between objects and improve computational efficiency.

[0163] Implementation steps of the BVH algorithm: Calculate the boundary information of each primitive: First calculate the minimum bounding box of each object (usually using AABB) and store this information in an array.

[0164] Constructing a BVH tree: Use a partitioning algorithm (such as SAH, HLBVH, Middle, EqualCounts) to divide the primitives into disjoint sets and construct a binary tree. Each internal node holds a pointer to its child nodes, and each leaf node holds a reference to one or more primitives.

[0165] Optimize storage: Convert the tree to a more compact pointer-free representation for use during rendering.

[0166] The Separating Axis Theorem (SAT) algorithm is used to determine whether two convex polygons intersect. The core idea is to determine whether two polygons intersect by finding an axis such that the projections of the two polygons on this axis do not intersect. If no such axis can be found, the two polygons intersect.

[0167] The basic steps of the SAT algorithm are as follows: Select a separating axis: An axis, typically the normal direction of a polygon edge, is chosen as the separating axis. For each edge of each polygon, its normal direction is calculated as a potential separating axis. Projection: All vertices of the two polygons are projected onto the selected separating axis. If the projected intervals of the two polygons do not overlap, they are considered disjoint.

[0168] Traverse all possible separating axes: For each edge of each polygon, calculate its normal direction as the separating axis and check whether the projections overlap. If the projections of all separating axes do not overlap, the two polygons do not intersect.

[0169] The complexity of the SAT algorithm depends mainly on the number of vertices in the polygon. For each edge of each polygon, its normal direction needs to be calculated and a projection operation needs to be performed.

[0170] The SAT algorithm is suitable for collision detection between convex polygons. Since it only works with convex polygons, concave polygons must first be decomposed into multiple convex polygons before detection. Furthermore, SAT can also be used for collision detection between three-dimensional objects, determining whether a collision has occurred by determining whether the projections of any two convex polyhedrons overlap at any angle.

[0171] The SAT algorithm is an effective collision detection method for convex polygons and is suitable for scenarios that require accurate collision detection.

[0172] The GJK algorithm is a collision detection algorithm based on the Gilbert-Johnson-Keerthi mathematical model. This algorithm is primarily used to determine whether two or more objects are in contact or have penetrated each other, and it is widely used in robotics, animation simulation, and virtual reality.

[0173] The basic steps of the GJK algorithm are: Selecting the initial direction: The initial direction can be a vector formed by the center points of the two polygons, a vector formed by selecting one vertex from each polygon, or any given vector.

[0174] Calculate the first support point: Based on the initial direction, use the Support function to get the first support point and place it in the simplex.

[0175] Iterative calculation: The initial direction is reversed and used as the direction for the next iteration. The loop begins, and a new support point is calculated based on the iteration direction and the Support function. If the dot product of the new support point and the iteration direction is less than 0, it means that no support point can be found that crosses the origin in this iteration direction, i.e., the two polygons do not intersect, and the algorithm exits; otherwise, it is added to the simplex, and the simplex and iteration direction are updated.

[0176] To determine if the simplex contains the origin: If the simplex has two points, these two points form a straight line, and the direction of the perpendicular line from this line towards the origin is used as the direction of the next iteration. If there are three points, it is necessary to determine whether the triangle formed by these three points contains the origin. If it does not, retain the two points on the side closest to the origin, and the direction of the perpendicular line from these two points towards the origin is used as the direction of the next iteration.

[0177] Furthermore, after a prolonged interaction between a specific collider on the user's hand and an object, various forms of physical constraints are added between the user's specific skeletal joints and the object based on the collision detection results. This allows the user to achieve highly realistic physical simulation of grasping based on the hand movements output by motion capture. The types of physical constraints designed in this embodiment include: a. Using a physics engine for geometric offset: This is the most common method and is suitable for normal-sized, massy objects. It is used to simulate realistic physical interaction effects, such as hammers, building blocks, etc. b. Using scanning for geometric offset to push physically simulated objects: This method does not use object weight. It is suitable for objects with large or small weights, such as picking up a large object with both hands. Because the object is too heavy to be moved by the intelligent agent Baby, the weight is set to infinity. c. Adding stiffness and damping settings to the geometric offset using a physics engine can simulate movements with high damping that require reduced swaying, such as holding a chain, spring, rubber band, etc. d. Directly bind the relative transform to a fixed value without offsetting the geometry: some imaginary captures without entities in the scene, such as user interaction lights (UI) and static projections.

[0178] like Figure 3 As shown, this embodiment also discloses a virtual reality interaction device, including:

[0179] The fusion module is used to fuse the skeletal poses of motion capture with the skeletal poses of the virtual world animation system when interacting with the virtual world.

[0180] The physics simulation module is used to add physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world.

[0181] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0182] The processor can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the virtual reality interaction method described in the various embodiments of the present disclosure.

[0183] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0184] like Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0185] like Figure 4 As shown, an electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0186] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 4 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0187] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the virtual reality interaction method of embodiments of this disclosure are performed.

[0188] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0189] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the virtual reality interaction methods of the foregoing embodiments of this disclosure are performed.

[0190] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0191] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0192] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0193] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0194] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0195] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0196] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A virtual reality interaction method, characterized in that: include: When interacting with the virtual world, the motion-captured skeletal poses are merged with the skeletal poses of the virtual world animation system; Add physical effects to the virtual world during the interaction process, thereby simulating real interactive behaviors in the virtual world; The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: Get the center point of the skeleton according to the motion capture skeleton posture, map the change value of the center point in each frame to the root skeleton point of the animation system, and subtract the change value from the relative motion of the root node of the motion capture; The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: The animation weight of the bone posture change is obtained according to the transition sequence of the interaction mode and the preset curve. The interpolation method is used based on the animation weight to fuse the motion captured bone posture with the bone posture of the virtual world animation system to obtain the fused bone posture.

2. The virtual reality interaction method according to claim 1, characterized in that: The method of mapping the change value of each frame of the center point to the root bone point of the animation system and subtracting the change value from the relative motion of the motion-captured bone pose includes: Map the change value of the center point in each frame to the X-axis and Y-axis of the root bone point of the animation system, and subtract the change value on the X-axis and Y-axis from the relative motion of the pelvis captured by motion capture.

3. The virtual reality interaction method according to claim 1, characterized in that: The step of obtaining the center point of the skeleton according to the skeleton posture captured by motion includes: The midpoints of the paired bone points in the motion captured skeleton pose are projected onto the ground and then the center points of the bones are obtained by counting.

4. The virtual reality interaction method according to claim 1, characterized in that: The fusion of the motion captured skeleton posture with the skeleton posture of the virtual world animation system includes: Get the bone chain that needs to be fused according to the interaction mode.

5. The virtual reality interaction method according to claim 4, characterized in that: The skeleton chain includes a left arm skeleton chain, a right arm skeleton chain, a spine skeleton chain, a head skeleton chain, a left leg skeleton chain and a right leg skeleton chain.

6. The virtual reality interaction method according to claim 1, characterized in that: Adding physical effects to the virtual world during the interaction process includes: Add collision bodies to the motion-captured bones and implement physical interaction through collision detection.

7. The virtual reality interaction method according to claim 6, characterized in that: The collision detection comprises: Use collision bodies to define the collision shape of objects; Filter out the collision bodies that need to be detected based on the collision shape, and use the detection algorithm to determine whether the collision occurs in the screening results; If a collision occurs, a collision response is calculated based on the physics information of the colliding surfaces.

8. The virtual reality interaction method according to claim 6, characterized in that: Collision detection, including: Add physical constraints at the collision point according to the collision detection results. The physical constraints include: Use the physics engine for geometry offset; Use sweeps for geometry offset; Add stiffness and damping settings based on the geometry offset using the physics engine; and / or Directly bind the relative transformation to a fixed value without offsetting the geometry.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual reality interaction method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the virtual reality interaction method described in any one of claims 1-8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the virtual reality interaction method described in any one of claims 1 to 8 is implemented.

Images