Virtual object model generation method and virtual object model generation device
By acquiring and processing the standard pose information and pre-calibration information of the target object, and combining the motion capture algorithm to process the standard skeleton model, the pose accuracy and real-time problems caused by insufficient sensors in the prior art are solved, and efficient motion capture in XR scenarios is achieved, improving the immersive experience.
Patent Information
- Application Number
- CN202311516509.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-14
- Publication Date
- 2025-05-16
AI Technical Summary
The existing motion capture technology needs to bind more motion capture sensors to the operator, which affects the accuracy and real-time nature of the bone posture, and is not suitable for scenes such as extended reality where more sensors cannot be worn, affecting the immersive experience.
By obtaining the standard pose information of multiple joints of the target object when performing the target standard action, precalibration information is determined based on the corresponding information of the standard pose information and the standard skeleton model, and combining the motion capture algorithm to process the standard skeleton model of the target object, the target skeleton model of the target object is constructed, so as to achieve real-time and accurate acquisition of bone pose information in the virtual world with a small amount of sensor information.
It realizes the real-time accurate acquisition of bone posture information in the virtual world with a small amount of sensor information in the XR scene, improving the player's immersive experience.
Smart Images
Figure CN120014123A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of motion capture, and in particular, relates to a virtual object model generation method and a virtual object model generation device. Background Art
[0002] In the application scenario of motion capture, the relevant technologies mainly include the following two implementation methods: one is to collect sensor information and generate skeleton animation data offline for use in 3D games or movies, and the other is to collect sensor information in real time and project it into the skeleton for live broadcast, etc. However, the above methods all require the operator to bind more motion capture sensors. When the number of bound motion capture sensors is small, it will greatly affect the accuracy of the final established skeleton posture, and the real-time performance is poor; in addition, the above methods cannot be applied to scenes such as extended reality (Extended Reality, XR) where it is impossible to wear more motion capture sensors, affecting the operator's immersive experience. Summary of the invention
[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a virtual object model generation method and a virtual object model generation device, which can obtain the corresponding skeletal posture information in the virtual world in real time and accurately from a small amount of sensor information, which is suitable for XR scenarios and helps to improve the player's immersive experience.
[0004] In a first aspect, the present application provides a method for generating a virtual object model, the method comprising:
[0005] Acquire standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action;
[0006] Determining pre-calibration information based on the standard pose information and standard information corresponding to the standard skeleton model;
[0007] The standard skeleton model is processed based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
[0008] According to the virtual object model generation method of the present application, by making the target object perform the target standard action to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to obtain the corresponding skeleton posture information in the virtual world in real time and accurately using a small amount of sensor information. This is suitable for XR scenarios and helps to improve the player's immersive experience.
[0009] According to an embodiment of the present application, the processing of the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object includes:
[0010] Calibrate the standard skeleton model based on the pre-calibration information to construct a first skeleton model corresponding to the target object;
[0011] The first skeleton model is processed by using the motion capture algorithm to construct the target skeleton model.
[0012] According to an embodiment of the present application, the step of processing the first skeleton model using the motion capture algorithm to construct the target skeleton model includes:
[0013] Processing the first skeleton model using the motion capture algorithm to construct a second skeleton model;
[0014] The second skeleton model is optimized using a whole-body inverse kinematics algorithm to construct the target skeleton model.
[0015] According to one embodiment of the present application, the step of optimizing the second skeleton model using a whole-body inverse kinematics algorithm to construct the target skeleton model includes:
[0016] Acquire a target kinematic feature corresponding to a target joint among the multiple joints;
[0017] Based on the target kinematic characteristics, the whole-body inverse kinematics algorithm is used to optimize the second skeleton model to construct the target skeleton model.
[0018] According to an embodiment of the present application, the step of processing the first skeleton model using the motion capture algorithm to construct a second skeleton model includes:
[0019] Based on the type of the target joint among the multiple joints, constructing a first sub-model corresponding to the target joint using a motion capture algorithm corresponding to the type;
[0020] Mapping the real-time posture information corresponding to the target joint with the standard posture information corresponding to the target joint, and constructing a second sub-model corresponding to other joints associated with the target joint; the other joints are other joints except the multiple joints;
[0021] The first sub-model and the second sub-model are fused to obtain the second skeleton model.
[0022] According to an embodiment of the present application, determining the pre-calibration information based on the standard pose information and the standard information corresponding to the standard skeleton model includes:
[0023] Based on the standard posture information and the standard information, calibration data is determined; the calibration data includes at least one of the height difference information between the target object and the standard skeleton model, the waist difference information between the target object and the standard skeleton model, the distance information between each bone point on the arm between the target object and the standard skeleton model when performing the target standard action, and the distance information between the spine bone point and each bone point on the arm between the target object and the standard skeleton model;
[0024] Determine the empirical parameter based on the spatial offset between the actual binding position of the joint and the actual bone point of the joint when the collector collects the standard posture information;
[0025] The standard posture information is corrected based on the calibration data and the empirical parameters to determine the pre-calibration information.
[0026] According to one embodiment of the present application, the multiple joints include: at least one of: head joints, hand joints, corresponding arm joints, waist joints, corresponding leg joints and foot joints.
[0027] According to an embodiment of the present application, after processing the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object, the method further includes:
[0028] Acquire real-time posture information corresponding to the multiple joints of the target object;
[0029] A redirection algorithm is used to process the real-time posture information and the target skeleton model to obtain a real-time posture model corresponding to the target object.
[0030] In a second aspect, the present application provides a virtual object model generation device, the device comprising:
[0031] A first processing module is used to obtain standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action;
[0032] A second processing module, configured to determine pre-calibration information based on the standard posture information and standard information corresponding to the standard skeleton model;
[0033] The third processing module is used to process the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
[0034] According to the virtual object model generating device of the present application, by making the target object perform the target standard action to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to obtain the corresponding skeleton posture information in the virtual world in real time and accurately using a small amount of sensor information, which is suitable for XR scenarios and helps to improve the player's immersive experience.
[0035] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating a virtual object model as described in the first aspect above is implemented.
[0036] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for generating a virtual object model as described in the first aspect above is implemented.
[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the virtual object model generation method as described in the first aspect above.
[0038] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0039] By making the target object perform target standard actions to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to obtain the corresponding skeleton posture information in the virtual world in real time and accurately using a small amount of sensor information. This is suitable for XR scenarios and helps to improve the player's immersive experience.
[0040] Furthermore, by combining the inverse kinematics algorithm and the pre-calibration information to restore the second skeleton model, specific joints are further optimized based on the target kinematic characteristics to obtain the target skeleton model. This can make the final target skeleton model more consistent with the skeleton characteristics of the target object, further improving the accuracy and precision of modeling.
[0041] Furthermore, by combining the whole-body inverse kinematics algorithm to solve the second skeleton model, specific joints are further optimized based on the target kinematic characteristics to obtain the target skeleton model. This can make the final target skeleton model more consistent with the skeleton characteristics of the target object, further improving the accuracy and precision of modeling.
[0042] Furthermore, after constructing the target skeleton model, a redirection algorithm is introduced to process the real-time posture information and the target skeleton model to infer the corresponding posture results of the target object in the virtual environment when performing different actions. Only the posture information corresponding to a small number of skeleton points needs to be obtained in real time to perform accurate motion capture, which is suitable for XR scenarios and helps to improve the player's immersive experience.
[0043] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0045] Figure 1 It is one of the flowcharts of the method for generating a virtual object model provided in the embodiment of the present application;
[0046] Figure 2 This is one of the principle schematic diagrams of the virtual object model generation method provided in the embodiment of the present application;
[0047] Figure 3 This is the second principle schematic diagram of the method for generating a virtual object model provided in an embodiment of the present application;
[0048] Figure 4 This is the third principle schematic diagram of the method for generating a virtual object model provided in an embodiment of the present application;
[0049] Figure 5 This is the fourth principle schematic diagram of the virtual object model generation method provided in the embodiment of the present application;
[0050] Figure 6 This is the second flow chart of the method for generating a virtual object model provided in an embodiment of the present application;
[0051] Figure 7 is a structural schematic diagram of a virtual object model generating device provided in an embodiment of the present application;
[0052] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0054] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0055] The following, in conjunction with the accompanying drawings, describes in detail the virtual object model generation method, virtual object model generation device, electronic device and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.
[0056] The virtual object model generation method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.
[0057] The terminal includes but is not limited to portable communication devices such as mobile phones or tablet computers. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.
[0058] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.
[0059] The virtual object model generation method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the virtual object model generation method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The virtual object model generation method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.
[0060] like Figure 1 As shown, the virtual object model generation method includes: step 110, step 120 and step 130.
[0061] Step 110: obtaining standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action;
[0062] In this step, the target object may be an object for motion capture, such as a player, an animal or other object, which is not limited in this application.
[0063] It is understandable that during the motion capture process, a tracker needs to be attached to the target object's body to capture standard posture information.
[0064] The target standard action is a pre-designed action.
[0065] The number of target standard actions can be one or more.
[0066] It can be understood that the target standard action is designed based on the category of the target object. If the category of the target object changes, the corresponding target standard action may also change accordingly. For example, the target standard actions corresponding to humans and animals may be different.
[0067] The standard pose information is the world transform information of the tracker in the virtual world, including position, rotation, size and other information.
[0068] Each tracker corresponds to a joint, and each joint has standard pose information.
[0069] Among them, joints are bone points.
[0070] In some embodiments, the number of multiple joints does not exceed ten.
[0071] The following explanation is made by taking the target object as a person.
[0072] like Figure 2 As shown, in some embodiments, the plurality of joints may include at least one of: a head joint, a hand joint, a corresponding arm joint, a waist joint, a corresponding leg joint, and a foot joint.
[0073] In this embodiment, trackers may be attached to the player's head, two hands, two arms, waist, two knees, and two feet to capture standard posture information.
[0074] In the actual implementation process, an HTC headset can be used for the head, a VR handle can be used for the hands, and 7 infrared trackers can be set up accordingly for other joints of the body.
[0075] Each tracker will directly obtain the world transform information of each tracker in the virtual world through the SteamVR software and the OpenXR plug-in under Unreal, including standard posture information such as position, rotation and size.
[0076] like Figure 3 As shown, in some embodiments, the target standard action may include at least two of: standing upright with both hands raised to the sides, standing upright with both hands raised, and standing upright with both hands hanging naturally.
[0077] In this embodiment, Figure 3 (a) illustrates the standard movement of standing upright with both hands raised to the sides. Figure 3 (b) illustrates the standard action of standing upright with both hands raised. Figure 3 (c) illustrates the standard movement of standing upright with both hands hanging naturally.
[0078] In the actual execution process, the target object can make the following Figure 3 The three poses shown are used to collect standard pose information through the tracker.
[0079] The collected standard posture information can be stored in a local or cloud database and retrieved when needed later.
[0080] Step 120, determining pre-calibration information based on the standard pose information and the standard information corresponding to the standard skeleton model;
[0081] In this step, the standard skeleton model is a general skeleton model used to characterize the category to which the target object belongs.
[0082] For example, when the target object is a human, the standard skeleton model is a general human skeleton model. Figure 4 As shown in (a); when the category of the target object is an animal, the standard skeleton model is the universal skeleton model corresponding to this type of animal.
[0083] The standard skeleton model can be designed in advance, and in actual application, a matching standard skeleton model can be selected based on the target object.
[0084] Figure 5 This example shows an overview of the bone points in a standard bone model. Figure 5 As shown, the bone points in the standard bone model may include: pelvis, spine, clavicles on both sides, upper arms on both sides, forearms on both sides, hands on both sides, rotational joints on both sides, neck, head, thighs on both sides, calves on both sides, feet on both sides, and rotational joints on both sides, etc.
[0085] The standard information is used to characterize the positional relationship between the joints (i.e., bone points) in the standard skeleton, the kinematic characteristics of each joint, and the distance between the joints.
[0086] The pre-calibration information is used to calibrate the standard skeleton model to obtain a skeleton model that conforms to the target object.
[0087] The pre-calibration information may be determined based on the difference between the standard pose information and the standard information corresponding to the standard skeleton model.
[0088] It is understandable that different target objects may have different standard posture information collected, and the corresponding calibration information may also be different.
[0089] like Figure 6 As shown, in some embodiments, step 120 may include:
[0090] Determine calibration data based on standard posture information and standard information;
[0091] Determine the empirical parameter based on the spatial offset between the actual binding position of the binding joint and the actual bone point of the joint when the collector collects standard posture information;
[0092] The standard pose information is corrected based on the calibration data and empirical parameters to determine the pre-calibration information.
[0093] In this embodiment, the calibration data may include at least one of height difference information between the target object and the standard skeleton model, waist difference information between the target object and the standard skeleton model, distance information between each bone point on the arm between the target object and the standard skeleton model when performing the target standard action, and distance information from the spine bone point to each bone point on the arm between the target object and the standard skeleton model.
[0094] The difference information may be expressed as a difference or a ratio, etc., which may be determined based on actual conditions and is not limited in this application.
[0095] The distance information may include: length or height, etc.
[0096] The calibration data may include: the height ratio with the standard skeleton; the waist height ratio with the standard skeleton; the height and rotation of footstep tracking when the standard skeleton is touching the ground; the distance between each bone point on the arm of the standard skeleton; the distance between the spine bone point of the standard skeleton to each bone point on the arm and the distance between each bone point on the arm in three calibration postures, etc.
[0097] Taking a child as the target object, in the actual implementation process, the posture information collected by the tracker bound to the specific joint of the child can be used to determine the actual height of the child. Then, the determined actual height is compared with the standard height of the standard skeleton model to obtain the height ratio of the target object and the standard skeleton. Based on a similar method, other comparison information can be determined to obtain calibration data.
[0098] It can be understood that, considering the impact of the deviation between the binding position of the tracker (Tracker) and the actual skeletal joint position of the target object, the spatial offset (Location Offset) between the Tracker position on the standard skeleton and the actual bone point can be calculated based on the fixed Tracker orientation (local XYZ) coordinate system, and converted into an empirical parameter as prior information.
[0099] In actual implementation, the closer the bound tracker is to the standard position indicated by the empirical parameters, the better the motion capture effect.
[0100] The richer the content of the determined pre-calibration information is, the higher the accuracy of the target skeleton model obtained by subsequent calibration will be.
[0101] According to the virtual object model generation method provided by the embodiment of the present application, by determining the calibration data based on the standard posture information and the standard information, the empirical parameters are further determined based on the spatial offset between the actual binding position of the collector's bound joint and the actual bone point of the joint, so that in the subsequent model construction process, the model can be jointly corrected based on the calibration data and the empirical parameters, so that the final model is closer to the bone posture of the target object, thereby improving the authenticity and accuracy of the final constructed model and improving the motion capture effect.
[0102] Step 130: Process the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
[0103] In this step, the target skeleton model is a posture model that is consistent with or close to the actual skeleton features of the target object and has kinematic characteristics that conform to the target object.
[0104] It is understandable that for target objects of the same category, such as humans, different ages or genders may result in different heights, weights, and body shapes, and thus their actual bone features may also vary.
[0105] The pre-calibration information corresponding to different target objects is not completely the same, so the target skeleton models corresponding to different target objects corrected based on the pre-calibration information may also be different.
[0106] The motion capture algorithm may be any relevant algorithm, including but not limited to: a cyclic coordinate descent inverse dynamics (CCD IK) algorithm, a TwoBoneIK algorithm, a FABRIK algorithm, and a full body inverse kinematics (FullBodyIK) algorithm.
[0107] In the present application, the target object's posture information is collected to infer the target object's bone point position information, and then the bone point position information is input into the motion capture algorithm to obtain the target bone model, thereby completing the preliminary restoration of the target object's posture.
[0108] The specific solution method for processing the position information of the skeleton points based on the motion capture algorithm will be explained below and will not be elaborated here.
[0109] Continue to refer Figure 4 When the target object is an adult female, the pre-calibration information determined by the collected standard posture information corresponding to the adult female and the motion capture algorithm are used to process the following: Figure 4 The standard skeleton model shown in (a) can be used to obtain the target skeleton model corresponding to the adult female, such as Figure 4 (b) As shown; when the target object is a child, the pre-calibration information determined by the standard posture information corresponding to the child is collected and processed by the motion capture algorithm. Figure 4 The standard skeleton model shown in (a) can be used to obtain the target skeleton model corresponding to the child, such as Figure 4 (c) as shown.
[0110] After multiple experiments and verifications by the inventor, the virtual object model generation method of the present application only requires 6 sensors to accurately restore the corresponding skeletal posture of the target object in the virtual world. The operation is simple and convenient and the real-time performance is high.
[0111] In the present application, by making the target object perform the target standard action to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to achieve in the motion capture scenario that only the posture information corresponding to no more than 10 skeleton points needs to be obtained in real time to restore the corresponding skeleton posture of the operator, which has high real-time and accuracy, and is simple and convenient to operate;
[0112] In addition, the method provided in this application can be combined with XR to enhance the operator's immersive experience.
[0113] According to the virtual object model generation method provided in the embodiment of the present application, by making the target object perform the target standard action to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to obtain the corresponding skeleton posture information in the virtual world in real time and accurately using a small amount of sensor information. This is suitable for XR scenarios and helps to improve the player's immersive experience.
[0114] Continue to refer Figure 6 In some embodiments, step 130 may include:
[0115] Calibrate the standard skeleton model based on the pre-calibration information to construct a first skeleton model corresponding to the target object;
[0116] The first skeleton model is processed by using a motion capture algorithm to construct a target skeleton model.
[0117] In this embodiment, the first skeleton model is a model obtained by correcting key skeleton points in the standard skeleton model by using pre-calibration information, and is used to characterize the skeleton point position information of the target object.
[0118] Taking a child as the target object as an example, in the actual implementation process, the actual height of the child can be determined through the posture information collected by the tracker bound to the specific joints of the child, and then the determined actual height is compared with the standard height of the standard skeleton model to obtain the height ratio of the target object and the standard skeleton. Based on the height ratio, the standard skeleton model is proportionally reduced to obtain a skeleton model that matches the child's height. Other pre-calibration information can be determined based on a similar method, and the pre-calibration information corresponding to different skeleton points can be used to correct the corresponding skeleton points in the standard skeleton model to obtain the first skeleton model.
[0119] After obtaining the first skeleton model, the corresponding skeleton point position information of the child is input into the motion capture algorithm for solving, so as to restore the child's skeleton posture and obtain the target skeleton model.
[0120] It should be noted that for different nodes, the corresponding solution methods may be different.
[0121] Continue to refer Figure 6 In some embodiments, processing the first skeleton model using a motion capture algorithm to construct a target skeleton model may include:
[0122] The first skeleton model is processed by using a motion capture algorithm to construct a second skeleton model;
[0123] The whole-body inverse kinematics algorithm is used to optimize the second skeleton model and construct the target skeleton model.
[0124] In this embodiment, an inverse kinematics (IK) algorithm is used to determine the joint parameters of the manipulator using the robot kinematics equations so as to move the end effector to a desired position.
[0125] After obtaining the second skeleton model, FullBody IK can be used to smooth the motion capture result posture. The input parameters may include the position and rotation information of eight nodes (two hands, two feet, two elbows, and two knees).
[0126] In some embodiments, optimizing the second skeleton model using a whole-body inverse kinematics algorithm to construct a target skeleton model may include:
[0127] Obtain target kinematic features corresponding to a target joint among multiple joints;
[0128] Based on the target kinematic characteristics, the whole-body inverse kinematics algorithm is used to optimize the second skeleton model and construct the target skeleton model.
[0129] In this embodiment, the target joint may be any joint among the bound joints.
[0130] Kinematic features are used to characterize the kinematic characteristics of each bone point, such as rotation or bending, which are parameters that are strongly related to the bone structure of the target object.
[0131] Different bone points may have different corresponding kinematic characteristics.
[0132] In some embodiments, target kinematic characteristics may be determined empirically.
[0133] Continuing with the example of a human as the target object, for the hand joints, based on experience, it can be roughly assumed that the rotation angle of the hand is roughly within the range of 270 degrees, so the range of 270 degrees can be set as the empirical parameter; the elbow tends to rotate inward, and the maximum outward rotation is limited to 0; the maximum value of the left and right rotation of the head is set to 90 degrees, etc.
[0134] In the actual implementation process, based on the second skeleton model obtained by combining the whole-body inverse kinematics algorithm to solve the second skeleton model, specific joints can be further optimized to obtain the target skeleton model.
[0135] According to the virtual object model generation method provided in the embodiment of the present application, on the basis of obtaining the second skeleton model by combining the whole-body inverse kinematics algorithm to solve the second skeleton model, specific joints are further optimized based on the target kinematic characteristics to obtain the target skeleton model. This can make the final target skeleton model more consistent with the skeleton characteristics of the target object, further improving the accuracy and precision of modeling.
[0136] In some embodiments, processing the first skeleton model using a motion capture algorithm to construct the second skeleton model may include:
[0137] Based on the type of the target joint among the multiple joints, a motion capture algorithm corresponding to the type is used to construct a first sub-model corresponding to the target joint;
[0138] Mapping the real-time pose information corresponding to the target joint with the standard pose information corresponding to the target joint, and constructing a second sub-model corresponding to other joints associated with the target joint;
[0139] The first sub-model and the second sub-model are fused to obtain a second skeleton model.
[0140] In this embodiment, other joints are joints other than the multiple joints, that is, joints to which the tracker is not bound, such as clavicle, shoulder, and spine.
[0141] The following is a detailed explanation of the solution method from different angles, including the main body torso, head, legs, arms, and hands.
[0142] 1. Main body and head
[0143] Step 1: Use the pelvis as the root bone and establish a correspondence between the waist tracker and the pelvis.
[0144] Step 2: Based on the position of the head tracker, use CCD IK as a solver to infer the position of the upper spine (spine_03).
[0145] Step 3: Adjust the overall rotation of the central spine according to the rotation of the waist tracker.
[0146] Step 4: According to the rotation of the VR headset, adjust the rotation of the head bones to obtain the first sub-model corresponding to the main body torso and head.
[0147] 2. Clavicle
[0148] It is understandable that during the motion capture process, the tracker is not bound to joints such as the clavicle, shoulder, and spine. The movement of such joints will not be mapped directly, but through the posture information corresponding to other joints related to the joint that are bound to the tracker, and the position input information of the tracker during calibration.
[0149] During the calibration process, if Figure 3 The three postures in the target standard action shown represent the maximum range of motion of the operator's arm in different directions; by collecting the operator's skeletal information and the skeletal information of the standard skeleton at that time, all subsequent operator skeletal information will be within the maximum range of motion, so that the operator's skeletal information can be proportionally converted into the skeletal information of the standard skeleton, that is, the second sub-model corresponding to the clavicle is obtained.
[0150] According to the virtual object model generation method provided in the embodiment of the present application, a virtual skeletal motion model is constructed by mapping the real-time posture information corresponding to other joints bound to trackers related to the joint to which the tracker is not bound, and the position input information of the tracker during calibration. In this way, a multi-joint virtual skeletal motion model can be constructed based on fewer trackers, thereby further improving the accuracy of the target skeletal model finally obtained.
[0151] 3. Legs
[0152] Step 1: Use TwoBoneIK to infer the knee bend based on the tracker positions on the knee and foot.
[0153] Step 2: Based on the foot rotation information and height information during calibration, the foot bone point position information is calculated based on the foot tracker position information.
[0154] Step 3: Based on the difference in rotation between the current foot tracker and the foot tracker during calibration, calculate the rotation of the current foot bone point (the human knee joint has very little rotational freedom, so only TwoBoneIK can roughly restore the bone information without using the rotation information of the knee tracker), thereby obtaining the first sub-model corresponding to the leg.
[0155] 4. Arms
[0156] First, for the clavicle part, the position input information of the four trackers of the hand is mapped with the position input information of the tracker during calibration to obtain the position information of the elbow bone points under the standard skeleton and the position and rotation information of the hand bone points.
[0157] Next, use TwoBoneIK to infer the movement of the elbow with the same method as the leg, and get the position and rotation of the hand bone points based on the calibration information, thus obtaining the sub-model corresponding to the arm.
[0158] 5. Hands (Twist)
[0159] When the human palm rotates, some muscles of the forearm will rotate together. For this part, a mapping based on a large number of empirical parameters (such as the rotation angle of the hand is roughly within the range of 270 degrees and other parameters that are strongly related to the human bone structure) can be introduced (similar to the clavicle mapping process), and the Twist node in the forearm can be rotated on the standard bone to simulate the process.
[0160] Continue to refer Figure 6 In some embodiments, after step 130, the method may further include:
[0161] Obtain real-time pose information corresponding to multiple joints of the target object;
[0162] The redirection algorithm is used to process the real-time posture information and the target skeleton model to obtain the real-time posture model corresponding to the target object.
[0163] In this embodiment, the real-time posture information is tracker data collected in real time by a bound tracker in a motion capture scenario.
[0164] The real-time pose model is used to characterize the pose results of the target object when performing different actions.
[0165] The redirection algorithm is used to enable the virtual target skeleton model to perform corresponding actions based on the actions of the target object.
[0166] For example, when a player makes a “waving” action, the real-time posture information is the posture data of multiple joints bound when the player makes the “waving” action.
[0167] In the actual implementation process, Unrea's IK Rig-based redirection function can be used to redirect the bone binding chain corresponding to each target joint (such as 5 fingers on each hand, two arms, two clavicles, spine, head and two legs), and finally redirect the motion capture results to each target joint, and infer the posture results of different target human skeletons.
[0168] According to the virtual object model generation method provided in the embodiment of the present application, after constructing the target skeleton model, a redirection algorithm is introduced to process the real-time posture information and the target skeleton model to infer the corresponding posture results of the target object in the virtual environment when performing different actions. Accurate motion capture can be performed by only obtaining the posture information corresponding to a small number of skeleton points in real time. This is suitable for XR scenarios and helps to improve the player's immersive experience.
[0169] The virtual object model generation method provided in the embodiment of the present application can be executed by a virtual object model generation device. In the embodiment of the present application, the virtual object model generation device executing the virtual object model generation method is taken as an example to illustrate the virtual object model generation device provided in the embodiment of the present application.
[0170] The embodiment of the present application also provides a virtual object model generating device.
[0171] like Figure 7 As shown, the virtual object model generating device includes: a first processing module 710 , a second processing module 720 and a third processing module 730 .
[0172] The first processing module 710 is used to obtain standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action;
[0173] A second processing module 720 is used to determine pre-calibration information based on the standard posture information and the standard information corresponding to the standard skeleton model;
[0174] The third processing module 730 is used to process the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
[0175] According to the virtual object model generation device provided in the embodiment of the present application, by making the target object perform the target standard action to collect standard posture information, and then determining the pre-calibration information based on the collected standard posture information and the standard information corresponding to the standard skeleton model, and constructing the target skeleton model corresponding to the target object based on the pre-calibration information and the motion capture algorithm to process the standard skeleton model, it is possible to obtain the corresponding skeleton posture information in the virtual world in real time and accurately using a small amount of sensor information, which is suitable for XR scenarios and helps to improve the player's immersive experience.
[0176] In some embodiments, the third processing module 730 may also be used to:
[0177] Calibrate the standard skeleton model based on the pre-calibration information to construct a first skeleton model corresponding to the target object;
[0178] The first skeleton model is processed by using a motion capture algorithm to construct a target skeleton model.
[0179] In some embodiments, the third processing module 730 may also be used to:
[0180] The first skeleton model is processed by using a motion capture algorithm to construct a second skeleton model;
[0181] The whole-body inverse kinematics algorithm is used to optimize the second skeleton model and construct the target skeleton model.
[0182] In some embodiments, the third processing module 730 may also be used to:
[0183] Obtain target kinematic features corresponding to a target joint among multiple joints;
[0184] Based on the target kinematic characteristics, the whole-body inverse kinematics algorithm is used to optimize the second skeleton model and construct the target skeleton model.
[0185] In some embodiments, the third processing module 730 may also be used to:
[0186] Based on the type of the target joint among the multiple joints, a motion capture algorithm corresponding to the type is used to construct a first sub-model corresponding to the target joint;
[0187] The real-time pose information corresponding to the target joint is mapped with the standard pose information corresponding to the target joint, and a second sub-model corresponding to other joints associated with the target joint is constructed; the other joints are other joints except the multiple joints;
[0188] The first sub-model and the second sub-model are fused to obtain a second skeleton model.
[0189] In some embodiments, the second processing module 720 may also be used to:
[0190] Based on the standard posture information and the standard information, calibration data is determined; the calibration data includes at least one of height difference information between the target object and the standard skeleton model, waist difference information between the target object and the standard skeleton model, distance information between each bone point on the arm between the target object and the standard skeleton model when performing the target standard action, and distance information between the spine bone point and each bone point on the arm between the target object and the standard skeleton model;
[0191] Determine the empirical parameter based on the spatial offset between the actual binding position of the binding joint and the actual bone point of the joint when the collector collects standard posture information;
[0192] The standard pose information is corrected based on the calibration data and empirical parameters to determine the pre-calibration information.
[0193] In some embodiments, the apparatus may further include a fourth processing module, configured to:
[0194] After processing the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object, real-time posture information corresponding to multiple joints of the target object is obtained;
[0195] The redirection algorithm is used to process the real-time posture information and the target skeleton model to obtain the real-time posture model corresponding to the target object.
[0196] The virtual object model generation device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0197] The virtual object model generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0198] The virtual object model generation device provided in the embodiment of the present application can achieve Figures 1 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0199] In some embodiments, Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned virtual object model generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0200] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0201] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned virtual object model generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0202] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0203] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned virtual object model generation method when executed by a processor.
[0204] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0205] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned virtual object model generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0206] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0207] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0208] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0209] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0210] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0211] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A method for generating a virtual object model, characterized in that: include: Acquire standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action; Determining pre-calibration information based on the standard pose information and standard information corresponding to the standard skeleton model; The standard skeleton model is processed based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
2. The method for generating a virtual object model according to claim 1, characterized in that: The step of processing the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object includes: Calibrate the standard skeleton model based on the pre-calibration information to construct a first skeleton model corresponding to the target object; The first skeleton model is processed by using the motion capture algorithm to construct the target skeleton model.
3. The method for generating a virtual object model according to claim 2, characterized in that: The step of processing the first skeleton model using the motion capture algorithm to construct the target skeleton model includes: Processing the first skeleton model using the motion capture algorithm to construct a second skeleton model; The second skeleton model is optimized using a whole-body inverse kinematics algorithm to construct the target skeleton model.
4. The method for generating a virtual object model according to claim 3, characterized in that: The step of optimizing the second skeleton model by using a whole-body inverse kinematics algorithm to construct the target skeleton model comprises: Acquire a target kinematic feature corresponding to a target joint among the multiple joints; Based on the target kinematic characteristics, the whole-body inverse kinematics algorithm is used to optimize the second skeleton model to construct the target skeleton model.
5. The method for generating a virtual object model according to claim 3, characterized in that: The step of using the motion capture algorithm to process the first skeleton model to construct a second skeleton model includes: Based on the type of the target joint among the multiple joints, constructing a first sub-model corresponding to the target joint using a motion capture algorithm corresponding to the type; Mapping the real-time posture information corresponding to the target joint with the standard posture information corresponding to the target joint, and constructing a second sub-model corresponding to other joints associated with the target joint; the other joints are other joints except the multiple joints; The first sub-model and the second sub-model are fused to obtain the second skeleton model.
6. The method for generating a virtual object model according to any one of claims 1 to 5, characterized in that: The determining of pre-calibration information based on the standard pose information and the standard information corresponding to the standard skeleton model includes: Based on the standard posture information and the standard information, calibration data is determined; the calibration data includes at least one of the height difference information between the target object and the standard skeleton model, the waist difference information between the target object and the standard skeleton model, the distance information between each bone point on the arm between the target object and the standard skeleton model when performing the target standard action, and the distance information between the spine bone point and each bone point on the arm between the target object and the standard skeleton model; Determine the empirical parameter based on the spatial offset between the actual binding position of the joint and the actual bone point of the joint when the collector collects the standard posture information; The standard posture information is corrected based on the calibration data and the empirical parameters to determine the pre-calibration information.
7. The method for generating a virtual object model according to any one of claims 1 to 5, characterized in that: The multiple joints include: at least one of: head joints, hand joints, arm corresponding joints, waist joints, leg corresponding joints and foot joints.
8. The method for generating a virtual object model according to any one of claims 1 to 5, characterized in that: After processing the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object, the method further includes: Acquire real-time posture information corresponding to the multiple joints of the target object; A redirection algorithm is used to process the real-time posture information and the target skeleton model to obtain a real-time posture model corresponding to the target object.
9. A virtual object model generation device, characterized in that: include: A first processing module is used to obtain standard posture information corresponding to multiple joints of the target object when the target object performs a target standard action; A second processing module, configured to determine pre-calibration information based on the standard posture information and standard information corresponding to the standard skeleton model; The third processing module is used to process the standard skeleton model based on the pre-calibration information and the motion capture algorithm to construct a target skeleton model corresponding to the target object.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the virtual object model generation method according to any one of claims 1 to 8 is implemented.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a virtual object model as described in any one of claims 1 to 8 is implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a virtual object model according to any one of claims 1 to 8 is implemented.