Video motion capture optimization method, equipment and apparatus based on ground sensing and dynamics
By sampling and dynamic analysis of three-dimensional attitude data, and combining it with ground reaction force to optimize attitude data, the problem of unstable attitude optimization in existing technologies has been solved, achieving a more stable and accurate attitude optimization effect.
Patent Information
- Application Number
- CN202410305820.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-18
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-03-18
AI Technical Summary
In existing technologies, the 3D pose optimization effect is unstable, and there are problems such as global trajectory instability and jitter.
By acquiring displacement and rotation data of the target object's center point and joints, sampling and processing are performed. Combining dynamic principles, the centerline acceleration, jointline acceleration, and angular acceleration are determined. The attitude data is optimized using ground reaction force, and an objective function is constructed for optimization.
It improves the stability and accuracy of 3D pose optimization, reduces pose jitter, and yields more reasonable optimization results that conform to natural laws.
Smart Images

Figure CN118212263B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a video motion capture optimization method, device, and apparatus based on ground sensing and dynamics. Background Technology
[0002] Motion capture technology has a long history. In recent years, with the rapid development of 3D pose estimation, 3D joint coordinates with depth information can more accurately describe 3D pose, thus becoming a hot topic in fields such as medical rehabilitation, sports training assessment, film and television games and intelligent monitoring.
[0003] In existing technologies, 3D pose estimation is performed using images or videos to obtain the 3D pose, and then the 3D pose is optimized to obtain the optimized motion.
[0004] However, existing motion optimization techniques do not take into account the instability and jitter of the global trajectory in 3D pose, resulting in unstable optimization results. Summary of the Invention
[0005] This application provides a video motion capture optimization method, equipment, and apparatus based on ground contact sensing and dynamics to solve the problem of unstable optimization results in the prior art.
[0006] Firstly, this application provides a video motion capture optimization method based on ground contact sensing and dynamics, including:
[0007] The first center displacement data of the center point, the first joint displacement data of the joint point, and the first rotation data of the joint point are obtained on the target object. The first center displacement data, the first joint displacement data, and the first rotation data are the data of the target object in the first time frame.
[0008] Within the preset data adjustment range, the first center displacement data and the first joint displacement data are sampled to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point.
[0009] Based on the sampling center displacement data, sampling joint displacement data, the second center displacement data and the second center linear velocity of the center point, and the second joint displacement data and the second joint linear velocity of the joint point, the centerline acceleration of the center point and the jointline acceleration of the joint point are determined. Among them, the second center displacement data, the second joint displacement data, the second center linear velocity, and the second joint linear velocity are the data of the target object in the second time frame, and the second time frame is the time frame before the first time frame.
[0010] Based on the first rotation data, the second rotation data of the joint, and the second angular velocity of the joint, the angular acceleration of the joint is determined, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame;
[0011] Based on the centerline acceleration, jointline acceleration, and angular acceleration, the target optimized displacement data for the center point and the target optimized rotation data for the joints are determined.
[0012] In this application, the centerline acceleration and jointline acceleration satisfy:
[0013]
[0014] in, It is the combination of centerline acceleration and jointline acceleration. and R is the proportional-differential coefficient. ref Let r be the set of sampling center displacement data and sampling joint displacement data, and let r be the set of second center displacement data and second joint displacement data. It is the set of the second center linear velocity and the second joint linear velocity.
[0015] In this application, the angular acceleration of the joints satisfies:
[0016]
[0017] in, Angular acceleration, and These are the proportional-differential coefficients. θ1 is the first rotation data, and θ2 is the second rotation data. This is the second angular velocity.
[0018] In this application, the target optimized displacement data of the center point and the target optimized rotation data of the joints are determined based on the centerline acceleration, jointline acceleration, and angular acceleration, including:
[0019] Based on the first center displacement data and the first rotation data, the contact points in the joints are determined, and the contact points are the joints that are in contact with the ground.
[0020] Determine the ground reaction force between the contact point and the ground based on the contact point;
[0021] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, the target optimized displacement data of the center point and the target optimized rotation data of the joints are determined.
[0022] In this application, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, including:
[0023] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, determine the optimal centerline acceleration at the center point and the optimal angular acceleration at the joint point;
[0024] Based on the optimized centerline acceleration and optimized angular acceleration, determine the optimized displacement data of the center point and the optimized rotation data of the joints;
[0025] Based on the optimized displacement data, optimized rotation data, first center displacement data, and first rotation data, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined.
[0026] In this application, the centerline acceleration and angular acceleration are optimized to satisfy:
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033] in, Indicates the parameter to be optimized is The objective function of τ is given by q, which is the set of optimized center displacement data for the center point and optimized rotation data for the joints. This is a set of optimized centerline velocities for the center point and optimized angular velocities for the joints. This is a set of optimized centerline accelerations at the center point and optimized angular accelerations at the joints. To optimize angular acceleration, τ is the set of control torques at the center point and joints, J is the Jacobian matrix, K1, K2, and K3 are preset coefficients, λ is the ground reaction force, and τ :6 The control torque at the center point, τ 6: Let J be the control torque at the joint, st represent the constraint condition, and J be the control torque at the joint. c Let M(q) be the Jacobian matrix of the contact point obtained from q, T denotes the transpose of the matrix, and M(q) be the inertia matrix obtained from q. According to q, The obtained gravity, centrifugal force, and Coriolis force According to The contact line velocity at the contact point is obtained, and C is the contact line velocity that satisfies C j The set of C j A set of preset ranges for contact linear velocity. To characterize n j Number of It is a three-dimensional data format, n j For the number of contact points, For use in characterizing three-dimensional data formats, Let be the contact linear velocity of the contact point on the y-axis. Let be the contact linear velocity of the contact point on the x-axis. σ is the contact linear velocity of the contact point on the z-axis, and σ is a preset threshold.
[0034] In this application, optimized displacement data of the center point and optimized rotation data of the joint points are determined based on optimized centerline acceleration and optimized angular acceleration, including:
[0035] The optimized displacement data of the center point is determined based on the optimized centerline acceleration and the time interval between the first and second time frames.
[0036] The optimized rotation data of the joints are determined based on the optimized angular acceleration and the time interval between the first and second time frames.
[0037] In this application, after determining the target optimized displacement data of the center point and the target optimized rotation data of the joints based on the centerline acceleration, jointline acceleration, and angular acceleration, the method further includes:
[0038] Based on the target optimized displacement data, target optimized rotation data, second center displacement data, and second rotation data, determine the motion path of the target object;
[0039] Displays the movement path of the target object.
[0040] Secondly, this application provides a video motion capture optimization device based on ground contact sensing and dynamics, comprising:
[0041] The acquisition module is used to acquire the first center displacement data of the center point, the first joint displacement data of the joint point, and the first rotation data of the joint point on the target object, wherein the first center displacement data, the first joint displacement data, and the first rotation data are data of the target object in the first time frame;
[0042] The sampling module is used to sample the first center displacement data and the first joint displacement data within a preset data adjustment range to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point.
[0043] The first determining module is used to determine the centerline acceleration of the center point and the jointline acceleration of the joint point based on the sampling center displacement data, the sampling joint displacement data, the second center displacement data and the second center linear velocity of the center point, and the second joint displacement data and the second joint linear velocity of the joint point. The second center displacement data, the second joint displacement data, the second center linear velocity, and the second joint linear velocity are the data of the target object in the second time frame, and the second time frame is the time frame before the first time frame.
[0044] The second determining module is used to determine the angular acceleration of the joint point based on the first rotation data, the second rotation data of the joint point, and the second angular velocity of the joint point, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame;
[0045] The target module is used to determine the target optimized displacement data of the center point and the target optimized rotation data of the joints based on the centerline acceleration, jointline acceleration, and angular acceleration.
[0046] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0047] The memory stores instructions that the computer executes;
[0048] The processor executes computer execution instructions stored in memory to implement the method provided in this application.
[0049] The video motion capture optimization method, device, and apparatus based on ground contact sensing and dynamics provided in this application sample the acquired data within a preset data adjustment range to obtain sampled data that can cover situations where the three-dimensional posture has global trajectory instability and jitter. Then, based on the sampled data, the centerline acceleration of the center point of the target object and the jointline acceleration of the joint points are optimized and filtered. Finally, the relevant data of the center point and the joint points are combined to obtain the target optimized displacement data of the center point and the target optimized rotation data of the joint points. The posture of the target object is obtained through the target optimized displacement data and the target optimized rotation data, thereby improving the effect of posture optimization. Attached Figure Description
[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0051] Figure 1 A schematic diagram of three-dimensional poses provided for embodiments of this application;
[0052] Figure 2 A flowchart illustrating a video motion capture optimization method based on ground contact sensing and dynamics, provided for an embodiment of this application;
[0053] Figure 3 A flowchart illustrating another video motion capture optimization method based on ground contact sensing and dynamics provided in this application embodiment;
[0054] Figure 4 A schematic diagram of the structure of a video motion capture optimization device based on ground contact sensing and dynamics provided in this application embodiment;
[0055] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0058] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.
[0059] Motion capture technology has a long history. In recent years, with the rapid development of 3D pose estimation, 3D joint coordinates with depth information can more accurately describe 3D pose, thus becoming a hot topic in fields such as medical rehabilitation, sports training assessment, film and television games and intelligent monitoring.
[0060] In existing technologies, 3D pose estimation is performed using images or videos to obtain the 3D pose, and then the 3D pose is optimized to obtain the optimized motion.
[0061] However, existing motion optimization techniques do not take into account the instability and jitter of the global trajectory in 3D pose, resulting in unstable optimization results.
[0062] To address the issue of unstable optimization results, the inventors discovered in their research that a method can be used to sample the first center displacement data and the first joint displacement data from the posture data to be optimized, obtaining multiple sets of sampled center displacement data and sampled joint displacement data. Then, for each set of sampled center displacement data and sampled joint displacement data, posture optimization is performed on the center point and joint points to obtain optimized center displacement data and optimized rotation data. Finally, the target optimized displacement data and target optimized rotation data that meet the optimization objective are determined from the optimized center displacement data and optimized rotation data, thereby achieving the purpose of posture optimization.
[0063] The following describes the application scenarios of the video motion capture optimization method based on ground contact perception and dynamics provided in the embodiments of this application.
[0064] This application provides a scenario for attitude optimization, including: attitude data and a computer. The attitude data is attitude data of a target object containing time information. The computer samples the center displacement data in the attitude data to obtain multiple sampled center displacement data. Then, it performs attitude optimization based on each sampled center displacement data to obtain optimized attitude data. Finally, it determines the target optimized attitude data that meets the optimization objective from the optimized attitude data.
[0065] The target object can refer to the object that needs to be motion captured and its posture optimized. The target object can include human bodies and animals.
[0066] The attitude data includes the center point, the center displacement data of the center point, the joints, and the rotation data of the joints. The displacement data represents the movement state of the center point and includes values in three dimensions. The rotation data is used to represent the rotation state of each joint and can include the rotation center, rotation angle, and rotation direction.
[0067] Figure 1 A schematic diagram of the three-dimensional pose provided in the embodiments of this application, as shown below. Figure 1 As shown, 0 is the center point and 1-24 are joint points. The center point can refer to the point in the three-dimensional pose used to calibrate the overall displacement of the target object.
[0068] A joint can refer to a point in a 3D pose that is used to simplify the main moving joints of a target object. For example, joints can include the head, shoulder joint, elbow joint, hand, hip joint, knee joint, and foot.
[0069] The three-dimensional pose also includes the active and passive relationships between joints. For example, the displacement of 23 is affected by 21, so 21 is the active joint of 23 and 23 is the passive joint of 21. Similarly, 23, 21, and 19 are the passive joints of 17.
[0070] Methods for acquiring pose data can include obtaining it through video data of the target object and a trained pose estimation model. For example, a camera can record a video of a person running, and then a rectangle can be used to outline the person. Then, preprocessing operations such as image scaling, cropping, averaging, and normalization can be performed on the frame data of the video data to obtain a preprocessed motion video. The preprocessed motion video can then be transmitted to the pose estimation model to obtain pose data.
[0071] The pose estimation model can be trained on a deep learning model based on training video data and the training pose data of the training video data. The algorithm of this deep learning model mainly includes: extracting two-dimensional pose information from the training video data, and mapping the two-dimensional pose information to three-dimensional space through a three-dimensional pose recovery algorithm to obtain pose data.
[0072] Figure 2 A flowchart illustrating a video motion capture optimization method based on ground contact sensing and dynamics, as provided in this application embodiment, is shown below. Figure 2 As shown, the method includes:
[0073] S201. Obtain the first center displacement data of the center point on the target object, the first joint displacement data of the joint point, and the first rotation data of the joint point, wherein the first center displacement data, the first joint displacement data, and the first rotation data are data of the target object in the first time frame.
[0074] The first center displacement data can refer to the center displacement data of the center point in the first time frame, and the first joint displacement data can refer to the joint displacement data of the joint point in the first time frame. In this embodiment, the first center displacement data and the first joint displacement data can refer to the coordinate point data of the center point and the joint point in the three-dimensional coordinate system in the first time frame.
[0075] The first rotation data can refer to the rotation data of the joints of the target object in the first time frame.
[0076] The first time frame can refer to a specific moment in time when the target object is located. For example, the first time frame can be the time corresponding to a certain frame in the video data of the target object.
[0077] The method for determining the first center displacement data and the first rotation data includes: substituting the estimated posture data output by the posture estimation model into the virtual model to determine the first center displacement data and the first rotation data. The virtual model can refer to a model that simulates the body shape, weight, and limbs of the target object, thereby better optimizing the posture.
[0078] The method for determining the first joint displacement data includes: determining the coordinate point data of the joint in the three-dimensional coordinate system based on the first center displacement data, the first rotation data, and the three-dimensional posture, and determining the first joint displacement data based on the coordinate point data.
[0079] S202. Within the preset data adjustment range, the first center displacement data and the first joint displacement data are sampled to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point.
[0080] Before sampling, when the target object is a human body, a human body model and ground environment need to be set up based on the first center displacement data and the first rotation data. This allows the computer to perform sampling and subsequent calculations based on the weight, size, and height of the human body model. The ground environment is used by the computer to determine whether the human body model is in contact with the ground, the joints in contact with the ground, and the ground reaction force on the joints.
[0081] Sampling processing can refer to the process of taking values within a preset data adjustment range to obtain sampled data. In this embodiment, sampling processing is the operation of uniformly sampling within a preset data adjustment range.
[0082] Sampling center displacement data can refer to the result of sampling the center point according to the data adjustment range.
[0083] Sampling joint displacement data can refer to the result of sampling the first joint displacement data based on the difference between the first center displacement data and the sampling center displacement data. For example, if the difference between the first center displacement data and the sampling center displacement data is x, then the sampling joint displacement data is the first joint displacement data plus x.
[0084] The preset data adjustment range refers to a pre-defined range of data used for sampling. Methods for determining the data adjustment range may include: determining the movement limit of the human body's center point within the time interval between the second and first time frames, and then pre-setting the data adjustment range based on this movement limit. Alternatively, the data adjustment range can be determined based on a certain proportion of the distance between the center point of the human body and the first time frame. For example, if the distance between the position of the human hand and the position point between the second and first time frames is 10 centimeters, and the data adjustment range is 10% of that distance, then the data adjustment range is within 1 centimeter of the position point of the hand in the first time frame.
[0085] In this embodiment, the data adjustment range can refer to the range of sampling around the three-dimensional coordinate point. For example, if the displacement data is (0,0,0), the data adjustment range is within ±10 millimeters of the value in each dimension.
[0086] S203. Based on the sampling center displacement data, sampling joint displacement data, second center displacement data and second center linear velocity of the center point, and second joint displacement data and second joint linear velocity of the joint point, determine the centerline acceleration of the center point and the jointline acceleration of the joint point. The second center displacement data, second joint displacement data, second center linear velocity, and second joint linear velocity are the data of the target object in the second time frame, and the second time frame is the time frame preceding the first time frame.
[0087] The second center line velocity can refer to the second center line velocity determined based on the second center displacement data, the third center displacement data of the target object in the third time frame, and the time interval between the second and third time frames, where the third time frame is the time frame preceding the second time frame.
[0088] The second joint linear velocity can refer to the second joint linear velocity determined based on the second joint displacement data, the third joint displacement data of the target object in the third time frame, and the time interval between the second and third time frames.
[0089] In this embodiment of the application, the centerline acceleration and the jointline acceleration satisfy the following:
[0090]
[0091] in, It is the combination of centerline acceleration and jointline acceleration. and R is the proportional-differential coefficient. ref Let r be the set of sampling center displacement data and sampling joint displacement data, and let r be the set of second center displacement data and second joint displacement data. It is the set of the second center linear velocity and the second joint linear velocity.
[0092] One method for determining the centerline acceleration is to sample the center displacement data and the second center displacement data to determine the position of the center point in the first time frame and the second time frame, and then determine the linear acceleration of the center point from the second time frame to the first time frame using the two position points.
[0093] One method to determine joint linear acceleration is to sample joint displacement data and second joint displacement data to determine the position points of the joint points in the first and second time frames, and then determine the linear acceleration of the joint point from the second time frame to the first time frame using the two position points.
[0094] Linear acceleration can be calculated using a PD (proportional-derivative) controller. The PD controller is used to control the steady-state and dynamic behavior of the system. In this embodiment, the PD controller can be used to ensure that the center point and joints reach the desired position and velocity. By adjusting... and It can make the movements of the human body model smoother and more precise, while reducing overshoot and oscillation, and improving the accuracy and stability of the human body model's movements.
[0095] S204. Determine the angular acceleration of the joint point based on the first rotation data, the second rotation data of the joint point, and the second angular velocity of the joint point, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame.
[0096] The method for determining the second angular velocity may include: determining the second angular velocity based on the second rotation data, the third rotation data of the target object in the third time frame, and the time interval between the second and third time frames.
[0097] In this embodiment of the application, the angular acceleration of the joint point satisfies
[0098]
[0099] in, Angular acceleration, and These are the proportional-differential coefficients. θ1 is the first rotation data, and θ2 is the second rotation data. This is the second angular velocity.
[0100] in, In To estimate the rotation data in the attitude data, It can refer to the first rotation data obtained by substituting the estimated rotation data into the virtual model.
[0101] Since the motion of the target object needs to take into account the coordination of each joint, the connection length, and the overall motion effect, the angular acceleration of the joint is the collection of the angular accelerations of each joint.
[0102] The method for determining angular acceleration can be as follows: by using the first rotation data and the second rotation data, the angle values of the joint points in the first time frame and the second time frame are determined respectively, and the angular acceleration of the joint points from the second time frame to the first time frame is determined by the two angle values.
[0103] S205. Based on the centerline acceleration, jointline acceleration, and angular acceleration, determine the target optimized displacement data of the center point and the target optimized rotation data of the joints.
[0104] Among them, target optimized displacement data and target optimized rotation data can refer to attitude data that meet the optimization requirements.
[0105] In this embodiment of the application, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined based on the centerline acceleration, jointline acceleration, and angular acceleration, including:
[0106] Based on the first center displacement data and the first rotation data, the contact points in the joints are determined, and the contact points are the joints that are in contact with the ground.
[0107] Determine the ground reaction force between the contact point and the ground based on the contact point;
[0108] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, the target optimized displacement data of the center point and the target optimized rotation data of the joints are determined.
[0109] The ground can be an inelastic surface. When the joint contacts the ground, it is assumed that the contact point will not slide or vibrate, so that the ground reaction force at the contact point is within the friction cone.
[0110] The method for determining the contact point and the ground reaction force between the contact point and the ground may include: inputting first displacement data and first rotation data into a trained ground reaction force model to obtain the contact point in the joints output by the ground reaction force model and the ground reaction force between the contact point and the ground.
[0111] The ground reaction force model can be obtained by training a neural network model based on training posture data, training contact points corresponding to the training posture data, and training ground reaction force between the training contact points and the ground.
[0112] In this embodiment of the application, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, including:
[0113] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, determine the optimal centerline acceleration at the center point and the optimal angular acceleration at the joint point;
[0114] Based on the optimized centerline acceleration and optimized angular acceleration, determine the optimized displacement data of the center point and the optimized rotation data of the joints;
[0115] Based on the optimized displacement data, optimized rotation data, first center displacement data, and first rotation data, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined.
[0116] In this embodiment of the application, the centerline acceleration and angular acceleration are optimized to satisfy:
[0117]
[0118]
[0119]
[0120]
[0121]
[0122]
[0123] in, Indicates the parameter to be optimized is The objective function of τ is given by q, which is the set of optimized center displacement data for the center point and optimized rotation data for the joints. This is a set of optimized centerline velocities for the center point and optimized angular velocities for the joints. This is a set of optimized centerline accelerations at the center point and optimized angular accelerations at the joints. To optimize angular acceleration, τ is the set of control torques at the center point and joints, J is the Jacobian matrix, K1, K2, and K3 are preset coefficients, λ is the ground reaction force, and τ :6 The control torque at the center point, τ 6: Let J be the control torque at the joint, st represent the constraint condition, and J be the control torque at the joint. c Let M(q) be the Jacobian matrix of the contact point obtained from q, T denotes the transpose of the matrix, and M(q) be the inertia matrix obtained from q. According to q, The obtained gravity, centrifugal force, and Coriolis force According to The contact line velocity at the contact point is obtained, and C is the contact line velocity that satisfies C j The set of C j A set of preset ranges for contact linear velocity. To characterize n j Number of It is a three-dimensional data format, n j For the number of contact points, For use in characterizing three-dimensional data formats, Let be the contact linear velocity of the contact point on the y-axis. Let be the contact linear velocity of the contact point on the x-axis. σ is the contact linear velocity of the contact point on the z-axis, and σ is a preset threshold.
[0124] Among them, optimized centerline acceleration refers to the data after optimizing the centerline acceleration at the center point, that is... Optimized centerline acceleration.
[0125] Optimized angular acceleration refers to the data after optimizing the angular acceleration at the joints, i.e. Optimized angular acceleration.
[0126] Methods for determining the optimal centerline acceleration and optimal angular acceleration may include: when the target object is a human body, using a torque-controlled floating base to simulate a human body model, assuming the human body model's posture is determined by q = [r root The value of r is determined by θ, where r root Let q be the displacement data of the center point and θ be the rotation data of the joints. The displacement data can be obtained from q and the time interval between the first and second time frames. and The motion of the human body model is driven by control torque τ, which includes the control torque at the center point and the control torque at the joints. Based on the forces acting on the human body model and the dynamic equations, the first constraint condition formula is constructed. This ensures that the calculated optimized centerline acceleration and optimized angular acceleration satisfy the dynamic equations. Secondly, assuming no slippage or jitter occurs when the contact point contacts the ground, a velocity constraint is imposed on the contact point, constructing a second constraint formula. This ensures that the calculated contact line velocity at the contact point meets the contact point velocity limit. Then, based on optimized centerline acceleration, optimized angular acceleration, centerline acceleration, joint line acceleration, angular acceleration, control torque, and ground reaction force, a system is constructed... Let τ be the objective function of argmin for variables, where τ is an intermediate variable. The solution is obtained by using the objective function, the first constraint formula, and the second constraint formula. This allows us to obtain optimized centerline acceleration and optimized angular acceleration.
[0127] In this embodiment of the application, determining the optimized displacement data of the center point and the optimized rotation data of the joint points based on the optimized centerline acceleration and the optimized angular acceleration includes:
[0128] The optimized displacement data of the center point is determined based on the optimized centerline acceleration and the time interval between the first and second time frames.
[0129] The optimized rotation data of the joints are determined based on the optimized angular acceleration and the time interval between the first and second time frames.
[0130] The method for determining the optimized center displacement data may include: obtaining the optimized center displacement data by means of the finite difference method based on the optimized centerline acceleration and the time interval between the first time frame and the second time frame.
[0131] The method for determining the optimized rotation data may include: obtaining the optimized rotation data through a finite difference method based on the optimized angular acceleration and the time interval between the first and second time frames.
[0132] In some implementations, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined based on optimized displacement data, optimized rotation data, first center displacement data, and first rotation data, including:
[0133] Based on the optimized displacement data and optimized rotation data, the optimized position point of each joint is determined. Based on the first center displacement data and the first rotation data, the first position point of each joint is determined. By constructing a loss function, the optimized position point and the first position point are compared. The optimized position point that is closest to the first position point is selected as the target position point. Then, the optimized displacement data and optimized rotation data corresponding to the target position point are used as the target optimized displacement data and target optimized rotation data.
[0134] To improve the attitude optimization effect, N sets of sampling center displacement data and sampling joint displacement data can be constructed. Correspondingly, N sets of optimized displacement data and optimized rotation data are obtained. By determining the target number of target optimized displacement data and target optimized rotation data from the N sets of optimized displacement data and optimized rotation data, the problem of attitude data jitter is solved, making the obtained target optimized displacement data and target optimized rotation data more accurate and reasonable.
[0135] In this embodiment of the application, after determining the target sampling data in the sampling data based on the linear acceleration of the center point and the angular acceleration of the joint points, the method further includes:
[0136] Based on the target optimized displacement data, target optimized rotation data, second center displacement data, and second rotation data, determine the motion path of the target object;
[0137] Displays the movement path of the target object.
[0138] The motion path can refer to the change in the position of the center point and joints in the three-dimensional coordinate system between the second time frame and the first time frame.
[0139] By continuously changing the motion path across multiple time frames, the actual motion process of the target object can be accurately simulated, achieving the goal of posture optimization.
[0140] Methods for displaying the motion path of a target object may include: displaying the obtained motion path through a virtual model on a monitor for easy viewing or editing by staff, etc.
[0141] In some implementations, the effect of posture optimization can be further improved by obtaining a target number of motion paths and then determining the target motion path from multiple motion paths. The method for determining the target motion path can be determined by a loss function, for example, by aligning the motion path with the two-dimensional posture projection of the target object and determining the target motion path based on the projection alignment result.
[0142] This application provides a video motion capture optimization method based on ground contact perception and dynamics. It samples acquired data within a preset data adjustment range to obtain sampled data that covers situations where the three-dimensional posture exhibits global trajectory instability and jitter. Based on this sampled data, the centerline acceleration of the target object's center point and the jointline acceleration of its joints are optimized and filtered. Finally, by combining the relevant data of the center point and joints, the target optimized displacement data of the center point and the target optimized rotation data of the joints are obtained. The posture of the target object is then obtained using these target optimized displacement and rotation data, thereby improving the posture optimization effect. By obtaining the centerline acceleration, jointline acceleration, and angular acceleration, it is convenient to further construct objective functions to optimize the centerline acceleration and angular acceleration, resulting in optimized centerline acceleration and optimized angular acceleration. The method of estimating the ground reaction force between the contact point and the ground using three-dimensional first center displacement data and first rotation data provides depth information of the motion posture compared to two-dimensional data, solving the problem of poor robustness in motion ground contact estimation using two-dimensional data. By using the constraint formula constructed from the dynamic equations and ground contact constraints to limit the objective function, the problems of contact point jitter and clipping are solved, making the obtained target sampling data more reasonable and more in line with natural laws, and further improving the effect of attitude optimization.
[0143] Figure 3 A flowchart illustrating another video motion capture optimization method based on ground contact sensing and dynamics provided in this application embodiment, the method comprising:
[0144] S301. Perform 3D pose estimation based on the input video to obtain the 3D pose, which includes the video coordinate position of the joint in each frame of the input video, as well as the local rotation, global rotation and displacement of the joint in the 3D case.
[0145] Methods for obtaining three-dimensional pose can include:
[0146] The input video is used to detect human bodies in the image frames and then outline them with rectangles. The image frame data is preprocessed to improve the accuracy of subsequent pose estimation. The preprocessing includes image scaling, cropping, mean normalization, etc. After preprocessing, the data is input into the 3D pose estimation model to generate the 3D pose of the human body in the image frame data.
[0147] S302. Based on the three-dimensional posture, obtain the action-ground contact prior information, which includes the point of contact and the ground reaction force on the point of contact.
[0148] Methods for obtaining prior information about the action's ground contact can include:
[0149] The 3D attitude data is preprocessed to avoid data domain mismatch caused by using a global representation in the coordinate system. In order to extract the temporal information of the motion sequence, multiple image data frames are combined into a sliding window as input to the ground reaction force estimation model. The ground reaction force estimation model generates the ground reaction force of the ground on the point of contact in the current image frame.
[0150] S303. Based on the three-dimensional posture and ground contact prior information, perform reasonable motion optimization to obtain the optimized motion file.
[0151] Among them, methods for optimizing reasonable actions based on three-dimensional posture and prior information about ground contact can include:
[0152] Based on the dynamic equations, i.e. the relationship between force and acceleration, the first constraint condition is constructed;
[0153] The magnitude of the contact point velocity is constructed based on the prior information of the action contacting the ground, satisfying the second constraint condition of the friction cone constraint.
[0154] Import the 3D pose information into the human body physical model and set the ground environment;
[0155] Multiple sets of reference displacements are obtained by uniformly sampling within a preset range around the first displacement of the first time frame in the three-dimensional pose.
[0156] Based on the reference displacement and the second pose of the three-dimensional pose in the second time frame, the target linear acceleration of the three-dimensional pose and the target angular acceleration of the joints are obtained. The target linear acceleration includes the linear acceleration of the center point and the linear acceleration of the joints.
[0157] Construct an objective function based on the target angular acceleration, target linear acceleration, the first constraint, and the second constraint. Solve for the optimal angular acceleration and optimal linear acceleration when the objective function is minimized, where the optimal linear acceleration is the optimal linear acceleration at the center point.
[0158] Based on the second attitude, optimized angular acceleration, and optimized linear acceleration, the optimized position points of the joints of the target object in the first time frame are obtained by finite difference solution.
[0159] Based on the optimized location point, determine the optimized motion path and the target optimized motion path within the optimized motion path.
[0160] This application provides another video motion capture optimization method based on ground contact perception and dynamics. It performs 3D pose estimation on the input video to obtain the 3D pose, then obtains ground contact prior information based on the 3D pose, and finally optimizes the reasonable motion based on the 3D pose and the ground contact prior information to obtain the optimized motion file. Multiple sets of reference poses are obtained by uniformly sampling around the pose data to be optimized. These reference poses are then optimized separately to obtain multiple sets of optimized reference poses. Finally, the target optimized reference pose is determined from the optimized reference poses. This avoids the problem of global position control jitter, making the obtained target optimized reference pose more reasonable and in line with natural laws, thus improving the effect of pose optimization.
[0161] Figure 4 A schematic diagram of a video motion capture optimization device based on ground contact sensing and dynamics provided in this application embodiment is shown below. Figure 4 As shown, the device includes: an acquisition module 401, a sampling module 402, a first determination module 403, a second determination module 404, and a target module 405, wherein:
[0162] The acquisition module 401 is used to acquire the first center displacement data of the center point on the target object, the first joint displacement data of the joint point, and the first rotation data of the joint point, wherein the first center displacement data, the first joint displacement data, and the first rotation data are data of the target object in the first time frame;
[0163] The sampling module 402 is used to sample the first center displacement data and the first joint displacement data within a preset data adjustment range to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point.
[0164] The first determining module 403 is used to determine the centerline acceleration of the center point and the jointline acceleration of the joint point based on the sampling center displacement data, the sampling joint displacement data, the second center displacement data and the second center linear velocity of the center point, and the second joint displacement data and the second joint linear velocity of the joint point. The second center displacement data, the second joint displacement data, the second center linear velocity, and the second joint linear velocity are the data of the target object in the second time frame, and the second time frame is the time frame before the first time frame.
[0165] The second determining module 404 is used to determine the angular acceleration of the joint point based on the first rotation data, the second rotation data of the joint point, and the second angular velocity of the joint point, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame;
[0166] The target module 405 is used to determine the target optimized displacement data of the center point and the target optimized rotation data of the joints based on the centerline acceleration, jointline acceleration, and angular acceleration.
[0167] In this embodiment of the application, the first determining module 403 is further configured to:
[0168] The centerline acceleration and jointline acceleration satisfy:
[0169]
[0170] in, It is the combination of centerline acceleration and jointline acceleration. and R is the proportional-differential coefficient. ref Let r be the set of sampling center displacement data and sampling joint displacement data, r be the set of second center displacement data and second joint displacement data, and r be the set of second center linear velocity and second joint linear velocity.
[0171] In this embodiment of the application, the second determining module 404 is further configured to:
[0172] The angular acceleration at the joints satisfies
[0173]
[0174] in, Angular acceleration, and These are the proportional-differential coefficients. θ1 is the first rotation data, and θ2 is the second rotation data. This is the second angular velocity.
[0175] In this embodiment of the application, the target module 405 is further configured to:
[0176] Based on the centerline acceleration, jointline acceleration, and angular acceleration, determine the target optimized displacement data for the center point and the target optimized rotation data for the joints, including:
[0177] Based on the first center displacement data and the first rotation data, the contact points in the joints are determined, and the contact points are the joints that are in contact with the ground.
[0178] Determine the ground reaction force between the contact point and the ground based on the contact point;
[0179] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, the target optimized displacement data of the center point and the target optimized rotation data of the joints are determined.
[0180] In this embodiment of the application, the target module 405 is further configured to:
[0181] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, the target optimized displacement data for the center point and the target optimized rotation data for the joints are determined, including:
[0182] Based on the centerline acceleration, jointline acceleration, angular acceleration, contact point, and ground reaction force, determine the optimal centerline acceleration at the center point and the optimal angular acceleration at the joint point;
[0183] Based on the optimized centerline acceleration and optimized angular acceleration, determine the optimized displacement data of the center point and the optimized rotation data of the joints;
[0184] Based on the optimized displacement data, optimized rotation data, first center displacement data, and first rotation data, the target optimized displacement data of the center point and the target optimized rotation data of the joint points are determined.
[0185] In this embodiment of the application, the target module 405 is further configured to:
[0186] In this application, the centerline acceleration and angular acceleration are optimized to satisfy:
[0187]
[0188]
[0189]
[0190]
[0191]
[0192]
[0193] in, Indicates the parameter to be optimized is The objective function of τ is given by q, which is the set of optimized center displacement data for the center point and optimized rotation data for the joints. This is a set of optimized centerline velocities for the center point and optimized angular velocities for the joints. This is a set of optimized centerline accelerations at the center point and optimized angular accelerations at the joints. To optimize angular acceleration, τ is the set of control torques at the center point and joints, J is the Jacobian matrix, K1, K2, and K3 are preset coefficients, λ is the ground reaction force, and τ :6 The control torque at the center point, τ 6: Let J be the control torque at the joint, st represent the constraint condition, and J be the control torque at the joint. c Let M(q) be the Jacobian matrix of the contact point obtained from q, T denotes the transpose of the matrix, and M(q) be the inertia matrix obtained from q. According to q, The obtained gravity, centrifugal force, and Coriolis force According to The contact line velocity at the contact point is obtained, and C is the contact line velocity that satisfies C j The set of C j A set of preset ranges for contact linear velocity. To characterize n j Number of It is a three-dimensional data format, n j For the number of contact points, For use in characterizing three-dimensional data formats, Let be the contact linear velocity of the contact point on the y-axis. Let be the contact linear velocity of the contact point on the x-axis. σ is the contact linear velocity of the contact point on the z-axis, and σ is a preset threshold.
[0194] In this embodiment of the application, the target module 405 is further configured to:
[0195] Based on the optimized centerline acceleration and optimized angular acceleration, the optimized displacement data of the center point and the optimized rotation data of the joints are determined, including:
[0196] The optimized displacement data of the center point is determined based on the optimized centerline acceleration and the time interval between the first and second time frames.
[0197] The optimized rotation data of the joints are determined based on the optimized angular acceleration and the time interval between the first and second time frames.
[0198] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes:
[0199] The electronic device 50 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a communication component 503, and other components. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0200] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-mentioned video motion capture optimization method based on ground sensing and dynamics.
[0201] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0202] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0203] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0204] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0205] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described ground-sensing and dynamics-based video motion capture optimization methods.
[0206] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0207] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0208] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the video motion capture optimization methods based on ground sensing and dynamics provided in embodiments of this application.
[0209] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0210] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.
[0211] Since the instructions stored in the storage medium can execute the steps in any of the video motion capture optimization methods based on ground sensing and dynamics provided in the embodiments of this application, the beneficial effects that any of the video motion capture optimization methods based on ground sensing and dynamics provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.
[0212] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0213] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A video motion capture optimization method based on ground contact sensing and dynamics, characterized in that, include: Acquire the first center displacement data of the center point on the target object, the first joint displacement data of the joint point, and the first rotation data of the joint point, wherein the first center displacement data, the first joint displacement data, and the first rotation data are data of the target object in the first time frame; Within a preset data adjustment range, the first center displacement data and the first joint displacement data are sampled to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point. Based on the sampling center displacement data, sampling joint displacement data, the second center displacement data and the second center linear velocity of the center point, and the second joint displacement data and the second joint linear velocity of the joint point, the center linear acceleration of the center point and the joint linear acceleration of the joint point are determined, wherein the second center displacement data, the second joint displacement data, the second center linear velocity, and the second joint linear velocity are data of the target object in the second time frame, and the second time frame is the time frame preceding the first time frame; The angular acceleration of the joint is determined based on the first rotation data, the second rotation data of the joint, and the second angular velocity of the joint, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame; Based on the centerline acceleration, the jointline acceleration, and the angular acceleration, the target optimized displacement data of the center point and the target optimized rotation data of the joint point are determined, including: Based on the first center displacement data and the first rotation data, the contact points among the joints are determined, and the contact points are the joints that are in contact with the ground. Based on the contact point, determine the ground reaction force between the contact point and the ground; Based on the centerline acceleration, the jointline acceleration, the angular acceleration, the contact point, and the ground reaction force, determine the optimized centerline acceleration of the center point and the optimized angular acceleration of the joint point; Based on the optimized centerline acceleration and the optimized angular acceleration, determine the optimized displacement data of the center point and the optimized rotation data of the joint point; Based on the optimized displacement data, optimized rotation data, first center displacement data, and first rotation data, the target optimized displacement data of the center point and the target optimized rotation data of the joint point are determined.
2. The method according to claim 1, characterized in that, The centerline acceleration and the jointline acceleration satisfy: in, This is the set of the centerline acceleration and the jointline acceleration. and These are the proportional-differential coefficients. It is the set of the sampling center displacement data and the sampling joint displacement data. This is a set of the second center displacement data and the second joint displacement data. It is the set of the second centerline velocity and the second jointline velocity.
3. The method according to claim 1, characterized in that, The angular acceleration of the joint point satisfies: in, For the angular acceleration, and These are the proportional-differential coefficients. For the first rotation data, This is the second rotation data. This is the second angular velocity.
4. The method according to claim 1, characterized in that, The optimized centerline acceleration and the optimized angular acceleration satisfy: , in, Indicates the parameter to be optimized is The objective function, This is a set of optimized center displacement data for the center point and optimized rotation data for the joint points. This is the set of optimized centerline velocity of the center point and optimized angular velocity of the joint points. This is the set of optimized centerline acceleration at the center point and optimized angular acceleration at the joint points. To optimize angular acceleration, This is the set of control torques at the center point and control torques at the joint points. For Jacobian matrices, For preset coefficients, The ground reaction force, The control torque at the center point, The control torque at the joint point, Indicates a restriction condition. According to The Jacobian matrix of the contact point is obtained. Represents the transpose of a matrix. According to The obtained inertia matrix, According to The obtained gravity, centrifugal force, and Coriolis force According to The obtained contact linear velocity at the contact point, For the contact line velocity to satisfy The set, The set of preset ranges for the contact linear velocity. To characterize having Number of , It is a three-dimensional data format. The number of contact points. For use in characterizing three-dimensional data formats, For the contact point at The contact linear velocity on the shaft, For the contact point at The contact linear velocity on the shaft, For the contact point at The contact linear velocity on the shaft, This is a preset threshold.
5. The method according to claim 1, characterized in that, The step of determining the optimized displacement data of the center point and the optimized rotation data of the joint points based on the optimized centerline acceleration and the optimized angular acceleration includes: The optimized displacement data of the center point is determined based on the optimized centerline acceleration and the time interval between the first time frame and the second time frame. The optimized rotation data of the joint is determined based on the optimized angular acceleration and the time interval between the first and second time frames.
6. The method according to claim 1, characterized in that, After determining the target optimized displacement data of the center point and the target optimized rotation data of the joint point based on the centerline acceleration, the jointline acceleration, and the angular acceleration, the method further includes: Based on the target optimized displacement data, target optimized rotation data, and the second center displacement data and second rotation data, the motion path of the target object is determined; Display the movement path of the target object.
7. A video motion capture optimization device based on ground contact sensing and dynamics, characterized in that, include: The acquisition module is used to acquire the first center displacement data of the center point on the target object, the first joint displacement data of the joint point, and the first rotation data of the joint point, wherein the first center displacement data, the first joint displacement data, and the first rotation data are data of the target object in the first time frame; The sampling module is used to sample the first center displacement data and the first joint displacement data within a preset data adjustment range to obtain the sampled center displacement data of the center point and the sampled joint displacement data of the joint point. The first determining module is used to determine the centerline acceleration of the center point and the jointline acceleration of the joint point based on the sampling center displacement data, the sampling joint displacement data, the second center displacement data and the second centerline velocity of the center point, and the second joint displacement data and the second jointline velocity of the joint point, wherein the second center displacement data, the second joint displacement data, the second centerline velocity, and the second jointline velocity are data of the target object in the second time frame, and the second time frame is the time frame preceding the first time frame; The second determining module is used to determine the angular acceleration of the joint point based on the first rotation data, the second rotation data of the joint point, and the second angular velocity of the joint point, wherein the second rotation data and the second angular velocity are data of the target object in the second time frame; The target module is used to determine the target optimized displacement data of the center point and the target optimized rotation data of the joint point based on the centerline acceleration, the jointline acceleration, and the angular acceleration. The target module is further configured to: determine a contact point among the joints based on the first center displacement data and the first rotation data, wherein the contact point is a joint that contacts the ground; determine the ground reaction force between the contact point and the ground based on the contact point; determine the optimized centerline acceleration of the center point and the optimized angular acceleration of the joint based on the centerline acceleration, the jointline acceleration, the angular acceleration, the contact point, and the ground reaction force; determine the optimized displacement data of the center point and the optimized rotation data of the joint based on the optimized centerline acceleration and the optimized angular acceleration; and determine the target optimized displacement data of the center point and the target optimized rotation data of the joint based on the optimized displacement data, the optimized rotation data, the first center displacement data, and the first rotation data.
8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.
Citation Information
Patent Citations
Motion capture method and system based on parameterized model
CN117541646A
Weighted golf club head
US5409219A
