A motion capture method and system based on parameterized model

Through the motion capture method based on the parametric model, RGB video and binary classification algorithms are used to capture characters and hand movements, combined with the inverse kinematics optimization algorithm to eliminate foot sliding, solving the problems of hand neglect and foot drift in the existing technology. , achieving improvements in full-body motion capture and real effects.

CN117541646BActive Publication Date: 2024-11-08DARK MATTER ARTIFICIAL INTELLIGENT (BEIJING) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311754272.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-20
Publication Date
2024-11-08
Estimated Expiration
2043-12-20

AI Technical Summary

Technical Problem

Most existing human motion capture technologies ignore hand movements, and there are problems with foot drift and sliding, which affects the visual sense and makes it difficult to capture whole-body movements and achieve real action restoration.

Method used

Using a motion capture method based on parametric model, data is collected through RGB video or RGBD video, character and hands position are located, and the rotation value of the node is captured using binary classification algorithm and human parametric three-dimensional model, combining absolute position estimation algorithm and inverse kinematics Optimize algorithm to eliminate foot sliding and optimize coordinate information.

Benefits of technology

It realizes an inexpensive motion capture system that can capture hand details and whole-body movements, eliminate foot sliding, and provide a more realistic and stable motion capture effect. It is suitable for multi-view scenes and virtual digital human models with different bone structures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117541646B_ABST
    Figure CN117541646B_ABST
Patent Text Reader

Abstract

The invention discloses a motion capture method and system based on a parameterized model. A human body detection module collects RGB video or RGBD video with depth information to obtain a position boundary frame of a target person and both hands. A foot contact detection module obtains a binary classification result of both feet of the person by using a binary classification algorithm model according to a regional image in the boundary frame of the target person and a regional image in the boundary frame of both hands of the target person. A human body posture capture module uses a parameterized three-dimensional model of the human body to capture and estimate the rotation value of each joint point of the human body. An absolute position estimation module obtains the 3D coordinates of the target person in a camera coordinate system by using an absolute position estimation algorithm. A data optimization module obtains the rotation value of each joint point of the human body, the coordinates of the human body in the camera coordinate system, and the binary classification result of whether both feet of the person are grounded, and obtains the rotation value of eliminating foot sliding and floating and the coordinates of the human body in the camera coordinate system after optimization by using a mean filtering process and an inverse kinematics optimization algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and human motion capture, and more particularly to a motion capture method and system based on a parameterized model. Background Art

[0002] At present, human motion capture methods are technologies urgently needed for digital humans and the metaverse, and there are many mature solutions. The current human motion capture technology can capture relatively precise movements without wearing any equipment, only with a camera. Compared with motion capture methods that require wearable equipment, the cost of motion capture is reduced.

[0003] However, most methods of motion capture that rely on cameras only focus on the limbs and ignore the movements of the hands. In addition, the results of motion capture often have negative effects such as foot drifting and sliding, which affects the viewing experience.

[0004] Therefore, how to capture the movements of the whole body while eliminating the sliding of the feet and restore more realistic movements is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a motion capture method and system based on a parameterized model to solve some of the technical problems mentioned in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A motion capture method based on a parameterized model comprises the following steps:

[0008] S1. Collect RGB video or RGBD video with depth information;

[0009] S2. Locate the position of the target person in the video image, and simultaneously locate the position of the target person's hands, and obtain the position bounding box of the target person and his hands;

[0010] S3. Based on the regional image in the target person's bounding box, a binary classification algorithm model is used to obtain the binary classification results of the person's feet, and determine whether the person's feet are in contact with the ground;

[0011] According to the regional images in the target person's bounding box and the regional images in the target person's hands' bounding boxes, the human body parameterized three-dimensional model is used to capture and estimate the rotation values ​​of each joint point of the human body;

[0012] According to the regional image in the target person's bounding box, the 3D coordinates of the target person in the camera coordinate system are obtained through the absolute position estimation algorithm to estimate the displacement information of the target person;

[0013] S4. According to the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the feet of the person are grounded, the optimized rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained through mean filtering and inverse kinematics optimization algorithm.

[0014] Preferably, obtaining the position bounding box of the target person and his hands in S2 is implemented by YOLO, a mainstream target detection algorithm.

[0015] Preferably, the binary classification algorithm model in step S3 includes a multi-layer perceptron MLP, which includes an input layer, three hidden layers, and an output layer. All five layers of the network are fully connected layers, and the loss function of the binary classification algorithm model adopts a binary classification cross entropy loss function.

[0016] Preferably, the human body parameterized three-dimensional model in step S3 includes an encoder, a spatial feature pyramid network, and a regressor;

[0017] After the input image is output by the encoder as a feature map containing rich semantic information, it is input into the spatial pyramid network to further extract features. Finally, the input parameters and estimated camera parameters required for the human body parametric 3D model are output through the regressor. The input parameters are the rotation values ​​of each bone point. The 3D key point positions and 2D key point positions of the human body are obtained through forward kinematics and camera parameters, and are used to calculate the loss function. The human body parametric 3D model is trained by reconstructing the loss function.

[0018] Preferably, the reconstruction loss function for training the human body parameterized three-dimensional model is specifically:

[0019] L reg =λ 2d ||KK gt ||+λ 3d ||JJ gt ||+λ para ||Θ-Θ gt ||

[0020] Among them, K represents the position of 2D key points, J represents the position of 3D key points, θ represents the input parameters of the parameterized 3D model of the human body and the camera parameters, λ represents the weights of different parts, and ||·|| represents the L2 norm.

[0021] Preferably, the absolute position estimation algorithm in step S3 includes a backbone network and two regressors, wherein the backbone network is composed of multiple convolutional layers, and the regressors are mainly composed of fully connected layers. The image is subjected to the backbone network to extract features and is input into two regressors respectively to estimate the camera parameters and the 3D coordinates relative to the root node, and then the 3D coordinates relative to the root node are converted into absolute 3D coordinates of the camera coordinate system through the estimated camera parameters.

[0022] Preferably, the loss function of the absolute position estimation algorithm is the L1 norm, specifically:

[0023] L=||RR gt ||1

[0024] Where R represents the absolute 3D coordinates of the camera coordinate system.

[0025] Preferably, step S2 includes a single-player mode or a multi-player mode, wherein in the single-player mode, if multiple people appear in the picture, only the one whose bounding box occupies the largest proportion of the picture is output; in the multi-view mode, the bounding boxes of the same person in different view angles are matched by a matching algorithm to locate the position of the same person in different view angles;

[0026] In step S3, in multi-view mode, the binary classification results of multiple viewpoints are summarized, and the classification results that account for the majority in each viewpoint are used as the binary classification results of the character's feet; the rotation values ​​output by each viewpoint are output as the final rotation values ​​through a multi-view fusion algorithm; the average of the 3D coordinates estimated from multiple viewpoints is the 3D coordinate of the target character in the camera coordinate system.

[0027] Preferably, the specific contents of step S4 include:

[0028] S41. The rotation values ​​of each joint of the human body and the coordinates of the human body in the camera coordinate system are processed by mean filtering to eliminate jitter information in the data;

[0029] S42. Calculate the position of the new foot key point by interpolation method according to the binary classification result of whether the left and right feet of the human body are in contact with the ground;

[0030] S43. Taking the position of the new foot key point as a constraint, the rotation value of each joint point of the human body is optimized by iterative numerical optimization.

[0031] A motion capture system based on a parameterized model, comprising a human body detection module, a foot contact detection module, a human body posture capture module, an absolute position estimation module and a data optimization module;

[0032] The human body detection module is used to locate the position of the target person in the video screen by collecting RGB video or RGBD video with depth information, and locate the position of the target person's hands at the same time, and obtain the position boundary box of the target person and his hands;

[0033] The foot contact detection module uses a binary classification algorithm model to obtain the binary classification results of the person's feet based on the regional image in the target person's boundary box, and determines whether the person's feet are in contact with the ground;

[0034] A human posture capture module is used to capture and estimate the rotation value of each joint point of the human body by using a parametric 3D model of the human body according to the regional image in the target person's bounding box and the regional image in the target person's hands' bounding box;

[0035] The absolute position estimation module is used to obtain the 3D coordinates of the target person in the camera coordinate system based on the regional image in the target person's boundary box through an absolute position estimation algorithm, and estimate the displacement information of the target person;

[0036] The data optimization module uses the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the person's feet are grounded. Through mean filtering and inverse kinematics optimization algorithm, the optimized rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained.

[0037] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a motion capture method and system based on a parameterized model, which has the following advantages:

[0038] Use low-cost home RGB cameras to achieve motion capture and drive digital virtual humans, which is simple and quick to deploy.

[0039] It provides an end-to-end motion capture system and data optimization method. In actual operation, you only need to input RGB video to obtain optimized motion capture data, achieving a more realistic effect.

[0040] The optional multi-view solution makes the motion capture results more accurate and stable;

[0041] It can capture detailed information of the hands, which can be applied to more practical scenarios when combined with body information;

[0042] The data optimization method can further optimize the results of motion capture to achieve a more anthropomorphic and realistic driving effect. It has a fast processing speed, flexible settings, and universality. After simple modifications, it can be applied to virtual digital human models with different bone structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 The accompanying drawing is a schematic diagram of a motion capture method based on a parameterized model provided by the present invention;

[0045] Figure 2 The accompanying drawing is a schematic diagram of a human body posture estimation method based on a parameterized model provided by the present invention;

[0046] Figure 3 The accompanying drawing is a schematic diagram of a motion capture system based on a parameterized model provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The embodiment of the present invention discloses a motion capture method based on a parameterized model. Figure 1 , including the following steps:

[0049] S1. Collect RGB video or RGBD video with depth information;

[0050] S2. Locate the position of the target person in the video image, and simultaneously locate the position of the target person's hands, and obtain the position bounding box of the target person and his hands;

[0051] S3. Based on the regional image in the target person's bounding box, a binary classification algorithm model is used to obtain the binary classification results of the person's feet, and determine whether the person's feet are in contact with the ground;

[0052] According to the regional images in the target person's bounding box and the regional images in the target person's hands' bounding boxes, the human body parameterized three-dimensional model is used to capture and estimate the rotation values ​​of each joint point of the human body;

[0053] According to the regional image in the target person's bounding box, the 3D coordinates of the target person in the camera coordinate system are obtained through the absolute position estimation algorithm to estimate the displacement information of the target person;

[0054] S4. According to the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the feet of the person are grounded, the optimized rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained through mean filtering and inverse kinematics optimization algorithm.

[0055] In order to further implement the above technical solution, the position bounding box of the target person and his hands is obtained in S2 through the mainstream target detection algorithm YOLO.

[0056] In this embodiment, the video captured by the camera is passed through the mainstream target detection algorithm YOLO frame by frame, and the position of the target person in each frame and the position of his hands are output, where the position is represented by a bounding box, and the person and the hands will be completely included in the bounding box. To ensure this, the output bounding box is expanded as a whole.

[0057] In order to further implement the above technical solution, the binary classification algorithm model in step S3 includes a multi-layer perceptron MLP, which includes an input layer, three hidden layers, and an output layer. The five-layer network is a fully connected layer, and the loss function of the binary classification algorithm model adopts a binary classification cross entropy loss function.

[0058] In order to further implement the above technical solution, Figure 2 ,In step S3, the parameterized 3D model of the human body includes an encoder, a spatial feature pyramid network, and a regressor;

[0059] After the input image is output by the encoder as a feature map containing rich semantic information, it is input into the spatial pyramid network to further extract features. Finally, the input parameters and estimated camera parameters required for the human body parametric 3D model are output through the regressor. The input parameters are the rotation values ​​of each bone point. The 3D key point positions and 2D key point positions of the human body are obtained through forward kinematics and camera parameters, and are used to calculate the loss function. The human body parametric 3D model is trained by reconstructing the loss function.

[0060] In order to further implement the above technical solution, the reconstruction loss function for training the parametric 3D model of the human body is specifically:

[0061] L reg =λ 2d ||KK gt ||+λ 3d ||JJ gt ||+λ para ||Θ-Θ gt ||

[0062] Among them, K represents the position of 2D key points, J represents the position of 3D key points, θ represents the input parameters of the parameterized 3D model of the human body and the camera parameters, λ represents the weights of different parts, and ||·|| represents the L2 norm.

[0063] In order to further implement the above technical solution, the absolute position estimation algorithm in step S3 includes a backbone network and two regressors, wherein the backbone network is composed of multiple convolutional layers, and the regressors are mainly composed of fully connected layers. The image is extracted through the backbone network and input into the two regressors respectively to estimate the camera parameters and the 3D coordinates relative to the root node, and then the 3D coordinates relative to the root node are converted into absolute 3D coordinates of the camera coordinate system through the estimated camera parameters.

[0064] In order to further implement the above technical solution, the loss function of the absolute position estimation algorithm is the L1 norm, specifically:

[0065] L=||RR gt ||1

[0066] Where R represents the absolute 3D coordinates of the camera coordinate system.

[0067] In order to further implement the above technical solution, step S2 includes a single-player mode or a multi-player mode, wherein in the single-player mode, if there are multiple people in the picture, only the one whose bounding box occupies the largest proportion of the picture is output; in the multi-view mode, the bounding boxes of the same person in different perspectives are matched by a matching algorithm to locate the position of the same person in different perspectives;

[0068] In step S3, in multi-view mode, the binary classification results of multiple viewpoints are summarized, and the classification results that account for the majority in each viewpoint are used as the binary classification results of the character's feet; the rotation values ​​output by each viewpoint are output as the final rotation values ​​through a multi-view fusion algorithm; the average of the 3D coordinates estimated from multiple viewpoints is the 3D coordinate of the target character in the camera coordinate system.

[0069] In order to further implement the above technical solution, the specific content of step S4 includes:

[0070] S41. The rotation values ​​of each joint of the human body and the coordinates of the human body in the camera coordinate system are processed by mean filtering to eliminate jitter information in the data;

[0071] In actual applications, filtering processing has a good effect only on jitter with a mean value of 0. When driving a virtual digital human, the feet may slide and float. To address this, an optimization algorithm based on inverse kinematics is further used:

[0072] S42. Calculate the position of the new foot key point by interpolation method according to the binary classification result of whether the left and right feet of the human body are in contact with the ground;

[0073] S43. Taking the position of the new foot key point as a constraint, the rotation value of each joint point of the human body is optimized by iterative numerical optimization.

[0074] A motion capture system based on a parametric model, such as Figure 3 , including a human body detection module, a foot contact detection module, a human body posture capture module, an absolute position estimation module and a data optimization module;

[0075] The human body detection module is used to locate the position of the target person in the video screen by collecting RGB video or RGBD video with depth information, and locate the position of the target person's hands at the same time, and obtain the position boundary box of the target person and his hands;

[0076] The foot contact detection module uses a binary classification algorithm model to obtain the binary classification results of the person's feet based on the regional image in the target person's boundary box, and determines whether the person's feet are in contact with the ground;

[0077] A human posture capture module is used to capture and estimate the rotation value of each joint point of the human body by using a parametric 3D model of the human body according to the regional image in the target person's bounding box and the regional image in the target person's hands' bounding box;

[0078] The absolute position estimation module is used to obtain the 3D coordinates of the target person in the camera coordinate system based on the regional image in the target person's boundary box through an absolute position estimation algorithm, and estimate the displacement information of the target person;

[0079] The data optimization module uses the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the person's feet are grounded. Through mean filtering and inverse kinematics optimization algorithm, the optimized rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained.

[0080] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a motion capture method based on a parameterized model.

[0081] A processing terminal comprises a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a motion capture method based on a parameterized model is implemented.

[0082] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0083] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A motion capture method based on a parameterized model, characterized in that: The following steps are involved: S1. Collect RGB video or RGBD video with depth information; S2. Locate the position of the target person in the video screen, and locate the position of the target person's hands, and obtain the bounding box of the target person and the position bounding box of his hands; S3. Based on the regional image in the target person's bounding box, a binary classification algorithm model is used to obtain the binary classification results of the person's feet, and determine whether the person's feet are in contact with the ground; According to the regional images in the target person's bounding box and the regional images in the target person's hands' bounding boxes, the human body parameterized three-dimensional model is used to capture and estimate the rotation values ​​of each joint point of the human body; According to the regional image in the target person's bounding box, the 3D coordinates of the target person in the camera coordinate system are obtained through the absolute position estimation algorithm to estimate the displacement information of the target person; S4. According to the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the feet of the person are grounded, the rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained through mean filtering and inverse kinematics optimization algorithm; In step S3, the binary classification algorithm model includes a multi-layer perceptron MLP, which includes an input layer, three hidden layers, and an output layer. The five-layer network is a fully connected layer, and the loss function of the binary classification algorithm model adopts a binary classification cross entropy loss function; The human body parameterized 3D model in step S3 includes an encoder, a spatial feature pyramid network, and a regressor; After the input image is output by the encoder with a feature map containing rich semantic information, it is input into the spatial pyramid network to further extract features. Finally, the input parameters and estimated camera parameters required for the human body parametric 3D model are output through the regressor. The input parameters are the rotation values ​​of each bone point. The 3D key point positions and 2D key point positions of the human body are obtained through forward kinematics and camera parameters, and are used to calculate the loss function. The human body parametric 3D model is trained by reconstructing the loss function. In step S3, the absolute position estimation algorithm includes a backbone network and two regressors, wherein the backbone network is composed of multiple convolutional layers, and the regressors are mainly composed of fully connected layers. The image is extracted through the backbone network and input into the two regressors respectively, and the camera parameters and the 3D coordinates relative to the root node are estimated respectively. Then, the 3D coordinates relative to the root node are converted into the absolute 3D coordinates of the camera coordinate system through the estimated camera parameters; Step S2 includes a single-player mode or a multi-player mode. In the single-player mode, if multiple people appear in the picture, only the one whose bounding box occupies the largest proportion of the picture is output; in the multi-view mode, the bounding boxes of the same person in different views are matched by a matching algorithm to locate the position of the same person in different views; In step S3, in the multi-view mode, the binary classification results of multiple viewpoints are summarized, and the classification results that account for the majority in each viewpoint are used as the binary classification results of the person's feet; the rotation values ​​output by each viewpoint are output as the final rotation value through the multi-view fusion algorithm; the average of the 3D coordinates estimated by multiple viewpoints is the 3D coordinate of the target person in the camera coordinate system; The reconstruction loss function for training the parametric 3D model of the human body is: L reg =λ 2d ||KK g ||+λ 3d ||JJ gt ||+λ para ||I-I gt || Among them, K represents the position of 2D key points, J represents the position of 3D key points, θ represents the input parameters of the parameterized 3D model of the human body and the camera parameters, λ represents the weights of different parts, and · represents the L2 norm.

2. The motion capture method based on a parameterized model according to claim 1, characterized in that: In S2, the position bounding box of the target person and his hands is obtained by the mainstream target detection algorithm YOLO.

3. The motion capture method based on parameterized model according to claim 1, characterized in that: The loss function of the absolute position estimation algorithm is the L1 norm, specifically: L=||R-R gt ||1 Where R represents the absolute 3D coordinates of the camera coordinate system.

4. The motion capture method based on parameterized model according to claim 1, characterized in that: The specific contents of step S4 include: S41. The rotation values ​​of each joint of the human body and the coordinates of the human body in the camera coordinate system are processed by mean filtering to eliminate jitter information in the data; S42. Calculate the position of the new foot key point by interpolation method according to the binary classification result of whether the left and right feet of the human body are in contact with the ground; S43. Taking the position of the new foot key point as a constraint, the rotation value of each joint point of the human body is optimized by iterative numerical optimization.

5. A motion capture system based on a parameterized model, characterized in that: A motion capture method based on a parameterized model according to any one of claims 1 to 4, comprising a human body detection module, a foot contact detection module, a human body posture capture module, an absolute position estimation module and a data optimization module; The human body detection module is used to locate the position of the target person in the video screen by collecting RGB video or RGBD video with depth information, and locate the position of the target person's hands at the same time, and obtain the position boundary box of the target person and his hands; The foot contact detection module uses a binary classification algorithm model to obtain the binary classification results of the person's feet based on the regional image in the target person's boundary box, and determines whether the person's feet are in contact with the ground; A human posture capture module is used to capture and estimate the rotation value of each joint point of the human body by using a parametric 3D model of the human body according to the regional image in the target person's bounding box and the regional image in the target person's hands' bounding box; The absolute position estimation module is used to obtain the 3D coordinates of the target person in the camera coordinate system based on the regional image in the target person's boundary box through an absolute position estimation algorithm, and estimate the displacement information of the target person; The data optimization module uses the binary classification results of the rotation values ​​of each joint of the human body, the coordinates of the human body in the camera coordinate system, and whether the person's feet are grounded. Through mean filtering and inverse kinematics optimization algorithm, the optimized rotation values ​​and the coordinates of the human body in the camera coordinate system that eliminate foot sliding and floating are obtained.