Multi-stage human motion capture method and device based on monocular video and medium

Abstract

A multi-stage human motion capture method and device based on monocular video and a medium, for monocular RGB video captured in a fixed camera situation, human motion capture is divided into multiple stages: in the first stage, a human pose detector is used to estimate two-dimensional human joint points frame by frame for the input monocular video; in the second stage, deep learning is used to obtain spatial and temporal information in the video sequence, learn the mapping relationship from two-dimensional human joint points to three-dimensional human joint points in the camera space, and perceive the motion trajectory and ground contact of the human in the three-dimensional space; in the third stage, the idea of inverse kinematics is introduced, a reasonable punishment strategy is formulated, and a three-dimensional human mesh model is fitted to the three-dimensional skeleton to more realistically depict the motion sequence. The present application significantly improves the alignment degree of the reconstructed human model and the input RGB image, and more realistically estimates the contact between the two feet and the ground, which has obvious improvement in vision.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to human motion capture technology using a monocular camera. Specifically, it describes a method for predicting the 3D rotation of human joints based on monocular RGB video in a scene with a fixed camera position, thereby driving a parametric 3D human model. Background Technology

[0002] Motion capture is an emerging animation technology that accurately estimates human movements while generating corresponding virtual asset motion animations. This technology is now widely used in entertainment, sports, and medical fields. Among these, monocular motion capture technology has a larger market share due to its low equipment requirements and simple deployment, significantly lowering the barrier to entry. With the development of deep learning technology, the accuracy of monocular motion capture technology continues to improve. SMPL (Skinned Multi-Person Linear Model) is a vertex-based 3D human body model that can accurately represent different shapes and poses of the human body. An SMPL model contains 24 joints, including 23 joint nodes and 1 root node.

[0003] In the field of deep learning, existing monocular motion capture methods can be divided into two main categories based on their stage states. The first category is single-stage methods, such as HMR and PyMAF. These methods use neural networks to regress the pose and rotation parameters required for the human model from the original RGB input end-to-end, without explicit intermediate states or supervision. However, the mapping from the original image to the abstract model parameters is highly non-linear, leading to inaccurate alignment between the predicted results and the image. The second category is multi-stage methods, such as NBF and Pose2Mesh. These methods reduce the fitting difficulty at each step of the network by decomposing the task. The network gradually outputs intermediate representations, such as human joints, and continues prediction based on these until the target result is obtained. Because each stage in a multi-stage method uses the output value of the previous stage as input for prediction, the prediction error accumulates stage by stage.

[0004] Furthermore, existing motion capture methods are mostly image-based. For monocular video input, image-based methods can only predict frame by frame. Because they cannot extract and utilize temporal information, the estimated video results often have jitter and are sensitive to occlusion. In addition, mainstream methods use weak perspective projection cameras, which assume the camera is far away from the person and ignore the depth of the human body. This does not match the real scene and cannot represent the characteristics of perspective projection. Moreover, mainstream regression networks simply omit the supervision of the contact between the person and the environment, which can easily lead to visual inconsistencies and unrealistic results. Summary of the Invention

[0005] The problem this invention aims to solve is that the accuracy of the deep learning schemes used in existing monocular video motion capture methods cannot meet the requirements. On the other hand, existing motion capture methods are based on images and do not consider the temporal information and scene depth information of the video, which also affects the effectiveness of motion capture and the accuracy of the capture results.

[0006] The technical solution of this invention is as follows: a multi-stage human motion capture method based on monocular video. For monocular RGB video captured in a fixed camera environment, human motion capture is divided into multiple stages: the first stage performs human pose detection frame by frame on the input monocular video, estimates two-dimensional human key points, and constructs the local space of the three-dimensional human model SMPL in the camera coordinate system; the second stage uses deep learning methods to obtain spatial and temporal information in the video sequence, learns the mapping relationship from two-dimensional human key points in the video frame image to three-dimensional human joints in the camera space, and detects and perceives the human body's motion trajectory and ground contact in three-dimensional space; the third stage, based on inverse dynamics, formulates a penalty strategy to fit the three-dimensional human mesh model onto the three-dimensional skeleton to characterize the motion sequence and achieve motion capture.

[0007] Furthermore, the first stage involves data preprocessing through a preprocessing module. This module detects 2D human keypoints frame-by-frame in the monocular video and normalizes the 2D human keypoint sequence. Additionally, it transforms the coordinates of the 24 joints of the 3D human model SMPL from the world coordinate system to the camera coordinate system. Using the root node as a base, it converts other joints into relative coordinates relative to the root node, constructing a local space. Simultaneously, it uses a clustering method to cluster the heights of the left and right foot joints to estimate the ground height and generate ground contact labels for the left and right foot joints. The second stage involves pose estimation through a camera calibration module, a human pose estimation module, and a human trajectory estimation module. The three stages are as follows: the human pose estimation module estimates the position of the 3D human joints in local space and obtains the 3D human pose sequence based on the 2D human keypoint sequence; the camera calibration module obtains the camera's intrinsic and extrinsic parameters based on monocular video to obtain the video's viewpoint; the human trajectory estimation module predicts the human displacement and ground contact in the camera coordinate system and obtains the 3D human trajectory sequence and ground contact probability based on the 2D human keypoint sequence and the camera's intrinsic and extrinsic parameters; the third stage uses the model fitting module to fit the 3D human pose sequence, 3D human trajectory sequence, and ground contact probability based on inverse dynamics to obtain the 3D human model motion sequence.

[0008] The present invention also provides an electronic device, including a storage medium and a processor, for performing human motion capture based on monocular video. The storage medium is used to store a computer program, and the processor is used to execute the computer program. When the computer program is executed, it implements the above-described multi-stage human motion capture method based on monocular video.

[0009] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed, implements the above-described multi-stage human motion capture method based on monocular video.

[0010] This invention performs motion capture on captured human motion videos in a fixed monocular camera environment, specifically predicting the 3D rotation of human joints to ultimately generate a sequence of driven 3D human models. For monocular video input, this invention investigates how to estimate the motion sequence of a visually realistic 3D human model, i.e., how to efficiently capture long-term temporal information to improve accuracy and reduce inter-frame jitter. Furthermore, this invention proposes estimating the parameters of the perspective projection camera, the rotation of human joints, the displacement in the camera coordinate system, and the ground contact status to improve the capture effect.

[0011] The present invention has the following advantages compared with the prior art.

[0012] 1) Camera Settings: The camera calibration module estimates the camera's focal length and rotation. The estimated camera parameters are within a reasonable range, thus realistically depicting perspective projection rather than weak perspective projection. Furthermore, the camera parameters are fed into the human trajectory estimation module for coordinate system transformation and loss function calculation.

[0013] 2) Network Design: Both the human pose estimation module and the human trajectory estimation module are fully convolutional networks. Dilated convolutions are used to further improve the network's receptive field, enabling prediction of the current frame based on a full fusion of information from previous and subsequent frames. Since human limb movements have local attributes, the encoder of the human pose estimation module encodes the trunk, head, and limbs according to the motion chain grouping. Correspondingly, the decoder's six branches independently output the predicted values ​​of the joints on the trunk, head, and limbs. The decoder of the human trajectory estimation module is divided into two branches, predicting the displacement of the human root node and the ground contact label, respectively.

[0014] 3) Model Fitting Combining Displacement and Ground Touch Tags: Existing monocular model fitting methods typically only consider the joints of the human body itself in their loss functions, such as the distance between the 3D human joint positions and the joint positions output by the human pose estimation module, and the smoothness of joint positions between adjacent frames, without considering the position of the ground or the contact between the person and the ground. This invention estimates the human's motion trajectory and ground touch probability, and fits the ground height in the world coordinate system, which can constrain the height and velocity of the left and right foot joints. Attached Figure Description

[0015] Figure 1 This is an overall framework diagram of the present invention.

[0016] Figure 2It is the output format of 2D human key points estimated by OpenPose.

[0017] Figure 3 It consists of six relatively independent kinetic chains contained in a three-dimensional human body model.

[0018] Figure 4 This is a framework diagram of the human pose estimation module proposed in this invention. (a) is the encoder part, and (b) is the decoder part.

[0019] Figure 5 This is a framework diagram of the human trajectory estimation module proposed in this invention. Detailed Implementation

[0020] This invention proposes a multi-stage human motion capture method based on monocular video. The application scenario is monocular RGB video captured in a fixed camera environment. This invention divides human motion capture into multiple stages. The first stage uses a human pose detector to estimate 2D human keypoints frame-by-frame from the input monocular video. The second stage utilizes deep learning to acquire spatial and temporal information from the video sequence, learning the mapping relationship from 2D human keypoints to 3D human joints in camera space, and perceiving the human's motion trajectory and ground contact in 3D space. The third stage, based on inverse kinematics, uses a penalty strategy to fit a 3D human mesh model onto a 3D skeleton to more realistically depict the motion sequence. This invention significantly improves the alignment between the reconstructed human model and the input RGB image, and more realistically estimates the contact between the feet and the ground, resulting in a significant visual improvement. This invention involves 2D human images and 3D human models; computational nodes on the 2D plane are called "keypoints," and computational nodes in the 3D human model are called "joints."

[0021] The multi-stage human motion capture method of this invention includes the following stages: First, a preprocessing module performs data preprocessing, detecting human keypoints frame-by-frame in monocular video and normalizing the 2D human keypoint sequence. During training of the human posture detector, for labeled data, the 3D human joint position labels are converted to the camera coordinate system. Simultaneously, a clustering method is used to cluster the heights of the left and right foot joints to estimate the ground height, generating ground contact labels for the left and right foot joints. Second, a camera calibration module, a human posture estimation module, and a human trajectory estimation module perform posture estimation. The human posture estimation module predicts the relative positions of human joints in 3D space, obtaining a 3D human posture sequence based on the 2D human keypoint sequence. The camera calibration module acquires the camera's intrinsic and extrinsic parameters based on the monocular video to obtain the video's viewpoint. The human trajectory estimation module predicts human displacement and the ground contact status of each joint in the camera coordinate system, obtaining a 3D human trajectory sequence and ground contact probability based on the 2D human keypoint sequence and the camera's intrinsic and extrinsic parameters. Third, a model fitting module, based on inverse dynamics, fits the 3D human posture sequence, 3D human trajectory sequence, and ground contact probability to obtain a 3D human model motion sequence. Figure 1 This is an overall framework diagram of the present invention, showing the relationship between the preprocessing module, camera correction module, human pose estimation module, human trajectory estimation module, and model fitting module in each stage. The implementation of the present invention is illustrated below through a specific embodiment. This embodiment is trained and tested on the Human3.6M dataset, using the PyTorch 1.12 framework for training and inference of the deep neural network. The pose estimation network is based on the implementation of the VideoPose3D code library, with modifications made to it.

[0022] S10 Data Preprocessing: This stage includes three sub-tasks: extracting 2D human key points, converting 3D human joints, and generating ground touch tags.

[0023] S11 Extraction and Normalization of 2D Human Keypoints: The original video frame rate in the Human3.6M dataset was 50fps, downsampled to 10fps, and frame extraction was performed using ffmpeg. The video frames were then fed into the open-source human pose detector OpenPose to estimate the image coordinate sequence of 25 2D human keypoints. like Figure 2 The key points shown are 0-24, where T is the video frame number. Then, while maintaining the aspect ratio, the x-axis coordinates are normalized from [0, w] to [-1, 1]. Where w and h are the width and height of the video frame.

[0024] S12 Transformation of 3D Human Joints: During the training phase, preprocessing is required to transform the 3D human joint labels from the world coordinate system to the camera coordinate system for supervision; this is known as viewpoint transformation. Let R... cam ,T cam For the camera's rotation matrix and displacement, These are the position coordinates of the 24 joints of the 3D human body model in the world coordinate system, such as... Figure 3 As shown, the following areas are included: H0 pelvis, H1 left hip, H2 right hip, H3 spine 1, H4 left knee, H5 right knee, H6 spine 2, H7 left ankle, H8 right ankle, H9 spine 3, H10 left foot, H11 right foot, H12 neck, H13 left neck, H14 right neck, H15 head, H16 left arm, H17 right arm, H18 left elbow, H19 right elbow, H20 left wrist, H21 right wrist, H22 left hand, and H23 right hand. The conversion formula is: P represents the spatial coordinates of the human body's 3D joints in the camera coordinate system. The dimensions of P are (N, 24, 3), and N represents the number of frames in the video. Finally, the absolute position of the root node in the camera coordinate system is preserved; the root node is... Figure 3 In the H0 pelvic node, a slicing operation is performed, subtracting the spatial coordinates of the root node from the spatial coordinates of other joints to convert them into relative coordinates relative to the root node, i.e., P[:,1:]=P[:,1:]-P[:,:1]. The true value of the 3D trajectory traj is the 3D coordinate sequence of the root node in the camera coordinate system, traj=P[:,:1]. The dimension of traj is (N,1,3), and the depth of the root node is depth=traj[:,:,2]. traj[] represents the slicing operation on traj in the corresponding dimension, where [:] represents the slicing operation along the corresponding dimension.

[0025] S13 generates binary ground-touching labels: For a 3D joint sequence, the displacement velocity of a joint is first calculated by subtracting the values ​​of adjacent frames. If the velocity of a node is less than a set threshold, the joint is considered to be stationary in that frame. Then, the DBSCAN method from machine learning is used to cluster the heights of the stationary left and right foot joints, and the minimum median minus an offset constant is used as the ground height. For the eight joints H10 (left foot), H11 (right foot), H7 (left ankle), H8 (right ankle), H4 (left knee), H5 (right knee), H20 (left wrist), and H21 (right wrist), if the velocity is less than the set threshold and the height difference between the joint and the ground is within a certain range, the joint is considered to be in a ground-touching state, and the ground-touching label c=1.

[0026] S20 Camera Correction Module: We use a pre-trained ResNet-50 model as the backbone network and replace the last layer with three independent MLP head networks, which are used to estimate the camera's vertical field of view (vfov), the rotation pitch along the x-axis, and the rotation roll along the z-axis, respectively. The camera focal length can be calculated from the vfov. The camera's intrinsic parameter matrix is ​​then:

[0027]

[0028] The camera's projection matrix is ​​obtained based on its intrinsic and extrinsic parameters. This projection matrix projects the 3D nodes of the human body predicted by the network onto a 2D plane, obtaining the corresponding 2D coordinates of the joints on the image. This coordinates are then compared with the ground truth values ​​to calculate the distance loss for supervision. Since the processing of the three variables vfov, pitch, and roll is consistent, x represents any one of them. For the prediction of each of vfov, pitch, and roll, a reasonable range of values ​​[min, max] is predefined, and 256 candidate values ​​are uniformly sampled within this range. In terms of algorithm design, the network does not need to regress specific values ​​but rather predicts the probability corresponding to the candidate values. The candidate value corresponding to the position with the highest probability is considered the network's prediction result. In terms of network structure, each of the three MLP head networks outputs a 256-dimensional vector `pred` representing the probability of the candidate value. The `pred` vector can be approximated by the `softargmax` function to obtain the index ID corresponding to the highest probability. soft ∈(0,256), ID soft Normalized to the range [-1, 1] to obtain the normalized index ID soft_normalized Then, the index is mapped to the candidate value range [min, max] to obtain the candidate value at the corresponding position. This refers to the network's prediction results.

[0029]

[0030]

[0031]

[0032] Where pred refers to the probability of the network prediction for the vfov, pitch, or roll branches. This refers to the predicted values ​​of vfov, pitch, or roll. The value range for vfov is [0.2, 1.8], and the value range for pitch is... The range of values ​​for roll is During training, standard methods are used for pitch and roll prediction. Loss function; for VFov prediction, this embodiment uses the Geman-McClure loss function, i.e.

[0033] Furthermore, for real-time motion capture applications, the image captured by the camera at the current moment is sent to the camera correction module to estimate camera parameters, and the estimated values ​​based on past frames are recorded as follows: The network estimates the value x based on the current frame. t Afterwards, The update will be performed, and the update rules are as follows: Where momentum = 0.1. This serves as the latest camera estimate for use in subsequent modules.

[0034] S30 Human Pose Estimation Module: This module is used to estimate the positions of 3D joints in local space. Specifically... Figure 3 The 3D coordinates of the H0 pelvic node are (0,0,0), and the 3D coordinates of other joints are their relative positions to the pelvic node. This module includes two stages: encoding and decoding. In the encoding stage, this module takes the normalized 2D human pose keypoint coordinate sequence (0,…,T-1) obtained by S11 processing as input and extracts features from the input sequence, where the frame number T=27. The features are then fed into the decoder to predict the intermediate frame, i.e., the _th_ frame. The relative positions of human joints in a frame within 3D space. Macroscopically, the encoder in this module is a fully convolutional network. To further increase the receptive field, dilated convolution operations and residual learning are employed. Specifically, since the human skeleton can be divided into six local motion chains, corresponding to... Figure 3 The 24 joints are: head kinematic chain {H12,H13}, root node kinematic chain {H0,H3,H6,H9}, left arm kinematic chain {H12,H14,H17,H19,H21,H23}, right arm kinematic chain {H13,H16,H18,H18,H20,H22}, left leg kinematic chain {H2,H5,H8,H11}, and right leg kinematic chain {H1,H4,H7,H10}. Their movements are relatively independent, such as... Figure 3 As shown, the input is therefore divided into 6 groups along the channel dimension for feature extraction. In the encoder, the output of each network layer is divided into 7 parts, including 6 local features φ. g (g = 1, ..., 6) and a global feature ψ, such as Figure 4 As shown in (a), the output of the i-th layer network is denoted as... From the i-th layer network l i up to the (i+1)th layer of the network l i+1 The calculation process is as follows: global convolution is used to extract global information, and convolution is performed on the {global, grouped} connected features in groups, that is:

[0035]

[0036]

[0037] in, All operations are dilated convolutions with a stride of 3, a kernel size of 3, and a dilation coefficient of 3. `concate` is a concatenation operation along the channel dimension. The encoder ultimately outputs a joint grouped feature. Then in the decoding stage, such as Figure 4 As shown in (b), the grouping characteristic φ g The inputs are fed into 6 independent decoders. Predicting the position of joints on the corresponding local kinematic chain of a 3D human model Right now:

[0038]

[0039] S40 Human Trajectory Estimation Module: This module is used to predict the root node, i.e. Figure 3 The spatial coordinates of the H0 pelvic node in the camera coordinate system are used to obtain the human trajectory. Similar to S30, this module includes two stages: encoding and decoding. This module takes the coordinate sequence (0,…,T-1) of the normalized two-dimensional human pose keypoints obtained by S11 as input, where the number of frames T=27. The difference is that this model only inputs the two-dimensional keypoints on the left and right leg kinematic chains. In the encoding stage, the encoder also uses a dilated convolutional network to extract features from the input sequence; in the decoding stage, such as… Figure 5 As shown, the extracted features are passed through two different head networks to predict intermediate frames, namely the first... 3D displacement of human joints in camera coordinate system in a frame and the probability of ground contact like Figure 5 As shown, one branch of the network estimates a 23-dimensional vector, where each vector represents the ground contact probability of a joint. This is based on the camera intrinsic parameter matrix estimated by the S20 camera correction module. The S30 human pose estimation module predicts the 3D joint coordinates relative to the pelvic node. The S40 human body estimation and prediction module predicts the three-dimensional spatial coordinates of the pelvic nodes. The coordinates of the joints projected onto the two-dimensional image plane can then be obtained. The calculation process is as follows:

[0040]

[0041]

[0042] When training the human pose estimation and trajectory estimation modules, we adopted a combined fully supervised and semi-supervised training approach. In the Human3.6M dataset, data points S1, S5, S6, S7, S8, S9, and S11 have ground truth labels, while S2, S3, and S4 only contain video data without ground truth labels. Therefore, in training the human pose estimation and trajectory estimation modules, we used S1, S5, S6, S7, and S8 for fully supervised training, while simultaneously incorporating S2, S3, and S4 for semi-supervised training. Finally, we used S9 and S11 for testing. The loss function includes the error of the human body's 3D joints. Error of three-dimensional displacement Ground contact probability error Two-dimensional joint error after reprojection Specifically, it is stated as follows:

[0043]

[0044]

[0045]

[0046]

[0047] Where, ||·|2 represents the mean square error function, This represents the bitwise multiplication operation of vectors, BCELoss represents the binary cross-entropy, and σ(·) represents the sigmoid function. Let be the hit-and-ground probability estimate, and c be the ground truth value of the hit-and-ground label. We apply an algorithm to the labeled sample data. and Penalty, imposed on unlabeled samples Penalty. To ensure stable updates of network parameters in the early stages of training, fully supervised training is performed using only labeled data for the first 10 epochs. In subsequent epochs, unlabeled data and reprojection loss are introduced for semi-supervised learning.

[0048] S50 Model Fitting Module: In this module, SMPL is used to represent the human body. For example... Figure 3 As shown, it is a parametric human body model, and the entire human body mesh can be controlled through shape parameters and pose parameters. Shape parameters This represents the coefficients of the first 10 principal components after principal component analysis (PCA), attitude parameters. It depicts a human skeleton composed of 24 nodes, among which... This represents the rotation of node k relative to its parent node, where k = 1, ..., 23. This represents the global rotation amount. SMPL provides a differentiable function. The shape parameter θ and the pose parameter β are taken as inputs, and the output is a 3D mesh with N = 6980 vertices. 3D joint position P of SMPL smpl This can be obtained through the forward kinematics principle (FK), i.e., P smpl =FK(θ,β,T), where T represents the initial state of the human skeleton.

[0049] The model fitting module iteratively optimizes the shape parameters and joint rotations of the human body mesh model step by step based on the predicted 3D human posture sequence, 3D human trajectory sequence, and ground contact probability. This optimization is then assigned to the parameterized 3D human body model to obtain realistic driving results. The process consists of four steps: 1) Fitting human body shape parameters, which determines the length of each bone; 2) Calculating the global rotation of the human body, which determines the orientation; 3) Calculating the rotation of each joint along the skeletal kinematic chain, which determines the rotation of each joint and thus represents the moving 3D human body; 4) Projecting the 3D joints onto a 2D plane and constraining them with detected 2D keypoints to further optimize the alignment between the 3D model and the 2D image. This module fits the 3D human body model to the 3D joints predicted by the S40 human posture estimation module to more realistically represent the motion state. The fitting process is divided into four stages. In each stage, the parameters to be updated and the objective function are defined, the number of iteration cycles is set, and the L-BFGS optimizer is used for iterative updates.

[0050] S51 Iterative Fitting of 3D Human Shape Parameters: During real-time motion capture, the length of the human skeleton is dynamically updated. The image captured by the camera at the current moment is sent to the human pose estimation module to estimate the coordinates of 3D joints. Clearly, the length L of each bone in the human body can be calculated based on the coordinates of the human joints. Let the estimated length of the human skeleton based on past frames be denoted as . The human pose estimation module estimates the value L based on the current frame. t Afterwards, we The update will be performed, and the update rules are as follows: This is the estimated value for the current frame. It is calculated using all 3D keypoint positions P. smpl Easily obtain skeletal vectors Will The shape parameter θ is considered as the fitting target, and the parameters to be updated are the shape parameters. The loss function is defined as follows, where λ1=1, λ2=0.005, and the final fitting result is obtained after 100 iterations.

[0051]

[0052] S52 calculates the global rotation R0: R0 corresponds to w0 in the attitude parameter β. In the SMPL model, H1 (left hip), H2 (right hip), and H3 (spine) define a rigid structure. Based on the initial positions (t1, t2, t3) and predicted positions (p1, p2, p3) of these three joints, a rotation matrix R is obtained to minimize the sum of the distances between the vector (Rt1, Rt2, Rt3) obtained after the rotation transformation of (t1, t2, t3) and the predicted positions (p1, p2, p3). The R that minimizes the distance is R0. R0 is calculated, and the closed-form solution of this equation can be obtained through singular value decomposition (SVD).

[0053]

[0054] Where SO(3)∈{R|R T R = I, det(R) = ±1} represents a three-dimensional rotation group consisting of three-dimensional rotation matrices.

[0055] S53 calculates the rotation of each joint: The parent node in the 3D human body model is defined along the human skeletal chain, such as... Figure 3 As shown, H0 is the parent node of H1, H2, and H3; H1 is the parent node of H4; H2 is the parent node of H5; H3 is the parent node of H6, and so on. Let t be the initial position of node k in the 3D human body model. k The target location is p k In the reconstructed model, the position of node k is q. k Let the skeletal vector containing node k be . Attitude parameter w k R represents the rotation of node k relative to its parent node f(k). f(k),k Then the absolute rotation of node k First, align the root node of the model (i.e., the H0 pelvic node), so q0 = p0. After obtaining R0 in step S52, move along the tree-like kinematic chain of the 3D human model from the root node to the leaf node, and calculate the relative rotation R of each node step by step. f(k),k If the node currently being considered is k, the position of the parent node of the reconstructed node k is... The initial bone vector is After canceling out the rotation effect of the parent node, the target bone vector is obtained as follows: Let the axis of rotation be n and the angle of rotation be α, then The rotation matrix is ​​obtained using Rodrigues' formula:

[0056]

[0057] Where I represents a 3×3 identity matrix, n∧ Let R be the antisymmetric matrix of n. Then the relative rotation R... f(k),k That is

[0058] S54 optimizes based on detected 2D keypoints and ground-touching tags: The shape parameter β and pose parameter θ obtained through steps S51, S52, and S53 yield the corresponding SMPL model. The SMPL model is then linearly mapped and regressed to derive the corresponding OpenPose keypoints (see...). Figure 2 The three-dimensional coordinates J 3D Then, combined with the camera intrinsic parameter matrix estimated by the camera correction module The coordinates J of the two-dimensional key points after the SMPL model is projected can then be obtained. 2D The coordinates J of the two-dimensional key points regressed by the SMPL model. 2D The coordinates of the two-dimensional keypoints should be as close as possible to the input coordinates X. This step combines the two-dimensional keypoint detection result X generated in step S11 with the ground contact probability predicted by the human trajectory prediction module. Further iterative optimization of the human model SMPL is performed. The parameters to be updated are the pose parameter β and the shape parameter θ. The loss function is defined as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] Here, thresh is the set penalty threshold. For example, if a joint is judged to be touching the ground, but its height is higher than this threshold, it will be penalized. λ3 = 0.001, λ4 = 10, λ5 = 0.01, and the final optimized result is obtained after 100 iterations.

[0066] Based on the obtained SMPL model parameters and rotation parameters, the 3D human body mesh model is fitted onto the 3D skeleton to achieve motion capture.

[0067] In summary, this invention provides a multi-stage human motion capture method based on monocular video. Its advantage lies in combining video temporal sequence and scene depth, capturing dynamic human movements through a single camera, making it an economical and efficient solution. This invention is implemented based on a computer program, thus enabling the creation of corresponding human motion capture electronic devices or related computer-readable storage media. This invention utilizes deep learning to acquire spatial and temporal information from video sequences, learning the mapping relationship from two-dimensional human joints to three-dimensional human joints in camera space, and perceiving the human's movement trajectory in three-dimensional space. Then, a three-dimensional human mesh model is fitted onto a three-dimensional skeleton. This invention significantly improves the alignment between the human model and the input image and more realistically estimates the contact between the feet and the ground, enhancing the visual effect.

Claims

1. A multi-stage human motion capture method based on monocular video, characterized by: For monocular RGB video captured in a fixed camera context, human motion capture is divided into multiple stages: the first stage involves detecting human pose frame by frame in the input monocular video, estimating two-dimensional human key points, and constructing the local space of the three-dimensional human model SMPL in the camera coordinate system. The second stage uses deep learning methods to obtain spatial and temporal information in video sequences, learns the mapping relationship from two-dimensional human keypoints in video frame images to three-dimensional human joints in camera space, and detects and perceives the human body's motion trajectory and ground contact in three-dimensional space; the third stage, based on inverse dynamics, formulates a penalty strategy to fit a three-dimensional human mesh model onto a three-dimensional skeleton to characterize the motion sequence and achieve motion capture. The first stage involves data preprocessing through a preprocessing module. This module detects 2D human keypoints frame-by-frame in the monocular video and normalizes the 2D human keypoint sequence. Additionally, it transforms the coordinates of the 24 joints of the 3D human model SMPL from the world coordinate system to the camera coordinate system. Using the root node as a base, it converts other joints into relative coordinates relative to the root node, constructing a local space. Simultaneously, it uses a clustering method to cluster the heights of the left and right foot joints to estimate the ground height and generate ground contact labels for the left and right foot joints. The second stage involves pose estimation through a camera correction module, a human pose estimation module, and a human trajectory estimation module. The body pose estimation module estimates the position of the 3D human joints in local space, obtaining the 3D human pose sequence based on the 2D human keypoint sequence. The camera calibration module acquires the camera's intrinsic and extrinsic parameters based on monocular video to obtain the video's viewpoint. The human trajectory estimation module predicts the human displacement and ground contact in the camera coordinate system, obtaining the 3D human trajectory sequence and ground contact probability based on the 2D human keypoint sequence and the camera's intrinsic and extrinsic parameters. In the third stage, the model fitting module, based on inverse dynamics, fits the 3D human pose sequence, 3D human trajectory sequence, and ground contact probability to obtain the 3D human model motion sequence. The model fitting module iteratively optimizes the shape parameters and joint rotation of the human body mesh model step by step based on the predicted 3D human posture sequence, 3D human trajectory sequence, and ground contact probability, and assigns these parameters to the parameterized 3D human body model to obtain the realistic driving results. This process consists of four steps, as follows: S51 Iterative Fitting of 3D Human Shape Parameters: During real-time motion capture, the length of the human skeleton is dynamically updated, and the human pose estimation module estimates the 3D joint coordinates by inputting video frames. The lengths of each bone in the human body were calculated. Let the estimated length of the human skeleton based on past frames be denoted as . The human pose estimation module calculates an estimate based on the current frame. Afterwards, The update will be performed, and the update rules are as follows: ,in ,Will As the latest human skeleton length, it is used in the 3D human body model fitting stage; Using the 3D joint positions of the human body model Obtain the skeleton vector ,Will The shape parameters of the SMPL model are considered as the fitting target. The loss function is defined as follows for the parameters to be updated: in After iteration, the final fitting result is obtained; S52 Calculate global rotation : Corresponding SMPL model attitude parameters In , Representing the global rotation, a rigid structure is defined by three joints of the SMPL model, based on the initial positions of these three joints. and predicted location Find a rotation matrix ,make Vector obtained after rotation transformation With the predicted location The minimum sum of distances, which minimizes the distance. That is The formula is expressed as follows: in Let represent a three-dimensional rotation group consisting of three-dimensional rotation matrices. The above formula can be solved in a closed form by singular value decomposition (SVD). S53 Calculate the rotation of each joint: The parent node in the 3D human body model is defined along the human motion skeletal chain, and the node is denoted as... The initial position of the 3D human body model is The target location is Nodes in the reconstructed model The position is , record nodes The skeletal vector is Attitude parameters Represents a node Relative to its parent node rotation amount Then the node absolute rotation First, align the root nodes of the SMPL model. The global rotation is obtained through S52. Then, moving along the motion skeletal chain of the 3D human model from the root node to the leaf node, the relative rotation of each node is calculated step by step. If the currently calculated node is The reconstructed node The position of the parent node is The initial bone vector is After offsetting the rotation effect of the parent node, the target bone vector is obtained as follows: Let the axis of rotation be . The rotation angle is ,but , The rotation matrix is ​​obtained using the Rodriguez formula: in express identity matrix express For an antisymmetric matrix, the relative rotation... That is ; S54 Optimization based on detected 2D key points and ground-touching tags: Shape parameters obtained through steps S51, S52, and S53. With attitude parameters The SMPL model is obtained, and the SMPL model is used to regress three-dimensional coordinates through linear mapping. Then, combined with the camera intrinsic parameter matrix estimated by the camera correction module Find the coordinates of the two-dimensional key points after the SMPL model is projected. This step combines a normalized two-dimensional human body keypoint sequence. The probability of ground contact predicted by the human trajectory prediction module Further iterative optimization of the human body model is needed, with the pose parameters to be updated. Attitude parameters The loss function is defined as follows: in The final optimized result is obtained through iteration.

2. The multi-stage human motion capture method based on monocular video according to claim 1, characterized in that: The first stage of data preprocessing includes three sub-tasks: extracting 2D human key points, converting 3D human joint points, and generating ground touch tags. S11 Extraction and Normalization of 2D Human Keypoints: The original video in the training set is downsampled and frames are extracted. The video frames are then fed into the human pose detector to estimate the image coordinate sequence of 25 2D human keypoints. , The video frame rate is then used to determine the aspect ratio while maintaining the overall aspect ratio. The coordinates of the axis are determined by Normalization to ,Right now ,in For the width and height of the video frame; S12 Transformation of 3D Human Joints: This step transforms the coordinates of 3D human joints from the world coordinate system to the camera coordinate system for monitoring purposes. For the camera's rotation matrix and displacement, The conversion formula for the position coordinates of the 24 joints of the 3D human body model SMPL in the world coordinate system is as follows: ,in This represents the spatial coordinates of the human body's 3D joints in the camera coordinate system. Finally, the spatial position coordinates of the root node in the camera coordinate system are retained, and the 3D coordinates of the other joints are subtracted from the 3D coordinates of the root node to convert them into relative coordinates relative to the root node. The true value of the 3D trajectory is the sequence of 3D coordinates of the root node in the camera coordinate system, i.e. The depth of the root node is ,in This indicates a slicing operation along the corresponding dimension; S13 generates binary ground contact labels: For a 3D joint sequence, the displacement velocity of a joint is first calculated by subtracting the values ​​of two adjacent frames. If the velocity of a node is less than a set threshold, the joint is considered to be stationary in that frame. Then, the DBSCAN method in machine learning is used to cluster the heights of the stationary left and right foot joints. The minimum median minus an offset constant is used as the ground height. For the eight joints of left foot, right foot, left ankle, right ankle, left knee, right knee, left wrist, and right wrist, if the velocity is less than a set threshold and the height difference between the joint and the ground is within a set range, the joint is considered to be in a ground contact state, and the ground contact label is set to 1.

3. The multi-stage human motion capture method based on monocular video according to claim 1, characterized in that: The human pose estimation module consists of two stages: encoding and decoding, which normalize the two-dimensional human pose keypoint coordinate sequence. As input, features are extracted from the input sequence, and then these features are fed into the decoder to predict intermediate frames, i.e., the [number]th [frame]. The relative positions of the human body's joints in the frame in three-dimensional space are as follows: During the encoding phase, long-term features are extracted by grouping motion chains along the channel dimension and performing dilated convolutions along the temporal dimension. The encoder is a fully convolutional network, implemented using dilated convolution operations and residual learning. The human skeleton is divided into six local motion chains: head motion chain, root node motion chain, left arm motion chain, right arm motion chain, left leg motion chain, and right leg motion chain. Each movement is relatively independent. The input is divided into 6 groups along the channel dimension for feature extraction. In the encoder, the output of each layer is divided into 7 parts, including 6 local features. and a global feature , No. The output of the layer network is denoted as From the first Layered network To the Layered network The calculation process is as follows: global convolution is used to extract global information, and convolution is performed on the {global, grouped} connected features in groups, that is: in, All are dilated convolution operations with a stride of 3, a kernel size of 3, and a dilation coefficient of 3. For the connection operation along the channel dimension, the encoder ultimately outputs a joint grouped feature. ; Then, in the decoding stage, the 24 joints in the parameterized 3D human body model are regressed in a multi-branch manner, and the local features of the groups are analyzed. The inputs are fed into 6 independent decoders. Predicting the position of joints on the corresponding local kinetic chain of the human model ,Right now: 。 4. The multi-stage human motion capture method based on monocular video according to claim 1, characterized in that: The camera calibration module uses a pre-trained ResNet-50 model as the backbone network and replaces the last layer with three independent MLP head networks, which are used to estimate the camera's vertical field of view. Rotation along the x-axis and rotation angle along the z-axis The camera focal length is calculated using vfov: Then the camera's intrinsic parameter matrix is: The camera's projection matrix is ​​obtained based on the camera's intrinsic and extrinsic parameters. During the training phase, the projection matrix projects the three-dimensional nodes of the human body predicted by the network onto a two-dimensional plane to obtain the two-dimensional coordinates of the corresponding joints on the image, and calculates the distance loss with the two-dimensional ground truth for supervision. for , and For each quantity in the prediction, a range of values ​​is predefined. Furthermore, 256 candidate values ​​are uniformly sampled within this range, and each of the three MLP head networks outputs a 256-dimensional vector. Represents the probability of a candidate value. The vector is approximated by the index corresponding to the maximum probability using the softargmax function. ,Will Normalization to Normalized index obtained within the range Then the index is mapped to the range of candidate values. The candidate value at the corresponding position is obtained. , This is the prediction result of the network, and the calculation process is as follows: During training, standard methods are used for pitch and roll prediction. Loss function; for VFov prediction, the Geman-McClure loss function is used, i.e. .

5. The multi-stage human motion capture method based on monocular video according to claim 4, characterized in that, for real-time motion capture applications, the image captured by the camera at the current moment is sent to the camera calibration module to estimate camera parameters, and the estimated value based on past frames is recorded as... The network estimates the value based on the current frame. Afterwards, The update will be performed, and the update rules are as follows: ,in ,Will This serves as the latest camera estimate for use in subsequent modules.

6. The multi-stage human motion capture method based on monocular video according to claim 1, characterized in that: The human trajectory estimation module consists of two stages: encoding and decoding, based on the coordinate sequence of normalized two-dimensional human pose keypoints. The encoder takes only the key points on the left and right leg kinematic chains as input. During the encoding stage, the encoder uses a dilated convolutional network to extract features from the input sequence. During the decoding stage, the extracted features are processed by two different head networks to predict the 3D coordinates of the human root node in the intermediate frame in the camera coordinate system. The probability of contact with the ground at various joints of the human body Based on the camera intrinsic parameter matrix estimated by the camera correction module The human pose estimation module predicts the coordinates of the three-dimensional joints relative to the root node. The three-dimensional spatial coordinates of the root node predicted by the human trajectory prediction module. The coordinates of the joints projected onto the two-dimensional image plane are obtained. The calculation process is as follows: 。 7. An electronic device, characterized in that: The device includes a storage medium and a processor for performing human motion capture based on monocular video. The storage medium stores a computer program, and the processor executes the computer program. When the computer program is executed, it implements the multi-stage human motion capture method based on monocular video as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores a computer program that, when executed, implements the multi-stage human motion capture method based on monocular video as described in any one of claims 1-6.

Images